[03:45:21] 3Tool-Labs: Make HHVM based webservices available on toollabs - https://phabricator.wikimedia.org/T78783#969609 (10ori) I'm not sure that this is a great idea. Small web applications are probably not spending a lot of time executing code in user space, so the potential for gain is small, and must be weighed agai... [06:58:56] did someone else just go through the shell requests? [07:05:03] 3Tool-Labs: Make HHVM based webservices available on toollabs - https://phabricator.wikimedia.org/T78783#969721 (10yuvipanda) p:5Triage>3Low @ori: There are at least several tools that I think should benefit from this. A casual look at https://tools.wmflabs.org/?status shows quite a few web tools that have h... [07:12:37] 3Wikidata, Tool-Labs-tools-Other: Autolist keeps dying - https://phabricator.wikimedia.org/T86134#969724 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Yup, closing as fixed now. [07:14:57] 3Tool-Labs: Provide regular report on tools with single owner - https://phabricator.wikimedia.org/T86432#969727 (10yuvipanda) p:5Triage>3Normal [07:15:54] 3Tool-Labs: Uwsgi breaks flask project-relative URLs - https://phabricator.wikimedia.org/T85362#969729 (10yuvipanda) 5Open>3Resolved Fixed now, I think. [07:19:27] 3pywikibot-core, Pywikibot-compat-to-core, Tool-Labs, pywikibot-compat: patrol.py depends on mwlib.uparser not available on wmflabs - https://phabricator.wikimedia.org/T71980#969739 (10yuvipanda) If you're talking about installing https://github.com/pediapress/mwlib on toollabs, I could get that done (if this is... [07:21:37] 3Tool-Labs-tools-Other, Tool-Labs: https://tools.wmflabs.org/merlbot-web/ 404s - https://phabricator.wikimedia.org/T85739#969745 (10yuvipanda) [07:23:15] !log pubsubhubbub migrating pubsubhubbub to virt1006 [07:23:20] Logged the message, dummy [07:24:54] * YuviPanda waves at andrewbogott [07:24:57] hopefully feeling bettr [07:25:04] only barely :( [07:26:00] andrewbogott: :( ok. let me know if there’s anything I can do to help [07:26:27] 3pywikibot-core, Pywikibot-compat-to-core, Tool-Labs: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#969756 (10yuvipanda) Hmm, I'm wondering if we should figure out a virtualenv based way of setting up pywikibot on toollabs. This would give pywikibot maintainers... [07:28:59] 3pywikibot-core, Pywikibot-compat-to-core, Tool-Labs: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#969770 (10yuvipanda) However, if the pywikibot folks think that maintaining the virtualenv themselves is going to be too much work / not something they want to d... [07:29:26] 3pywikibot-core, Pywikibot-compat-to-core, Tool-Labs: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#969771 (10yuvipanda) [07:29:36] 3pywikibot-core, Pywikibot-compat-to-core, Tool-Labs: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#959861 (10yuvipanda) [07:29:48] 3Tool-Labs: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#969773 (10yuvipanda) [07:30:42] 3Tool-Labs: Make DynamicProxy be able to proxy back to non-http protocols - https://phabricator.wikimedia.org/T84983#969778 (10yuvipanda) p:5Triage>3Low [07:35:12] YuviPanda: it looks like you fixed (or greatly reduced) the DNS-failure issue while I was out of commission… that's the only thing that was stressing me :) [07:35:51] andrewbogott: :D yay! :) [07:40:18] andrewbogott: I’m trying to learn more about OpenStack now. trying to set up a local DevStack instance to play around in [07:40:47] sadly don’t have a spare linux machine around, so have to put it in a VM [07:40:54] Cool. Devstack doesn't very closely resemble our setup, but it's fun to tinker with [07:40:58] It runs OK in a VM [07:41:23] andrewbogott: yeah, at least to understand the components properly. [07:47:59] !log design migrating living-style-guide to virt1006 (or, at least, trying) [07:48:04] Logged the message, dummy [07:52:13] 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#969830 (10yuvipanda) p:5High>3Triage @Stryn: Works for me when I just tried. Reducing priority since it seems to work now. [07:52:31] 3Tool-Labs: Setup an icinga instance to monitor tools on tool-labs - https://phabricator.wikimedia.org/T53434#969833 (10yuvipanda) p:5High>3Triage We have shinken! [07:53:01] 3Tool-Labs: Toolserver redirect configuration broken after domain move - https://phabricator.wikimedia.org/T85166#969835 (10yuvipanda) p:5High>3Normal @Coren: Can we put the redirects on a git repo somewhere? The web redirects shouldn't be private info, I think. [07:54:07] valhallasw`cloud: merge? :D [07:57:18] YuviPanda: no __copy__ needed anymore/ [07:58:29] YuviPanda: needs rebase or has dep [07:58:38] valhallasw`cloud: yup. has two dependent patches :P [07:59:07] YuviPanda: -_- [07:59:17] so my +2 is completely useless :P [07:59:34] valhallasw`cloud: :P let me +2 the others [07:59:48] they’re all trivial [08:00:37] valhallasw`cloud: also, move g-p-u back to uwsgi? :D I gave you a local git [08:01:50] !log editor-engagement migrating instance 'legoktm' to virt1006 [08:01:55] Logged the message, dummy [08:07:21] YuviPanda: yrah at some point [08:15:30] 3Tool-Labs, pywikibot-core, Pywikibot-compat-to-core: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#969863 (10jayvdb) @yuvipanda, it would be great if you could install these dependencies in the first instance. Some of them are distributed as part of the OS pa... [08:17:17] 3Quarry: Number of queries shown in profile is wrong - https://phabricator.wikimedia.org/T86512#969867 (10Schnark) 3NEW [08:19:46] valhallasw`cloud: I sent you pull requests :) [08:19:55] [13gerrit-patch-uploader] 15yuvipanda opened pull request #31: Fix PEP8 errors (06master...06pep8-fixes) 02http://git.io/vlhM4Q [08:22:14] 3Gerrit-Patch-Uploader, Wikimedia-Git-or-Gerrit: Gerrit-patch-uploader fails under git 1.9 - https://phabricator.wikimedia.org/T86304#969875 (10yuvipanda) https://github.com/valhallasw/gerrit-patch-uploader/pull/30 should fix it. [08:25:30] valhallasw`cloud: also christ, why aren’t you just using a template directly instead of jinja2.Markup everywhrere?! [08:43:10] 3Tool-Labs, pywikibot-core, Pywikibot-compat-to-core: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#969900 (10yuvipanda) So I've installed: # unittest2 # pycountry # flickrapi # pydot pysearch seems to be just pointing to python-yahoo now in ubuntu. Is that o... [08:43:55] YuviPanda: because yield [08:44:16] valhallasw`cloud: oh. [08:44:18] valhallasw`cloud: hmm [08:44:22] I'm not using it everywhere, I'm specifically using it in the upload part [08:44:27] valhallasw`cloud: I didn’t actually read the code fully :P [08:44:28] so fine [08:45:29] [13gerrit-patch-uploader] 15valhallasw pushed 2 new commits to 06master: 02http://git.io/PV4DHQ [08:45:29] 13gerrit-patch-uploader/06master 14d12c424 15YuviPanda: Fix PEP8 errors... [08:45:30] 13gerrit-patch-uploader/06master 142d6db96 15Merlijn van Deen: Merge pull request #31 from yuvipanda/pep8-fixes... [08:48:02] YuviPanda: can't merge the local git one now :< [08:49:22] valhallasw`cloud: gah [08:49:24] valhallasw`cloud: let me rebase [08:49:31] YuviPanda: <3 [08:51:56] valhallasw`cloud: forcepushing [08:52:04] YuviPanda: >:( [08:52:19] oh, on your own branch [08:52:20] valhallasw`cloud: done [08:52:20] fine :-p [08:52:23] valhallasw`cloud: oh yeah :P [08:52:31] valhallasw`cloud: I’m not going to force push to your master :P [08:52:32] * valhallasw`cloud waits for phawkes [08:52:46] valhallasw`cloud: phawkes won’t update force pushes on PullRequests, I think [08:52:51] clearly not [08:52:52] GH doesn’t deal with them too well either [08:53:46] [13gerrit-patch-uploader] 15valhallasw closed pull request #29: Future-proof the comment posted on gerrit (06master...06patch-2) 02http://git.io/SQVcdQ [08:53:59] [13gerrit-patch-uploader] 15valhallasw pushed 2 new commits to 06master: 02http://git.io/48RToA [08:53:59] 13gerrit-patch-uploader/06master 143b1d206 15YuviPanda: Use a local copy of Git... [08:54:00] 13gerrit-patch-uploader/06master 1466e2e7a 15Merlijn van Deen: Merge pull request #30 from yuvipanda/local-git... [08:54:15] we should get a bouncer :< [08:54:15] valhallasw`cloud: \o/ [08:54:15] hi [08:54:30] hi Faisal [08:54:40] hi yuvi [08:55:05] what are you talking about [08:56:45] this is a channel for talking about Wikimedia Labs [08:56:51] (https://wikitech.wikimedia.org/wiki/Help:FAQ) [08:58:15] 3Wikimedia-Labs-Infrastructure: Allow login using mosh as an alternative to plain ssh on bastion - https://phabricator.wikimedia.org/T54693#969918 (10yuvipanda) p:5Triage>3Low [08:58:29] 3Wikimedia-Labs-Infrastructure: Allow login using mosh as an alternative to plain ssh on bastion - https://phabricator.wikimedia.org/T54693#969920 (10yuvipanda) p:5Low>3Volunteer? a:5yuvipanda>3None [09:01:18] what project are you working on [09:01:49] YuviPanda: go fix host based auth :P [09:02:11] valhallasw`cloud: heh, it’s fairly more difficult than I had originally realized. [09:02:33] maybe you should ask river :> [09:02:42] valhallasw`cloud: SSHFP perhaps http://www.openssh.com/txt/rfc4255.txt [09:02:54] YuviPanda: what do you need that for? :/ [09:03:02] valhallasw`cloud: uhm, host keys? [09:03:18] need to collect them all, so we know that the host that says it is bastion *is* bastion? [09:03:54] YuviPanda: I'd just hardcode the bastion signatures in puppet [09:04:09] hmmm [09:04:23] that could work. [09:04:28] it's not like we create new bastions every week or so [09:07:37] i like wikipedia [09:10:10] 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#969930 (10yuvipanda) [09:17:24] 3Tool-Labs, pywikibot-core, Pywikibot-compat-to-core: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#969931 (10jayvdb) daemonize supports the library command line option -daemonize. I dont know much more than that, but I can dig into it. crontab is used to par... [09:20:48] 3Tool-Labs, pywikibot-core, Pywikibot-compat-to-core: Install all pywikibot python dependencies on tool labs - https://phabricator.wikimedia.org/T86015#969932 (10yuvipanda) I'm not sure if we should support daemonize. Jobs should just go on the grid instead. [09:31:35] 3Labs-Team, operations: facter: VM detection incorrect in labs - https://phabricator.wikimedia.org/T78813#969949 (10faidon) 5Open>3Resolved a:3faidon I fixed this today with 272b0f4ba495d4b6fba12069bce3c457da4d46c7, for an entirely different reason, an hour before realizing this existed as a task. Funny! [09:36:13] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:41:39] ^ looking [09:46:16] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [09:46:26] heh, just needed a apt-get update before [10:25:11] (03CR) 10Hashar: "Thanks Merlijn and Legoktm for your follow up!" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/183371 (https://phabricator.wikimedia.org/T86053) (owner: 10Hashar) [10:49:46] YuviPanda: do you see those security emails for tools-login? [10:49:54] andrewbogott: yup. [10:50:08] andrewbogott: someone trying to use pycharm’s remote run capabilities [10:50:19] So you think non-malicious? [10:50:22] yeah [10:50:26] hm [10:50:31] they need to throttle their attempts [10:50:48] andrewbogott: well, or fix them at least. [10:51:22] In the meantime... [10:51:37] IP block? I don't actually know how to do that but it seems warranted [10:52:15] andrewbogott: yeah, let me do that [10:52:32] Want to talk me through it so I can learn? [10:52:45] andrewbogott: actually, I think I’ll just remove shell [10:52:46] from the user [10:53:00] ah, in ldap? That I know how to do. [10:53:10] Grab the email address while you're in there :) [10:53:37] andrewbogott: oh, I just did that in the wikitech interface [10:53:45] ah, sure. [10:53:51] OK, I'll dig up the email address [10:53:55] andrewbogott: I’ll send an email through that as well [10:54:02] ok :) [10:54:03] * andrewbogott does nothing [10:54:33] ok, that should’ve stopped [10:55:40] andrewbogott: I also killed current open ssh processes [10:55:54] great. [10:58:19] andrewbogott: and emailed [11:11:38] O_o what does pycharm do to trigger security notifications [11:11:43] sudo all the things? [11:11:55] valhallasw`cloud: ya [11:42:34] 3Wikibugs, Release-Engineering: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#970189 (10hashar) [11:42:53] 3Wikibugs, Release-Engineering: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#960529 (10hashar) We also need to migrate qa-morebots which reacts to `!log` commands. [12:08:54] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:09:22] 3Wikibugs, Wikidata, DataValues, MediaWiki-General-or-Unknown: Philosophical question: should bugs where *any* projects match be removed from -dev, or projects where *all* projects match? - https://phabricator.wikimedia.org/T86525#970538 (10valhallasw) 3NEW [12:10:18] 3Wikibugs: Philosophical question: should bugs where *any* projects match be removed from -dev, or projects where *all* projects match? - https://phabricator.wikimedia.org/T86525#970538 (10valhallasw) [12:11:13] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:15:25] ^ transient errors caused by stale file server on puppetmaster, these are succeeding now [12:28:56] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [12:36:16] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:46:08] 3Wikimedia-Labs-Infrastructure: Commons replica DB excessively out-of-date - https://phabricator.wikimedia.org/T75980#970626 (10Magnus) 5Open>3Resolved a:3Magnus Seems to work now. [13:09:26] 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#970672 (10Aklapper) > Reducing priority since it seems to work now. @YuviPanda: Why was "Needs triage" priority set? This is already triaged and you did not reduce priority but increased it. You could set "Stalled" status instead... [13:10:33] 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#970673 (10yuvipanda) p:5Triage>3Low [14:35:42] 3operations, Labs-Team: facter: VM detection incorrect in labs - https://phabricator.wikimedia.org/T78813#970807 (10Gage) Thanks! The above link doesn't work for me ("Access Denied: Restricted Application"), change is https://gerrit.wikimedia.org/r/#/c/184291/ [15:16:16] YuviPanda: thanks for fixing DNS. shinken was mostly quiet over the weekend. [15:29:14] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<66.67%) [15:39:16] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [16:00:27] 3Tool-Labs-tools-Other: Requesting all pages created creates a 502 Proxy Error - OOM? - https://phabricator.wikimedia.org/T61633#970949 (10Se4598) [16:26:29] 3Wikibugs, Release-Engineering: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#971039 (10greg) [16:50:49] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:52:52] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:52:56] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:53:52] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:54:15] 3Wikidata, Tool-Labs: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#971100 (10Magnus) #4 is moot, as I changed CatScat to not use temporary tables anymore. If you still see this happening, it's probably this copy: http://tools.wmflabs.org/catscan3/catscan2.php which w... [16:54:20] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:54:34] we don't have a way to spin up jessie labs nodes easily do we? [16:54:57] Coren: ^ :) [16:54:58] PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:55:02] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:55:04] I was having that conversation with Coren last week [16:55:14] there's one pending TODO item afaik [16:55:20] 3Tool-Labs: Multiple webservices running for one tool - https://phabricator.wikimedia.org/T76578#971102 (10Magnus) This was before webservice2, AFAIK. [16:55:34] well I kinda gave up on the horrible hack of trying to get pbuilder/cowdancer stuff to do chroot package builds for jessie on a trusty host [16:55:55] paravoid: Basically. I'll try to squeeze it in this week between "I'm ready to do the filesystem move" and "I'm doing the filesystem move". [16:56:08] so it's either jessie labs node, or I just go make my own custom VM for it at home or something, or I guess I could install build tools and whatnot onto cp1008 for now since it's a throwaway install anyways [16:56:13] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:56:14] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:56:29] With some pre-live rsync I think I should be able to get the actual downtime down to ~6h [16:56:33] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:56:38] * Coren knocks on wood. [16:57:00] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:57:11] o_O who broke puppet? [16:57:14] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:57:15] I think in the interest of expediency, then, I'll probably go with option 3 in the short term [16:57:24] Coren: me probably [16:58:00] how do I ssh to e.g. "tools-static" ? [16:58:08] Don't blame yourself too quickly, I'm looking into it now. [16:58:26] bblack: From any (labs) bastion, with that name. [16:58:32] well I just merged a couple default package installs that are hardware-related [16:58:39] E: Unable to locate package intel-microcode [16:58:46] PROBLEM - Puppet failure on tools-uwsgi-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:58:46] Warning: Permanently added 'tools-static.eqiad.wmflabs' (RSA) to the list of known hosts. [16:58:49] Connection closed by UNKNOWN [16:58:54] ? [16:59:01] and I was raising the point that with the SSL work it becomes a bit more imminent as a need now [16:59:01] bblack: d-i-test.eqiad.wmnet [16:59:02] bblack: do whatever you want with it [16:59:11] bblack: You have to be in the tools project to connect to tools instances. Labs SOP. :-) [16:59:14] it's a gross hack that I did for testing jessie [16:59:22] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:59:27] I should be in the "I'm fucking root let me into everything" project [16:59:34] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:59:59] bblack: Heh. You should be able to login with root if your public key is there. [17:00:14] well I generally log in as me everywhere and sudo, not as root directly [17:00:16] bblack: But your error is just that intel-microcode thing. [17:00:29] so, I need to conditionalize that package against labs? [17:00:34] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:00:42] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [17:00:46] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:00:48] may as well mcelog too I guess, since they aren't real machines [17:01:00] bblack: Well, it's virtual hardware so it'd be pointless anyways. [17:01:06] right [17:01:06] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:01:43] $::is_virtual is your friend [17:01:50] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:01:57] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:01:57] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:01:59] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:02:08] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [17:02:13] paravoid: Good point, and even better since we intend to have virtual misc hosts. [17:02:19] but that error above sounds like an apt error [17:02:29] It is, that package isn't found at all. [17:02:33] not an "installed but not working" [17:02:48] that's strange, cf. http://packages.ubuntu.com/search?keywords=intel-microcode [17:02:49] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:03:19] Hm. Proprietary and in multiverse? [17:03:43] We don't have multiverse in labs, iirc. [17:04:00] I wonder if jessie has it too [17:04:23] yes, it does [17:04:41] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:04:45] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:05:25] paravoid: bblack: confirmed, multiverse is not enabled in labs by default; and that's where that package lives. [17:05:44] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:05:46] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:06:14] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:06:28] Odd, though, I'd have though that'd live in restricted. [17:07:14] labstore1001 is low disk space [17:07:20] well, still, I'll push an unless $::is_virtual fixup [17:07:20] since about 3d [17:07:38] mutante: I know; I'm about to rejigger the filesystems and I need every bit of LVM unallocated space. [17:07:53] Coren: alright,'k [17:08:12] mutante: This will magically go away after Thursday. [17:08:20] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:08:30] Coren: cool [17:09:53] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:09:58] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:10:26] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:11:21] Coren: can you try one of the puppetfail hosts? [17:11:57] bblack: Trying now. [17:12:14] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:12:28] (also, root@ doesn't work either, although at least that way I get permission denied instead of connection closed by UNKNOWN) [17:14:03] bblack: root@ should work iff your key is in the /labs'/ public keys (in the increasingly misnamed) private repo [17:14:40] bblack: u can haz success, though there is some ugliness with a necessary apt-get update related to ssl. Things should recover on their own shortly. [17:15:31] Error: /Stage[main]/Certificates::Base/Package[openssl]/ensure: change from 1.0.1-4ubuntu5.20 to latest failed [...] Cache is out of sync, can't x-ref a package file"; that is fixed upon the next apt-get update [17:16:52] ok, thanks! [17:22:06] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:48] RECOVERY - Puppet failure on tools-uwsgi-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:30] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:38] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:08] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:14] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:18] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:17] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:47] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:24] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:42] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:44] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:48] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:31:14] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [17:31:54] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:31:56] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [17:31:58] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:48] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:30] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:17] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:16] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:18] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:52] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:54] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:56] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:02] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:44] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:46] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:50] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:36] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:58] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:00] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:50] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:58] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:43:54] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [18:04:30] 3Wikibugs, Release-Engineering: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#971186 (10greg) >>! In T86053#970189, @hashar wrote: > We also need to migrate qa-morebots which reacts to `!log` commands. @jeremyb: can you help with that part (moving morebots fr... [18:32:11] Coren: labstore's disk space alert went critical on friday btw [18:32:35] either the root cause should be fixed, or the alert should be marked as acknowledged in icinga, or its thresholds adjusted [18:33:07] I should ack it; I'm about to rejigger the filesystems (but also, I'm probably going to be able to clean a few 100s of G in the cleanup) [18:33:08] I dislike having critical alerts fired up as it makes us getting used to them and potentially ignoring a real alert [18:36:40] I'm hunting for outliers to clean before the copy today. [18:36:55] :) [18:37:58] andrewbogott_afk: Coren I just set up a local install of horizon + openstack (devstack). Horizon looks very nice! :) [18:38:32] It does. It has a lot of potential to be very good. [18:39:01] yup [18:39:13] devstack was also very easy to set up [18:57:00] 3Labs-Team, Wikimedia-Labs-Infrastructure: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#971367 (10faidon) p:5Triage>3High [18:57:43] 3Labs-Team, Wikimedia-Labs-Infrastructure: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#763305 (10faidon) Jessie is approaching production fast now so having a Labs image is becoming more important by the day. Increasing priority to High. [20:58:58] Hm. What is the 'dumps' project about, exactly? Does anyone know? [21:00:12] Nemo_bis: You're listed as an admin. Can you give me the nutshell? [21:10:22] Coren: https://archive.org/details/wikimediadownloads [21:10:46] It's all described on the wiki https://wikitech.wikimedia.org/wiki/Nova_Resource:Dumps [21:11:59] 3Release-Engineering, Wikibugs: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#971928 (10greg) [21:18:05] 3Release-Engineering, Wikibugs: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#971942 (10hashar) [21:18:42] 3Release-Engineering, Wikibugs: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#960529 (10hashar) [21:19:12] 3Release-Engineering, Wikibugs: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#960529 (10hashar) qa-morebots got moved thanks to @valhallasw Gotta switch wm-bot now :-D [21:33:18] 3Release-Engineering, Wikibugs: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#971986 (10hashar) [21:33:50] 3Release-Engineering, Wikibugs: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#960529 (10hashar) I have killed wm-bot in -qa and legoktm have it join the -releng channels. We are missing some authorization, but that is not a blocker. [21:39:01] Nemo_bis: I was mostly wondering about your storage requirements; they're not the worse but they pop as one of the bigger ones. Is all of that data live? [21:41:26] 3Release-Engineering, Wikibugs: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#972018 (10hashar) Nothing left to do apparently. Thank you to everyone involved, I am letting greg-g to confirm and close :] [21:59:12] (03PS1) 10Merlijn van Deen: team-practices -> #wikimedia-teampractices [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184488 [21:59:35] legoktm: could you review some of https://gerrit.wikimedia.org/r/#/q/status:open+project:labs/tools/wikibugs2,n,z ? [22:02:15] (03CR) 10Legoktm: [C: 032] Add Taxonomy cron job [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184128 (https://phabricator.wikimedia.org/T85096) (owner: 10Merlijn van Deen) [22:02:24] (03CR) 10Legoktm: [C: 032] Move taxonomy to Phabricator/Projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184129 (https://phabricator.wikimedia.org/T85096) (owner: 10Merlijn van Deen) [22:02:36] (03CR) 10Legoktm: [C: 032] Make phabricator and phabricator.org always report to -dev [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184196 (owner: 10Merlijn van Deen) [22:02:38] (03Merged) 10jenkins-bot: Add Taxonomy cron job [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184128 (https://phabricator.wikimedia.org/T85096) (owner: 10Merlijn van Deen) [22:02:42] (03CR) 10Legoktm: [C: 032] team-practices -> #wikimedia-teampractices [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184488 (owner: 10Merlijn van Deen) [22:02:55] (03Merged) 10jenkins-bot: Move taxonomy to Phabricator/Projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184129 (https://phabricator.wikimedia.org/T85096) (owner: 10Merlijn van Deen) [22:02:57] (03Merged) 10jenkins-bot: Make phabricator and phabricator.org always report to -dev [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184196 (owner: 10Merlijn van Deen) [22:03:03] (03Merged) 10jenkins-bot: team-practices -> #wikimedia-teampractices [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184488 (owner: 10Merlijn van Deen) [22:03:28] valhallasw`cloud: do you want me to install the crontab? [22:03:36] legoktm: it's already installed [22:03:37] mmm [22:03:58] !log tools.wikibugs Updated channels.yaml to: f1ee8fb8bc64186a1613ec9f9faf0aef6315759a Merge "team-practices -> #wikimedia-teampractices" [22:04:02] Logged the message, Master [22:04:11] (I actually tested my code :o) [22:04:22] heh [22:05:33] how does one go about getting sudo access on deployment-prep? [22:08:38] nevermind, I think I figured it out [22:12:08] legoktm: I just realised there's no license in wikibugs2 :< [22:12:37] legoktm: shall I just slap it with http://opensource.org/licenses/MIT ? [22:13:55] valhallasw`cloud: crap. Yeah that's fine, I also copied some code from pywikibugs, whatever that was licensed as. [22:14:04] that was mit [22:15:04] MIT sounds good then. [22:15:11] Probably also want to check with YuviPanda|zzz [22:16:25] yeah, if we want to be really nice, we should ask all authors for +1 :-p but I think ignoring channels.yaml for that is probably ok [22:18:40] * marktraceur disables wikibugs until it stops being non-free [22:18:42] 3Wikibugs, Release-Engineering: Unify RelEng related IRC channels to #wikimedia-releng - https://phabricator.wikimedia.org/T86053#972161 (10greg) 5Open>3Resolved {{DONE}}! Thanks for all the help from those who merged all the patches/etc. [22:18:48] GOD YOU GUYS. TOS. [22:20:32] (03PS1) 10Merlijn van Deen: Add LICENSE and CREDITS [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184497 [22:21:45] (03CR) 10Merlijn van Deen: [C: 031] "I don't think we need +1's from everyone who contributed channel configs, but Legoktm, Yuvipanda and myself have contributed significant p" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184497 (owner: 10Merlijn van Deen) [22:21:51] YuviPanda|zzz / legoktm ^ [22:29:14] (03CR) 10Legoktm: [C: 031] Add LICENSE and CREDITS [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184497 (owner: 10Merlijn van Deen) [22:35:51] legoktm: super simple [and probably broken] php rate limiter built \i/ [22:36:09] legoktm: try accessing https://tools.wmflabs.org/wikibugs/pull.php twice witin 5 secs :D [22:39:10] (03PS1) 10Merlijn van Deen: Sort taxonomy tags before adding to OrderedDict [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184503 [22:40:35] 3Phabricator, Wikibugs: Set up dumping Phabricator's project taxonomy to a wiki - https://phabricator.wikimedia.org/T85096#972242 (10valhallasw) 5Open>3Resolved This is now at https://www.mediawiki.org/wiki/Phabricator/Projects . Please feel free to update header/footer to your liking! [22:42:46] valhallasw`cloud: I think a whitelist by IP is better, if we merge a sequence of changes jenkins will hit the endpoint, next patch is merged and it'll trigger rate limiting [22:49:44] legoktm: we can't [22:49:48] legoktm: IP is not available [22:50:06] legoktm: and we can always merge manually [22:51:43] legoktm: I think it'll probably be fine. Besides, manual pulling is as easy as visiting a URL ;-) [22:52:32] anyway, bedtime [22:52:50] valhallasw`cloud: or fab pull!!!! [22:52:57] yep [22:53:21] but that's alt-tab ctrl-a 0 cd ~/src/wikibufab pull [22:53:40] this is ctrl-t pull. [22:53:41] ;-) [22:54:07] anyway, good night! [22:54:23] night :) [23:28:11] (03CR) 10Legoktm: [C: 032] Sort taxonomy tags before adding to OrderedDict [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184503 (owner: 10Merlijn van Deen) [23:28:22] (03Merged) 10jenkins-bot: Sort taxonomy tags before adding to OrderedDict [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/184503 (owner: 10Merlijn van Deen)