[00:01:22] 10Labs-project-Wikistats: wikistats: add new wikipedias: kbp, khw, dty and pt.wikimedia - https://phabricator.wikimedia.org/T160947#3373757 (10Dzahn) 05stalled>03Resolved kbp.wp - added just now khw.wp - invalid, wasn't created dty.wp - was alread added pt.wm - added just now [00:01:59] !log wikistats - added kbp.wikipedia and pt.wikimedia [00:02:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [00:03:47] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3373763 (10Samwilson) Good plan. Is it okay for a composer script to call `composer install`? I guess so. :-) The prod script is currently: ``` #!/bin/bash export SYMFONY_ENV=prod... [02:30:42] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:05:39] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [03:12:59] (03PS1) 10Minhtq15: first commit [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361020 [03:29:04] (03PS1) 10Minhtq15: second [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361021 [03:50:07] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3373918 (10Samwilson) I think this is pretty much done. I mean, there are still lots of SQL queries in controllers (and missing tests), but everywhere is at least using the P... [03:50:38] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3373919 (10Samwilson) a:03Samwilson [03:55:52] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3373921 (10bd808) Thanks for the detective work @zhuyifei1999! [03:56:38] bd808: ;) [03:57:31] zhuyifei1999_: I'd be more than happy to add you as a co-maintainer on that tool too. :) It's a rip off from the precise migration stuff we worked on together [03:57:59] I don't know what I would add though [03:59:20] (I somehow noticed the ticket, found the repo, and it looked strangely familiar. then I looked at the commit history, ah) [04:01:46] Maybe what I should fix is gridengine-status... [04:10:45] !log tools.grid-jobs Update to 8134b16 to fix None crash in T168653 [04:10:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.grid-jobs/SAL [04:10:48] T168653: grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653 [04:18:21] 10Labs: Use wmcloud.org domain for PAWS - https://phabricator.wikimedia.org/T168686#3373928 (10bd808) [04:20:54] PROBLEM - Puppet staleness on tools-elastic-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [04:21:23] (03PS1) 10Minhtq15: a [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361022 [04:27:27] PROBLEM - Puppet staleness on tools-elastic-02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [04:29:03] 10Labs, 10Phabricator, 10wikitech.wikimedia.org, 10LDAP, and 2 others: Blocking an account on wikitech should disable LDAP logins - https://phabricator.wikimedia.org/T168692#3373948 (10mmodell) [04:31:40] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:41:23] 10Labs, 10Tool-Labs: Rotate Grid Engine accounting file - https://phabricator.wikimedia.org/T168701#3373971 (10bd808) [05:00:20] PROBLEM - Puppet errors on tools-worker-1019 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:02:32] !help PAWS is not running (HTTP status 502 Bad Gateway) since at least yesterday evening. Can someone please have a look at this? Thanks [05:02:32] MisterSynergy: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [05:04:07] MisterSynergy: is it when you try to access your server or all the time? I was just able to login with no issues, but I haven't started a server yet [05:04:27] * andrewbogott was just about to chime in with 'works for me' [05:04:40] * bd808 got the 502 when trying to start a sever [05:04:55] ah yes, when I start the server [05:04:57] oh, maybe I stopped too soon [05:04:59] I'll try again [05:05:25] I know that yuvi has been doing some paws work but I think that is all happening in a separate install [05:05:43] paws.wmflabs.org works, but the 502 shows up when I start the server; I originally used a link which already linked to my server [05:06:03] unfortunately I don't know anything about how to actually fix the jupyterhub install there [05:06:33] * bd808 goes to look for docs [05:06:43] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [05:06:43] it is not that urgent for me ;-) [05:07:19] MisterSynergy: do you have time to make a phabricator bug about this? [05:08:40] I can, but I am remarkably talented to write phab tickets which are ignored for a long time then ;-) [05:09:23] that's a fairly well attended club. :) I've held birthday parties for many of mine [05:10:04] MisterSynergy: If you write the bug I'll try to poke the right people when they are online tomorrow morning [05:10:35] glad to hear that it not (only) me. I always wondered whether my style of writing is bad or whatever... [05:11:23] we have lots of software which means lots of problems and only so many people to fix things. Its can be a depressing cycle [05:12:56] 10PAWS: PAWS shows “502 Bad Gateway” on server start - https://phabricator.wikimedia.org/T168702#3373990 (10MisterSynergy) [05:12:57] https://phabricator.wikimedia.org/T168702 [05:13:09] feel free to add more tags, users, projects, etc [05:14:33] 10PAWS: PAWS shows “502 Bad Gateway” on server start - https://phabricator.wikimedia.org/T168702#3373990 (10bd808) The landing page and OAuth login are working, but it looks like launching a new container fails with the 502 error. [05:14:35] do you think that WMF is understaffed in this field? or is this work done by volunteer editors as well? [05:15:32] I don't think its limited to technical work at all. The projects all have very ambitious goals as well [05:16:31] I know from my time working in the CommTech team that a lot of the on-wiki curation workflows fall behind from lack of people with time to do them [05:17:14] Hm… I'm tempted to go back to 'works for me'. I can open a terminal and a notebook… that means I have a server running doesn't it? [05:17:29] this is correct, but it is the communities' duty to organize this [05:17:30] yeah [05:17:54] technical software stuff is done by paid WMF workers, isn't it? [05:18:01] andrewbogott: hmmm... so not universal. Let me try a 3rd time [05:18:21] still 502 here [05:18:27] MisterSynergy: some paids but also a lot of volunteer work [05:18:37] good to know [05:19:18] The Tool Labs project is full of volunteer developed things as one example [05:19:19] I am always wondering that WMF isn't really short on money, but for some reason it seems that they don't invest enough for software development [05:19:56] my terminal is VERY slow (like if I type 'ls' I can count to 20 before I get output.) I don't know if that's normal or not, but maybe we're having an overload issue rather than a 'totally down' issue [05:19:59] * andrewbogott speculates [05:20:19] we could always use more engineers, but in the big picture we do have something like 1/3 of our staff in technical roles [05:21:11] MisterSynergy: even if we had Google's money I don't think we would be able to fix all of the things. [05:21:33] and we are far away from having Google's money :-D [05:21:45] heh. yes very very far [05:22:05] to be measured in orders of magnitudes ... [05:22:31] at least three I guess, if not more [05:23:25] * andrewbogott -> sleep [05:27:29] https://www.statista.com/statistics/266472/googles-net-income/ says GOOG made 16B USD in 2015. The Foundation reported 98.8M USD for the 2014-2015 fiscal year. [05:28:01] 98,800,000 / 16,000,000,000 = 0.006175 [05:28:42] so 0.6% of GOOG's revenue roughly [05:29:46] money is not everything though :) [05:29:53] which is in fact close to my estimation ;-) [05:30:27] yup. good napkin math from you for sure [05:31:10] yes that's true as well, we perform surprisingly well compared to the other top 10 websites given the fact that they all have *much* more money [05:31:54] I thnk all of us who are lucky enough to work here are pretty proud of that fact [05:33:35] still, from an engaged Wikimedia editors point of view it is sometimes frustrating to see how slow technical progress is; this appears to be related to the number or workers, not their skills ;-) [05:34:18] a bit. also a bit the number of things we try to do at the same time [05:36:31] there are certainly some problems that throwing more money & people at would help. others are just hard problems no matter what [05:37:55] interesting to read; which kind of problems do you refer to as "hard" regardless of efforts? I'm curious now :-D [05:40:22] RECOVERY - Puppet errors on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [05:40:25] preventing harassment; figuring out how to attract, train, and retain more editors; figuring out how to reach more readers; ... [05:41:09] ah, non-technical problems [05:42:14] google, facebook, and others have shown that with enough capital resources many difficult technical problems can be solved, but all software has bugs and nothing makes everyone happy [05:42:52] in that sense it is probably a good idea to keep the 1/3 ratio of technical workers, and let (parts of) the others work on non-technical problems as well [05:44:42] yes, this is what I thought as well; we have impressive numbers of revisions, pages, items, whatever, but they are surprisingly small compared to what Facebook or Google are able to deal with. more servers would probably permit Wikimedia projects to deal with any imaginable amount of content [05:45:15] yet we have to manage it by the communities, which is the real bottleneck I guess [05:50:44] fwiw, paws isn't spawning new k8s pods - but is letting existing users with pods preexisting through [05:52:12] madhuvishy: thanks for looking. I was trying not to ping you in this late [05:52:40] i got the help ping :) no worries, i just happened to be online [05:52:50] * bd808 goes to find sleep himself [05:52:59] cya! :) [06:03:22] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3374039 (10jcrespo) > I propose Monday after ops meeting (17:00 UTC). I am sorry, but that is outside of my working hours. [06:36:05] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:41:03] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:46:51] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3374077 (10madhuvishy) @jcrespo no problem! let me know what time works for you :) I can do earlier on Monday too. Would 14:00 UTC work? Feel free to propose a suitable time if not. Tha... [07:00:48] MisterSynergy: try now? [07:01:09] (spawning server on paws) [07:04:46] yes, it is up again. Thanks a lot! [07:06:03] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:43] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:21:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [07:52:07] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3374119 (10jcrespo) So there are several things here- the dns change and the actual reboot. There should be a time between them. I say you (as in, anyone on your team) change dns day 1... [07:58:41] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [13:11:01] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3374699 (10DatGuy) Should I restart them, or will None be automatically changed to the toolname? [13:49:40] PROBLEM - Puppet errors on tools-worker-1022 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:29:39] RECOVERY - Puppet errors on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [14:35:00] Hello all, I seem to be having trouble on one of my instances with NFS [14:35:11] nfs-tools-project.svc.eqiad.wmnet:/project/fastcci/project is hanging as it seems [14:57:45] !log nfs-tools-project.svc.eqiad.wmnet [14:58:32] !help NFS is super slow on my instance (fastcci-worker1) [14:58:32] dschwen: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [15:04:09] holy cr@p, why is it so hard to log in to phabricator all of a sudden. I cannot us mt wikimedia unified login (both the username as well as teh email already exist), and I cannot use the ldap login, because a user with the same email already exists. I'm deadlocked [15:07:29] Could that be because of a recent change in phabricator? [15:07:34] Was deployed last night. [15:09:16] (looks like there was no phab update, but i think there was an update to the extension.) [15:11:04] argh... [15:11:36] It only offers me to register accounts, but both my email as well as my username are already associated with an existing account [15:11:41] dschwen: I'll see if I can spot anything with your NFS mount [15:11:52] thx [15:12:33] Could you file a task under tag phabricator please. [15:12:48] haha [15:12:50] for this, mmodell will be able to take a look when ever he can (if you file one) [15:12:58] me? [15:13:02] yep [15:13:07] about not being to log into phabricator? [15:13:10] yep [15:13:11] "If you don't get this email please send me an email" :) [15:13:20] lol [15:13:30] anon's can't create tickets paladox [15:13:36] ah [15:13:38] lol woops [15:13:50] :-) [15:13:52] I will file it on his behave then [15:13:58] thanks [15:14:40] What's your username on phab? [15:15:46] dschwen ^^ [15:16:56] dschwen (?) [15:17:04] same as my ldap [15:17:08] that's not comming up in auto search on phab [15:17:18] typing @dsc brings up nothing. [15:17:22] email on the account is wikimedia@schwen.de [15:17:28] https://phabricator.wikimedia.org/p/dschwen/ [15:18:03] thanks [15:18:11] dschwen (dschwen) ♪♫♬ [15:18:45] ah [15:18:50] the task form is broken [15:19:09] it is comming up with phid-* instead of project name now. [15:19:50] https://phab-01.wmflabs.org/T193 [15:19:53] ah [15:19:54] that [15:19:56] woops [15:20:02] i filled it in the wrong place [15:20:03] lol [15:20:40] https://phabricator.wikimedia.org/T168731 [15:21:25] thx [15:21:37] by the way, when did that rebrand labs -> cloud happen? [15:21:51] geez, am I completely out of touch? [15:22:16] "no, it's the kids that are wrong" - Principal Skinner [15:22:58] happened a few weeks ago and still ongoing :) [15:23:21] see labs mailing list [15:24:44] dschwen: I'm not seeing anything that looks particularly horrible for NFS traffic on your server. tests writing to the share with dd seem to have decent throughput [15:24:57] do you have a particular test case that shows the problems? [15:25:29] Hm, the fastcci server was unable to start up for the past 1.5 days [15:25:35] due to NFS timeouts [15:25:43] I rebooted teh instance an hour ago [15:25:49] which fixed teh timeout [15:26:08] but going into directories and loading the 1gig database file was very slow [15:26:34] dschwen: [offtopic] Hi! If you'd like to sort out logging into Phab with LDAP, I'd be around in #wikimedia-devtools for that - see https://phabricator.wikimedia.org/T168731#3375010 [15:26:36] I should probaly put that on a local disk :-/ [15:26:43] so reads rather than writes [15:27:19] reads, yes, lots [15:27:25] do you load balance across multiple hosts? If not local disk will be much faster for sure [15:27:44] my code mmaps that file, so initial reads are very slow, until the OS caches stuff [15:28:12] The file is generated on a different host, and I anticipated having more workers, but I can work around this [15:28:26] sure. streaming 1G from the NFS server is probably never going to be really fast [15:29:17] our NFS servers are better than they were, but NFS has lots of inherent issues with as many clients as we have [15:29:35] "someday" we will have a better durable storage system [15:30:18] right now we don't even have a plan for that though [15:36:17] yeah, I know about NFS in general [15:36:28] Ok, let me investigate local FS [15:36:35] for now it seems to be up and running [15:37:24] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3375029 (10bd808) >>! In T168653#3374699, @DatGuy wrote: > Should I restart them, or will None be automatically changed to the toolna... [15:47:39] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3375067 (10MusikAnimal) I can attest to Sam's claims. There's more cleanup to be done but we've come a long way and things are more testable. Now, on to writing those tests!... [16:20:41] PROBLEM - Puppet errors on tools-worker-1022 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:31:50] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375142 (10MusikAnimal) Here's what I've got for a splash page: https://xtools-dev.wmflabs.org/ Hopefully it doesn't look that stupid... (I love the logo, by the way, that's not the stupid part!... [16:33:31] 10Labs, 10Patch-For-Review, 10cloud-services-team (Kanban): labmon1001 disk filling up - https://phabricator.wikimedia.org/T168344#3375158 (10Andrew) 05Open>03Resolved a:03Andrew # du -h -d0 /srv/carbon/whisper/archived_metrics/ 897G /srv/carbon/whisper/archived_metrics/ [16:39:33] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375168 (10Niharika) >>! In T167345#3375142, @MusikAnimal wrote: > Here's what I've got for a splash page: https://xtools-dev.wmflabs.org/ Hopefully it doesn't look that stupid... (I love the lo... [16:42:24] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375176 (10MusikAnimal) >>! In T167345#3375168, @Niharika wrote: > > How about making the logo a little smaller? I'd prefer it all fit in one screen instead of having to scroll. What device/di... [16:43:49] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375182 (10MusikAnimal) >>! In T167345#3375176, @MusikAnimal wrote: > ... assuming there is some standard ratio (browser's width is somewhat greater than height). For desktop, I mean. On mobile... [16:52:09] 10PAWS: PAWS shows “502 Bad Gateway” on server start - https://phabricator.wikimedia.org/T168702#3375188 (10madhuvishy) 05Open>03Resolved a:03madhuvishy Fixed this last night with a restart of the hub and proxy pods in the paws tool. [16:52:40] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375191 (10Niharika) MacBook 15". Here's a screenshot of what I see: {F8515866} I like the logo but I don't see a point to having it be that huge instead of in a corner like in Copypatrol. [16:55:41] RECOVERY - Puppet errors on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [17:15:47] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375252 (10MusikAnimal) >>! In T167345#3375191, @Niharika wrote: > I like the logo but I don't see a point to having it be that huge instead of just in a corner like in Copypatrol. Well, unlik... [17:24:01] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375272 (10MusikAnimal) Actually the logo was smaller but I broke the CSS rule. Try it now. Still a little flashy but I personally like that. The thing I was more unsure about were the buttons a... [18:15:11] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375406 (10DannyH) That logo is crazy big. It's a nice logo, but what you really want people to focus on are the tools. What do you think about this? {F8516200} [18:15:20] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375407 (10DannyH) Also: I thought it was XTools? [18:34:02] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:42:49] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375455 (10MusikAnimal) What about this: {F8516272} I downsized it a bit more. I don't think that's too obtrusive? I couldn't get it to look right with the logo on the left, and with the Bootstr... [19:06:49] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:07:19] PROBLEM - Puppet errors on tools-worker-1009 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:09:01] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [20:19:34] !log tools Dropped garbage indexes in elasticsearch cluster [20:19:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:20:38] !log tools Reindexing various elasticsearch indexes created before we upgraded to v2.x [20:20:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:51:39] PROBLEM - Puppet errors on tools-worker-1022 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:53:23] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Fix "Notice: Undefined index: allusers" in Adminstats when the wiki is unreachable - https://phabricator.wikimedia.org/T165707#3375748 (10Matthewrbowker) https://github.com/x-tools/xtools-rebirth/pull/36 [21:19:40] 10MediaWiki-extensions-OpenStackManager, 10Wikimedia-log-errors: PHP Fatal Error: Call to a member function doLogout() on a non-object - https://phabricator.wikimedia.org/T168750#3375769 (10hashar) [21:26:42] RECOVERY - Puppet errors on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [21:28:57] !log git upgrading gerrit-mysql to stretch (jessie -> stretch) sed -i 's/jessie/stretch/g' /etc/apt/sources.list apt-get update apt-get upgrade apt-get dist-upgrade [21:28:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [21:58:31] !log git restarting gerrit-mysql, will startup as stretch [21:58:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [22:07:43] (03Draft1) 10Paladox: Fix support for stretch [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361183 [22:07:45] (03PS2) 10Paladox: Fix support for stretch [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361183 [22:07:47] (03CR) 10Paladox: [V: 032 C: 032] Fix support for stretch [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361183 (owner: 10Paladox) [22:33:14] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3375995 (10kaldari) The logo still looks pretty huge to me. Can we shrink it a bit more? Also, as long as I'm bike-shedding... I think I would prefer the tool listings without the icons and with... [23:13:29] (03Draft1) 10Paladox: Disable letsecnrypt temp [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361189 [23:13:31] (03PS2) 10Paladox: Disable letsecnrypt temp [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361189 [23:13:34] (03CR) 10Paladox: [V: 032 C: 032] Disable letsecnrypt temp [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361189 (owner: 10Paladox) [23:17:46] PROBLEM - Puppet errors on tools-docker-builder-05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:52:47] RECOVERY - Puppet errors on tools-docker-builder-05 is OK: OK: Less than 1.00% above the threshold [0.0]