[01:07:18] (03CR) 10Legoktm: [C: 04-1] "Yes, you need a phpcs.xml file. See for instructions on how to s" (031 comment) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (owner: 10MarcoAurelio) [01:18:08] 10Cloud-VPS (Project-requests), 10User-Zppix: Request creation of Zppix-Wiki-AI VPS project - https://phabricator.wikimedia.org/T175846#3630620 (10Andrew) > which due to requirements of mw-vagrant isnt possible for me Can you elaborate? Are you on a chromebook for example? [01:30:17] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:59:50] 10VPS-Projects, 10Social-Tools: social-tools.wmflabs.org is running MediaWiki v1.28.0 - https://phabricator.wikimedia.org/T174958#3630624 (10Legoktm) a:05Legoktm>03ashley OK, I created a new server at http://social-tools3.wmflabs.org/wiki/Special:Version and explained to @ashley where everything is and how... [02:35:18] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:49] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [04:25:52] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [04:27:03] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:07:04] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [06:17:35] 10Cloud-VPS (Project-requests): Request creation of webperf VPS project - https://phabricator.wikimedia.org/T176597#3630730 (10Peter) [06:41:18] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:16:15] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [08:11:57] (03PS2) 10Lokal Profil: Add default instructions to top of unused images reports [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/380058 [08:13:46] (03CR) 10jerkins-bot: [V: 04-1] Add default instructions to top of unused images reports [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/380058 (owner: 10Lokal Profil) [08:30:22] PROBLEM - Puppet errors on tools-worker-1016 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:51:35] (03CR) 10Jean-Frédéric: [C: 032] Add monuments_config for Australia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375537 (https://phabricator.wikimedia.org/T174333) (owner: 10Lokal Profil) [08:52:42] (03CR) 10Jean-Frédéric: [C: 032] "Oh man. I checked that locally to test it and spent 10 minutes debugging why it was exiting instantly with no output. Until I realise the " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375537 (https://phabricator.wikimedia.org/T174333) (owner: 10Lokal Profil) [08:54:23] (03Merged) 10jenkins-bot: Add monuments_config for Australia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375537 (https://phabricator.wikimedia.org/T174333) (owner: 10Lokal Profil) [08:55:21] (03CR) 10jenkins-bot: Add monuments_config for Australia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375537 (https://phabricator.wikimedia.org/T174333) (owner: 10Lokal Profil) [09:10:22] RECOVERY - Puppet errors on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [09:49:08] !log tools.heritage Deploy latest from Git master: 1ab75f9 (T174333) [09:49:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [09:49:12] T174333: Add Australia to Monuments Database - https://phabricator.wikimedia.org/T174333 [09:51:59] PROBLEM - High iowait on tools-exec-1420 is CRITICAL: CRITICAL: tools.tools-exec-1420.cpu.total.iowait (>11.11%) [10:01:59] RECOVERY - High iowait on tools-exec-1420 is OK: OK: All targets OK [10:21:18] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [10:36:16] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [10:53:29] (03CR) 10MarcoAurelio: ">" (031 comment) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (owner: 10MarcoAurelio) [10:55:40] PROBLEM - SSH on tools-exec-1420 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:44] (03PS5) 10MarcoAurelio: Update composer.json to add more tests and dependencies [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 [10:58:03] (03CR) 10jerkins-bot: [V: 04-1] Update composer.json to add more tests and dependencies [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (owner: 10MarcoAurelio) [10:58:44] (03PS6) 10MarcoAurelio: Update composer.json to add more tests and dependencies [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 [10:59:08] (03CR) 10jerkins-bot: [V: 04-1] Update composer.json to add more tests and dependencies [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (owner: 10MarcoAurelio) [11:00:31] RECOVERY - SSH on tools-exec-1420 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 (protocol 2.0) [11:04:21] (03CR) 10MarcoAurelio: [C: 04-2] "So I don't mess up. This needs a lot of work :(" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (owner: 10MarcoAurelio) [11:04:28] 10Cloud-VPS, 10Operations-Software-Development: Cumin: fine tune WMCS setup - https://phabricator.wikimedia.org/T176609#3631268 (10Volans) [11:07:18] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:21:42] PROBLEM - SSH on tools-exec-1420 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:03] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631300 (10MarcoAurelio) @Marostegui Wiki creation is happening now. [11:33:37] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631348 (10Marostegui) >>! In T173027#3631300, @MarcoAurelio wrote: > @Marostegui Wiki creation is happening now. I can see it is already created on the master. I will check the triggers and... [11:57:18] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631402 (10Marostegui) I have ran redact_sanitarium on db1095 and I can see that the user, revision and recentchanges tables have been sanitized correctly. I am running the check_private data... [12:46:08] (03PS7) 10MarcoAurelio: Update composer.json to add more tests and dependencies [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 [12:46:30] (03CR) 10jerkins-bot: [V: 04-1] Update composer.json to add more tests and dependencies [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (owner: 10MarcoAurelio) [12:48:57] (03CR) 10MarcoAurelio: "Run composer test and composer fix and most sniff violations seems to have been fixed. Others seems not to be automatically fixable or we " [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (owner: 10MarcoAurelio) [12:51:12] 10VPS-project-Wikistats: Add hi.wikivoyage to wikistats - https://phabricator.wikimedia.org/T173033#3631663 (10MarcoAurelio) 05stalled>03Open [12:56:30] RECOVERY - SSH on tools-exec-1420 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 (protocol 2.0) [13:00:00] PROBLEM - High iowait on tools-exec-1420 is CRITICAL: CRITICAL: tools.tools-exec-1420.cpu.total.iowait (>12.50%) [13:00:03] PROBLEM - High iowait on tools-exec-1420 is CRITICAL: CRITICAL: tools.tools-exec-1420.cpu.total.iowait (>12.50%) [13:03:52] 10Toolforge, 10Outreachy (Round-15): Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3631721 (10Sowjanyavemuri) [13:06:17] 10Toolforge, 10Outreachy (Round-15): Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3631730 (10Sowjanyavemuri) p:05Triage>03Normal [13:09:59] RECOVERY - High iowait on tools-exec-1420 is OK: OK: All targets OK [13:12:47] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631740 (10Marostegui) The report finished on labsdb1009 and I can see the new users are being sanitized correctly on all hosts. I will wait for the report to finish on labsdb1001 before handl... [13:37:16] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [13:47:17] 10cloud-services-team (Kanban), 10Operations, 10Operations-Software-Development, 10Goal, 10Technical-Debt: Remove salt master (and related packages) from labcontrol1001 - https://phabricator.wikimedia.org/T176632#3631899 (10Andrew) [13:57:24] 10Tool-stewardbots: Update composer.json to use MediaWiki CodeSniffer and fix detected issues - https://phabricator.wikimedia.org/T176635#3631963 (10MarcoAurelio) [13:58:06] (03PS8) 10MarcoAurelio: [WIP] Update composer.json to add more tests and dependencies [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (https://phabricator.wikimedia.org/T176635) [13:58:46] (03PS9) 10MarcoAurelio: [WIP] Update composer.json to use MW_CodeSniffer and fix detected issues [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (https://phabricator.wikimedia.org/T176635) [14:10:07] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update composer.json to use MW_CodeSniffer and fix detected issues [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (https://phabricator.wikimedia.org/T176635) (owner: 10MarcoAurelio) [14:47:41] PROBLEM - SSH on tools-exec-1420 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:00] scrollback indicates something is rotten w/ tools-exec-1420 :) [14:57:02] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:57:10] !log tools OS_TENANT_NAME=tools openstack server reboot 2c0cf363-c7c3-42ad-94bd-e586f2492321 (unresponsive) [14:57:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:02:31] RECOVERY - SSH on tools-exec-1420 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 (protocol 2.0) [15:12:03] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:14:37] !log tools rebooting tools-paws-worker-1006 since I can't access it [15:14:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:15:57] RECOVERY - SSH on tools-paws-worker-1006 is OK: SSH OK - OpenSSH_7.4p1 Debian-10 (protocol 2.0) [15:22:17] PROBLEM - Puppet staleness on tools-paws-worker-1006 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [43200.0] [15:27:19] RECOVERY - Puppet staleness on tools-paws-worker-1006 is OK: OK: Less than 1.00% above the threshold [3600.0] [15:31:34] 10cloud-services-team (Kanban): lots of cloud-local puppetmasters broken - https://phabricator.wikimedia.org/T176645#3632347 (10Andrew) [15:31:45] 10cloud-services-team (Kanban): lots of cloud-local puppetmasters broken - https://phabricator.wikimedia.org/T176645#3632335 (10Andrew) p:05Triage>03High [15:32:15] 10Cloud-Services, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3632349 (10Marostegui) All good on labsdb1001 as well. #cloud-services-team feel free to create the views Thanks! [16:15:26] 10cloud-services-team (Kanban): lots of cloud-local puppetmasters broken - https://phabricator.wikimedia.org/T176645#3632335 (10madhuvishy) @Andrew do we have a list of instances, they just need a `apt-get install --upgrade apache2` and restart for apache. There was an unattended apache upgrade that rolled out l... [16:37:36] PROBLEM - Puppet errors on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:58:08] 10cloud-services-team (Kanban): lots of cloud-local puppetmasters broken - https://phabricator.wikimedia.org/T176645#3632626 (10madhuvishy) I went through the list here -https://tools.wmflabs.org/openstack-browser/puppetclass/role::puppetmaster::standalone and upgraded apache and ran puppet. [17:12:39] RECOVERY - Puppet errors on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [17:12:53] !log dashiki Deleted instance vitalsigns-01 [17:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dashiki/SAL [17:20:02] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:22:51] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:55:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [17:57:50] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [18:19:36] PROBLEM - Puppet errors on tools-exec-1429 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:26:19] 10Toolforge, 10Outreachy (Round-15): Outreachy - webservice microtask for Sowjanyavemuri - https://phabricator.wikimedia.org/T176624#3632973 (10Sowjanyavemuri) [18:30:50] 10Toolforge, 10Outreachy (Round-15): Outreachy - webservice microtask for Mridu_Bhatnagar - https://phabricator.wikimedia.org/T176018#3611196 (10srishakatux) @Mridu_Bhatnagar Hello! just checking, how is your progress on the microtask going? Stuck somewhere, or have questions, then do not hesitate to ask :) [18:33:37] !log graphite moving 'grafana' back to the normal labs puppetmaster. There aren't any local changes, and puppet has been broken here for ages [18:33:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Graphite/SAL [18:49:39] !log wmt rebooting wmt-exec; it is unreachable [18:49:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmt/SAL [18:55:00] !log gitblit rebooting 'danny' instance; it is unreachable [18:55:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitblit/SAL [18:56:24] !log gitblit rebooting 'test' instance; unreachable [18:56:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitblit/SAL [18:59:35] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:17] 10VPS-project-Wikistats: Add hi.wikivoyage to wikistats - https://phabricator.wikimedia.org/T173033#3633092 (10Dzahn) a:03Dzahn [20:16:28] 10VPS-project-Wikistats: Add hi.wikivoyage to wikistats - https://phabricator.wikimedia.org/T173033#3633298 (10Dzahn) 05Open>03Resolved ``` MariaDB [wikistats]> insert into wikivoyage (lang,prefix,loclang,loclanglink,method) values ("Hindi","hi","हिन्दी","Hindi",8); Query... [20:23:26] 10VPS-project-Wikistats: Add hi.wikivoyage to wikistats - https://phabricator.wikimedia.org/T173033#3633322 (10MarcoAurelio) Thank you! [20:23:58] tgr: is the 'sentry' project defunct? There are some VMs in there with broken puppet that I'd like to get rid of (or would like you to fix) [20:24:08] 10Cloud-Services, 10cloud-services-team, 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3633325 (10MarcoAurelio) [20:30:23] andrewbogott: it's comatose, at the very least [20:30:39] what needs to be fixed? [20:30:57] puppet fails on sentry-01 and sentry-alpha [20:34:34] huh, I can't even log in to sentry-alpha [20:35:51] i think that may be because of the ldap issue a couple of months ago if puppet is not working [20:35:52] tgr ^^ [20:37:33] tgr: my root key doesn't work on sentry-alpha either, so I'd guess that puppet has been busted there for a long time. [20:38:00] andrewbogott: fyi referring to T175846 I've talked to scoring platform team and if all goes well i may end up using their testing instance they have in their vps project if so ill close the task but until then ill keep it open this that ok? [20:38:00] T175846: Request creation of Zppix-Wiki-AI VPS project - https://phabricator.wikimedia.org/T175846 [20:38:21] Zppix: sounds good, maybe mark the ticket accordingly? [20:38:38] andrewbogott: how do you want me to mark it and i will [20:38:50] Zppix: just say what you just said [20:38:53] Zppix: write that comment in the ticket :) [20:39:13] bd808: andrewbogott i thougt you meant like change the status... i was going to comment that anyway xD [20:39:16] I don't think I changed keys since last logging in there [20:39:17] i swear im blond sometimes [20:39:42] Zppix: that's not a great statement to make. [20:39:46] 10Cloud-VPS (Project-requests), 10User-Zppix: Request creation of Zppix-Wiki-AI VPS project - https://phabricator.wikimedia.org/T175846#3633377 (10Zppix) I've talked to scoring platform team and if all goes well i may end up using their testing instance they have in their vps project if so ill close the task b... [20:40:04] blonde jokes are in very poor taste and highly sexist [20:40:17] bd808: yeah i realised that as soon as i sent it :/ [20:40:20] I can still recover the machine via a Horizon console, right? [20:40:24] tgr: is there really anything on those VMs worth saving? You can always start anew when you have time to think about it... [20:40:34] I just want to check if there are any patches not in gerrit [20:40:49] tgr: I'll look [20:41:22] or amended compared to what's in gerrit [20:41:30] Zppix: thanks for recognizing it. We all make small mistakes. :) [20:41:37] I don't think sentry01 has anything of value [20:42:15] tgr: there's no diff on sentry-alpha other than a bunch of out-of-date submodules [20:42:40] wait, I take that back [20:42:45] 10Cloud-VPS (Project-requests), 10User-Zppix: Request creation of Zppix-Wiki-AI VPS project - https://phabricator.wikimedia.org/T175846#3633390 (10Zppix) >>! In T175846#3630620, @Andrew wrote: > Can you elaborate? Are you on a chromebook for example? No, my system specs do not meet the min requirements for v... [20:43:22] sentry01 is broken because it still uses the old upstartd roles [20:43:52] not hard to fix but I don't want to deal with that right now so feel free to delete it if it is causing problems [20:44:49] the beta cluster sends errors there but it uses the HTML5 beacon API so I don't think there is any harm in the box going down [20:56:16] (03PS10) 10MarcoAurelio: [WIP] Update composer.json to use MW_CodeSniffer and fix detected issues [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (https://phabricator.wikimedia.org/T176635) [20:56:40] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update composer.json to use MW_CodeSniffer and fix detected issues [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (https://phabricator.wikimedia.org/T176635) (owner: 10MarcoAurelio) [21:22:26] (03PS8) 10Lokal Profil: Group unused images per source page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/379141 (https://phabricator.wikimedia.org/T117327) [21:25:04] (03PS3) 10Lokal Profil: Add default instructions to top of unused images reports [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/380058 [21:25:23] (03CR) 10Lokal Profil: "rebased to make this behave with isort" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/379141 (https://phabricator.wikimedia.org/T117327) (owner: 10Lokal Profil) [21:42:57] bd808: is T173027 something you can do? [21:42:58] T173027: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027 [21:43:06] dba review is done [21:43:49] bd808 hi, i wonder should the task about wikitech upgrading to hhvm be declined and replaced with a php 7.0 one? [21:43:50] tabbycat: I can put it on our workboard :) [21:44:07] bd808: works for me :) [21:44:17] 10Cloud-Services, 10cloud-services-team (Kanban), 10DBA: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3633590 (10bd808) [21:45:03] !log rcm Xenon: Doing software update (Phabricator) [21:45:03] !log rcm CAC: Doing software update (vagrant git-update) [21:45:03] !log rcm Tin: Doing software update (Jenkins) [21:45:03] !log rcm Oxygen: Doing software update (Packages) [21:45:03] !log rcm Neon: Doing software update (Packages) [21:45:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:45:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:45:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:45:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:45:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [21:45:23] paladox: that all depends on timelines. I think we want to get wikitech on a Jessie host with HHVM before the world will be ready for PHP7 on Stretch. [21:46:03] ok [21:46:27] Ideally from my point of view by the time things move from HHVM to PHP7 wikitech will be in the main cluster [21:47:23] there are a lot of small steps from where we are today to there though [21:47:39] ok [21:51:00] I like the discussion about going back to php 7 [21:51:14] i think i will talk about it in the dev summit [21:51:28] (if i will come) [21:51:46] I like that PHP7 is worth switching to :) [21:51:47] *at the [21:51:55] that too [21:52:04] I'm pretty sure it will al be a done deal by January [21:52:29] there is a bigger discussion to have there though about FLOSS communities and the products we use and support [21:52:41] you think? as in an execution plan ? [21:53:01] yeah, its honestly not that hard to map out [21:53:08] i was planning to focus on those areas [21:53:28] the tech part of now or in 3 months is not so interesting [21:53:30] migrate MW servers to stretch, then switch from our HHVM build to PHP7.1 from stretch [21:53:51] i thought php 7.0 is only in stretch? [21:54:06] if php 7.1 will be there, it will allow us to migrate phabricator too :). [21:54:31] php7.0 is in mainline. I expect 7.1 to land in backports at some point [21:54:59] I haven't looked to see if anyone has posted for that in the backports bug tracker yet [21:55:06] heh, was typing backport ... [22:10:12] 10Tools, 10cloud-services-team (Kanban), 10User-bd808: Update replag tool to show per-host lag - https://phabricator.wikimedia.org/T175600#3633633 (10bd808) 05Open>03Resolved Mostly resolved. Replag now shows lag on: * wikireplica-analytics.eqiad.wmnet * wikireplica-web.eqiad.wmnet * c1.labsdb * c3.labsdb [22:57:25] 10Striker: Striker should not allow tool names to include '_' for Kubernetes compatibility - https://phabricator.wikimedia.org/T176681#3633708 (10bd808) [23:05:02] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:15:38] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Investigate and implement alternative for showmount based check at instance boot time - https://phabricator.wikimedia.org/T171508#3633757 (10madhuvishy) Thanks @MoritzMuehlenhoff, I tried that and came up with these two plots, don't really... [23:32:58] 10Data-Services, 10cloud-services-team (Kanban), 10DBA: Create and announce timeline for shutting down labsdb100[13] - https://phabricator.wikimedia.org/T175086#3633776 (10bd808) @jcrespo has suggested Wednesday 2017-12-13 as the target shutdown date for the servers. We also need to choose a date sometime in... [23:44:58] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [23:46:14] 10Data-Services, 10cloud-services-team (FY2017-18), 10DBA, 10Goal: Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#3633796 (10bd808) [23:46:17] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Promote initial use of new Wiki Replica servers - https://phabricator.wikimedia.org/T172704#3633794 (10bd808) 05Open>03Resolved * https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_servers_ready_for_use/ * https://lists.wik... [23:47:17] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10User-bd808: Create and announce timeline for shutting down labsdb100[13] - https://phabricator.wikimedia.org/T175086#3633797 (10bd808) a:03bd808 [23:51:42] 10VPS-project-Wikistats: Miraheze wikistats new wikis not updating - https://phabricator.wikimedia.org/T176535#3633811 (10Dzahn) Thanks for reporting this. I did have a cron to automate this too, but apparently it fails then. I'll take a look.