[11:15:35] <mutante>	 hi. i'll remove a few more old appservers
[11:21:28] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10ops-monitoring-bot) Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 4 host(s) and their services with reason: decom ` mw[1232-1235].eqiad.wmnet `
[11:22:20] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10ops-monitoring-bot) Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 4 host(s) and their services with reason: decom ` mw[1250-1253].eqiad.wmnet `
[11:33:49] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw[1232-1235].eqiad.wmnet` -  mw1232.eqiad.wmnet (**PASS**)   - Downtimed host on Icinga...
[11:39:38] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw[1250-1253].eqiad.wmnet` -  mw1250.eqiad.wmnet (**PASS**)   - Downtimed host on Icinga...
[12:23:39] <wikibugs>	 10serviceops, 10MediaWiki-JobQueue, 10WMF-JobQueue, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: Enable MW REST API on job runners and video scalers (for the new rest.php job executor) - https://phabricator.wikimedia.org/T246389 (10hnowlan) One concern around harmonisation of c...
[14:50:07] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet - https://phabricator.wikimedia.org/T228924 (10Dzahn) @akosiaris fix for partman? https://gerrit.wikimedia.org/r/c/operations/puppet/+/576887
[15:18:25] <mutante>	 could somebody review https://gerrit.wikimedia.org/r/c/operations/puppet/+/574902 ?  it might be trivial but the question to confirm is "when using the canary role in site.pp they also need to be listed as such in dsh.yaml"
[15:18:47] <mutante>	 (codfw canaries that i added a while ago)
[15:31:07] <wikibugs>	 10serviceops, 10Operations, 10Continuous-Integration-Infrastructure (phase-out-jessie): Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10Dzahn) >>! In T226236#5505636, @MoritzMuehlenhoff wrote: >>>! In T226236#5505608, @hashar wrote: >> Anywa,y I am declini...
[18:59:14] <wikibugs>	 10serviceops, 10Release-Engineering-Team: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Jdforrester-WMF) p:05Triage→03High
[19:01:13] <wikibugs>	 10serviceops, 10Analytics, 10Event-Platform, 10Patch-For-Review, 10Wikimedia-production-error: Lots of "EventBus: Unable to deliver all events" - https://phabricator.wikimedia.org/T247484 (10Ottomata) Checking in, how goes?
[19:29:57] <wikibugs>	 10serviceops, 10Release-Engineering-Team: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Dzahn) a:03Dzahn
[19:33:27] <wikibugs>	 10serviceops, 10Release-Engineering-Team: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Dzahn) should have been removed by this change:  https://gerrit.wikimedia.org/r/c/operations/puppet/+/583114/2/conftool-data/node/eqiad.yaml  conftool generates dsh groups.. unl...
[22:35:46] <wikibugs>	 10serviceops, 10Operations, 10Prod-Kubernetes: `helmfile --interactive apply` logs to SAL even if cancelled - https://phabricator.wikimedia.org/T248523 (10RLazarus) p:05Triage→03Low
[23:48:45] <wikibugs>	 10serviceops, 10Release-Engineering-Team: mw1251 down (no ssh) but still in dsh group? - https://phabricator.wikimedia.org/T248501 (10Catrope) This is still broken, and was causing confusion during the 4pm SWAT deployment. Thankfully scap appears to route around broken proxies, so it didn't fail to sync 1/9th...