[00:04:57] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers(mw2158 through mw2172) in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10ops-monitoring-bot) Icinga downtime for 12:00:00 set by dzahn@cumin1001 on 15 host(s) and thei... [00:10:33] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers(mw2158 through mw2172) in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Dzahn) >>! In T247018#5966345, @Dzahn wrote: > mw2158 through mw2172 are permanently depooled... [12:21:50] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10akosiaris) 05Open→03Stalled >>! In T242861#5941623, @Ottomata wrote: > @akosiaris for my purposes I'm satisfied, but I'm not sure... [13:30:06] 10serviceops, 10Analytics, 10Event-Platform, 10Patch-For-Review, 10Wikimedia-production-error: Lots of "EventBus: Unable to deliver all events" - https://phabricator.wikimedia.org/T247484 (10akosiaris) At 12:16 https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/579531/ was merged, rolling b... [13:33:52] 10serviceops, 10Graphoid, 10Operations, 10Core Platform Team (Icebox): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10AMooney) [13:34:09] 10serviceops, 10Operations, 10Release Pipeline, 10CPT Initiatives (RESTBase Split (CDP2)), and 4 others: Deploy the RESTBase front-end service (RESTRouter) to Kubernetes - https://phabricator.wikimedia.org/T223953 (10AMooney) [15:38:32] oh no [15:57:11] 10serviceops, 10Analytics, 10Event-Platform, 10Patch-For-Review, 10Wikimedia-production-error: Lots of "EventBus: Unable to deliver all events" - https://phabricator.wikimedia.org/T247484 (10akosiaris) p:05High→03Medium We are going to leave it like this for the weekend. codfw mw hosts talking to loc... [17:14:40] 10serviceops, 10Core Platform Team, 10WMF-JobQueue: RunSingleJob.php timeout too low at 180 seconds - https://phabricator.wikimedia.org/T247622 (10Krinkle) [17:15:13] 10serviceops, 10Core Platform Team, 10WMF-JobQueue: RunSingleJob.php timeout too low at 180 seconds - https://phabricator.wikimedia.org/T247622 (10Krinkle) p:05Triage→03Unbreak! [17:33:02] 10serviceops, 10Core Platform Team, 10WMF-JobQueue: RunSingleJob.php timeout too low at 180 seconds - https://phabricator.wikimedia.org/T247622 (10Joe) Nothing regressed, we just never had jobs that executed php code for more than 180 seconds. max_execution_time in fact doesn't count time spent in syscalls,... [17:43:37] 10serviceops, 10Core Platform Team, 10WMF-JobQueue: RunSingleJob.php timeout too low at 180 seconds - https://phabricator.wikimedia.org/T247622 (10Joe) p:05Unbreak!→03High While this needs fixing quickly, it's not a regression and we can wait monday to manage the rollout of the fix and any unintended con... [18:50:05] 10serviceops, 10Deployments, 10Patch-For-Review, 10Performance-Team (Radar), and 3 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10thcipriani) FWIW, ran some tests with apachebench today and the patchset on this task and it didn't seem to ma... [20:49:13] James_F: i updated https://gerrit.wikimedia.org/r/c/operations/puppet/+/576493 from Horizon just now [20:50:13] i'll merge that and then we can remove it in the web UI [20:52:11] Brill. [20:59:24] i guess adding "role::beta::mediawiki" could not be avoided [21:12:13] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforre... [21:12:18] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforre... [23:55:55] 10serviceops, 10Operations: upgrade people.wikimedia.org backend to buster - https://phabricator.wikimedia.org/T247649 (10Dzahn) [23:56:02] 10serviceops, 10Operations: upgrade people.wikimedia.org backend to buster - https://phabricator.wikimedia.org/T247649 (10Dzahn) a:03Dzahn [23:56:24] 10serviceops, 10Operations: miscweb1001/2001 - upgrade to buster or decom - https://phabricator.wikimedia.org/T247648 (10Dzahn) [23:59:50] 10serviceops, 10Operations: replace bromine and vega with buster VMs - https://phabricator.wikimedia.org/T247650 (10Dzahn) [23:59:56] 10serviceops, 10Operations: replace bromine and vega with buster VMs - https://phabricator.wikimedia.org/T247650 (10Dzahn) a:03Dzahn