[05:19:29] 10serviceops, 10Operations, 10TechCom-RFC, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) [08:21:06] 10serviceops: Plan to scale up termbox service to be able to render the termbox for desktop pageviews - https://phabricator.wikimedia.org/T261486 (10Addshore) [08:22:56] 10serviceops, 10Operations, 10Wikidata, 10Wikidata-Termbox, 10User-Addshore: Plan to scale up termbox service to be able to render the termbox for desktop pageviews - https://phabricator.wikimedia.org/T261486 (10Addshore) [08:38:31] 10serviceops, 10Patch-For-Review: Sporadic issues on helm dependency build in CI - https://phabricator.wikimedia.org/T261313 (10JMeybohm) 05Open→03Resolved a:03JMeybohm Should be fixed now. [08:38:34] 10serviceops, 10Prod-Kubernetes, 10Release Pipeline, 10Patch-For-Review: Refactor our helmfile.d dir structure for services - https://phabricator.wikimedia.org/T258572 (10JMeybohm) [13:34:50] 10serviceops, 10Operations, 10Release-Engineering-Team, 10Scap, 10Platform Team Workboards (Clinic Duty Team): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10thcipriani) Scap3 was built as a general deployment tool, and should be able to deploy PHP code (I... [13:40:32] 10serviceops, 10Operations, 10Release-Engineering-Team, 10Scap, 10Platform Team Workboards (Clinic Duty Team): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Joe) @thcipriani this is a service that will have to be deployed to kubernetes. So I think the actu... [13:48:38] 10serviceops, 10Operations, 10Release-Engineering-Team, 10Platform Team Workboards (Clinic Duty Team), 10Release Pipeline (Blubber): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Joe) [16:00:18] mutante: nothing still ongoing with codfw appservers, right? volans and I might test out the warmup script [16:00:56] rzl: correct. nothing ongoing with the exceptation that i want to decom the very last old server standing because it's an mcrouter proxy [16:01:03] exception [16:01:42] _joe_: was there anything else needed besides editing mcrouter_wancache.yaml to switch an mcrouter proxy to a new machine/rack ? https://gerrit.wikimedia.org/r/c/operations/puppet/+/622900 [16:01:50] mutante: cool -- mind holding off for an hour or so? I can ping you when we're done [16:02:08] sure, no problem at all. holding off [16:02:14] much obliged [16:02:59] (I think j.oe is off today, not sure) [16:03:06] ack [16:04:28] got the answer from elukey. looks good to go(later) [16:40:12] mutante: all clear, thanks! [16:46:48] rzl: thanks, switching that mcrouter proxy [17:34:51] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 4 others: Drop official PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Jdforrester-WMF) Let's do that, then. [17:39:28] 10serviceops, 10Operations, 10Patch-For-Review: Decommission mw2135-mw2147, mw2187-mw2199, mw2200-mw2214 (all PowerEdge R420) - https://phabricator.wikimedia.org/T260654 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2196.codfw.wmnet` - mw2196.codfw.wmnet (*... [17:41:34] 10serviceops, 10Operations, 10Patch-For-Review: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) - https://phabricator.wikimedia.org/T260654 (10Dzahn) [17:43:55] 10serviceops, 10Operations, 10Patch-For-Review: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) - https://phabricator.wikimedia.org/T260654 (10Dzahn) [17:45:37] 10serviceops, 10Operations, 10Wikidata, 10Wikidata-Termbox, 10User-Addshore: Plan to scale up termbox service to be able to render the termbox for desktop pageviews - https://phabricator.wikimedia.org/T261486 (10jijiki) p:05Triage→03Medium [17:45:41] 10serviceops, 10Operations, 10Patch-For-Review: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) - https://phabricator.wikimedia.org/T260654 (10Dzahn) [17:45:44] 10serviceops, 10Operations: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) - https://phabricator.wikimedia.org/T247021 (10Dzahn) [17:49:29] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 4 others: Drop official PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Reedy) I had another look (yesterday was a long day)... https://cve.mitre.org/cgi-bin/cvename.cgi?name=... [19:17:19] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Dzahn) [19:21:45] 10serviceops, 10Operations, 10Patch-For-Review: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) - https://phabricator.wikimedia.org/T260654 (10Dzahn) 05Open→03Resolved [19:21:47] 10serviceops, 10Operations: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) - https://phabricator.wikimedia.org/T247021 (10Dzahn) [19:22:11] 10serviceops, 10Operations, 10Patch-For-Review: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) - https://phabricator.wikimedia.org/T260654 (10Dzahn) All steps that need to be done by us as the service owner are done. 40 hosts removed from repos, shutdown and wiped bootloader by decom cookboo... [19:22:38] rzl: effie: decom complete on our side, servers have wiped bootloaders and prod IPs. I made a new ticket for dcops because that needs 40 x the template, which means 680 checkboxes.. i scripted that :p [19:27:32] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Dzahn) The shorter version to say this is: "all R420's in codfw that have an mw* name and are already in state 'decommissioning' in netbox' The tota... [19:59:53] <_joe_> rzl: re the dns issues, I think we should just raise the retry time [20:00:12] <_joe_> so that it waits ~ 10 seconds before failing