[07:29:42] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Update wikikube codfw to kubernetes 1.31 - https://phabricator.wikimedia.org/T397148#10926714 (10JMeybohm) [10:23:05] 06serviceops, 10ChangeProp, 10Page Content Service, 13Patch-For-Review: Investigate significant changeprop backlogs after PCS migration - https://phabricator.wikimedia.org/T397072#10927354 (10hnowlan) `topic: eqiad.change-prop.transcludes.resource-change group: changeprop-pcs_rerender_mobile_html_native_tr... [10:25:59] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Update wikikube codfw to kubernetes 1.31 - https://phabricator.wikimedia.org/T397148#10927371 (10JMeybohm) Unfortunately the pool-depool-cluster cookbook immediately fails: ` jayme@cumin1002:~$ sudo cookbook sre.k8s.pool-depool-cluster -r pre-upgrade-test depool... [11:26:21] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#10927634 (10taavi) [11:30:41] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#10927696 (10MoritzMuehlenhoff) [13:24:43] 06serviceops, 06Data-Engineering, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#10928172 (10HCoplin-WMF) p:05High→03Low Updating the priority to low. We are curren... [13:34:12] 06serviceops, 06Data-Engineering, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#10928208 (10hnowlan) I can't say with certainty, but there is a reasonable chance that... [14:31:36] 06serviceops, 06Abstract Wikipedia team, 07Essential-Work: Wikifunctions orchestrator service in staging k8s cannot make network calls, gets getaddrinfo EAI_AGAIN / no healthy upstream - https://phabricator.wikimedia.org/T397341 (10Jdforrester-WMF) 03NEW [14:52:03] 06serviceops, 06Commons, 06MW-Interfaces-Team, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10928576 (10akosiaris) 05Open→03Resolved a:03akosiaris >>! In T380544#10924765, @LucasWerkmeister wrote: > Overall, I’m inclined to say that... [15:33:59] 06serviceops, 06Abstract Wikipedia team, 07Essential-Work: Wikifunctions orchestrator service in staging k8s cannot make network calls, gets getaddrinfo EAI_AGAIN / no healthy upstream - https://phabricator.wikimedia.org/T397341#10928740 (10Jdforrester-WMF) p:05Triage→03High [15:34:33] 06serviceops, 06Abstract Wikipedia team, 07Essential-Work, 07Wikimedia-production-error: Wikifunctions orchestrator service in staging k8s cannot make network calls, gets getaddrinfo EAI_AGAIN / no healthy upstream - https://phabricator.wikimedia.org/T397341#10928745 (10Jdforrester-WMF) Production wiring/c... [15:34:56] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Tidy helm app chart modules for function-orchestrator in deployment-charts - https://phabricator.wikimedia.org/T395254#10928750 (10Jdforrester-WMF) [15:34:57] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10928751 (10Jdforrester-WMF) [15:35:08] 06serviceops, 06Abstract Wikipedia team, 10function-orchestrator: Tidy helm app chart modules for function-orchestrator in deployment-charts - https://phabricator.wikimedia.org/T395254#10928754 (10Jdforrester-WMF) [16:16:00] jayme: I assume applying the cfssl-issuer-0.4.4 bump in admin_ng is okay? [16:16:27] hnowlan: I hope it's staging?! [16:16:48] ah, no...thats probably in prod as well...but no diff apart from the version, right? [16:17:09] just the chart [16:17:33] hnowlan: yeah, it's fine. Sorry - I forgot to ensure it's deployed everywhere [16:17:33] chart version that is. cfssl-issuer-0.4.3->cfssl-issuer-0.4.4 [16:17:38] cool [17:50:32] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install rdb201[12] - https://phabricator.wikimedia.org/T393121#10929271 (10Jhancock.wm) a:03akosiaris [17:51:19] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install rdb201[12] - https://phabricator.wikimedia.org/T393121#10929274 (10Jhancock.wm) @akosiaris can you add these two servers to site.pp for me please? i saw they're already covered in preseed. Should be able to hand these over to you pretty quic... [18:20:58] 06serviceops, 06Commons, 06MW-Interfaces-Team, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10929363 (10LucasWerkmeister) I guess the most likely explanation is that the processing of the updates for MariaDB and OpenSearch happens separat... [18:26:57] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade httpd images to bullseye or bookworm - https://phabricator.wikimedia.org/T378128#10929380 (10Scott_French) After no issues were uncovered for shellbox-syntaxhighlight with ~ 24h on the new images, the remaining (5) shellbox instances hav... [19:30:51] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host aux-k8s-worker2006.codfw.wmnet with OS bookworm [19:31:07] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929506 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host aux-k8s-worker2007.codfw.wmnet with OS bookworm [19:31:21] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929510 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host aux-k8s-worker2008.codfw.wmnet with OS bookworm [19:31:37] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929511 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host aux-k8s-worker2009.codfw.wmnet with OS bookworm [20:03:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host aux-k8s-worker2006.codfw.wmnet with OS bookworm comple... [20:07:22] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929637 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host aux-k8s-worker2009.codfw.wmnet with OS bookworm comple... [20:11:12] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929649 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host aux-k8s-worker2007.codfw.wmnet with OS bookworm comple... [20:13:17] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929661 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host aux-k8s-worker2008.codfw.wmnet with OS bookworm comple... [20:13:30] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929662 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [20:13:46] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install aux-k8s-worker200[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T393054#10929666 (10Jhancock.wm) @akosiaris this one is complete [21:49:22] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install aux-k8s-worker100[6-9].eqiad.wmnet - https://phabricator.wikimedia.org/T393053#10930054 (10Jclark-ctr) a:03Jclark-ctr [22:57:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install aux-k8s-worker100[6-9].eqiad.wmnet - https://phabricator.wikimedia.org/T393053#10930176 (10Jclark-ctr)