[00:00:05] Deploy window Web Team deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T0000) [00:05:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1053:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1053 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [00:08:13] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on restbase1035 - https://phabricator.wikimedia.org/T413678#11494323 (10Jclark-ctr) Service request with Dell 220855023 [00:14:09] FIRING: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate config-master.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [00:22:51] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1168 - https://phabricator.wikimedia.org/T413704#11494389 (10Jclark-ctr) This server is out of warranty Please advise if you want to replace drive. Replacement servers have been racked cabled and handed over already T407032. [00:23:58] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on aqs1015 - https://phabricator.wikimedia.org/T413559#11494394 (10Jclark-ctr) This server is out of warranty Please advise if you want to replace drive. Replacement servers have been racked cabled and handed over already T407032. [00:24:01] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on aqs1015 - https://phabricator.wikimedia.org/T413559#11494396 (10Jclark-ctr) a:03Jclark-ctr [00:24:02] (03PS1) 10Arlolra: Deploy PRV to 27 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223283 (https://phabricator.wikimedia.org/T413108) [00:28:15] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1168 - https://phabricator.wikimedia.org/T413704#11494405 (10Jclark-ctr) Service Request 220855299 [00:40:08] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1223286 [00:40:08] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1223286 (owner: 10TrainBranchBot) [00:51:28] (03PS1) 10Zabe: Enable phan on more php files [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223289 [00:53:38] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1223286 (owner: 10TrainBranchBot) [01:00:41] !log mwpresync@deploy2002 Started scap build-images: Publishing wmf/next image [01:09:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1223293 [01:09:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1223293 (owner: 10TrainBranchBot) [01:13:14] !log mwpresync@deploy2002 Finished scap build-images: Publishing wmf/next image (duration: 12m 32s) [01:13:57] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [01:23:57] RECOVERY - Check unit status of statograph_post on alert1002 is OK: OK: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [01:24:09] FIRING: KubernetesCalicoDown: ml-serve2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-mlserve&var-instance=ml-serve2004.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [01:36:30] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1223293 (owner: 10TrainBranchBot) [01:55:16] (03PS1) 10Cwhite: icinga: restore parameters to fix puppet [puppet] - 10https://gerrit.wikimedia.org/r/1223302 (https://phabricator.wikimedia.org/T413842) [02:00:04] (03CR) 10Cwhite: [C:03+2] "PCC OK: https://puppet-compiler.wmflabs.org/output/1223302/7851/" [puppet] - 10https://gerrit.wikimedia.org/r/1223302 (https://phabricator.wikimedia.org/T413842) (owner: 10Cwhite) [02:09:43] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.46.0-wmf.10 [core] (wmf/1.46.0-wmf.10) - 10https://gerrit.wikimedia.org/r/1223305 (https://phabricator.wikimedia.org/T408280) [02:09:45] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.46.0-wmf.10 [core] (wmf/1.46.0-wmf.10) - 10https://gerrit.wikimedia.org/r/1223305 (https://phabricator.wikimedia.org/T408280) (owner: 10TrainBranchBot) [02:22:32] (03Merged) 10jenkins-bot: Branch commit for wmf/1.46.0-wmf.10 [core] (wmf/1.46.0-wmf.10) - 10https://gerrit.wikimedia.org/r/1223305 (https://phabricator.wikimedia.org/T408280) (owner: 10TrainBranchBot) [03:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T0300) [03:07:14] (03PS2) 10C. Scott Ananian: Deploy PRV to 27 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223283 (https://phabricator.wikimedia.org/T413108) (owner: 10Arlolra) [04:00:04] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T0400) [04:01:56] (03PS1) 10TrainBranchBot: testwikis to 1.46.0-wmf.10 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223315 (https://phabricator.wikimedia.org/T408280) [04:01:58] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy2002" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223315 (https://phabricator.wikimedia.org/T408280) (owner: 10TrainBranchBot) [04:02:49] (03Merged) 10jenkins-bot: testwikis to 1.46.0-wmf.10 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223315 (https://phabricator.wikimedia.org/T408280) (owner: 10TrainBranchBot) [04:03:18] !log mwpresync@deploy2002 Started scap sync-world: testwikis to 1.46.0-wmf.10 refs T408280 [04:03:21] T408280: 1.46.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T408280 [04:14:09] FIRING: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate config-master.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [04:47:34] !log mwpresync@deploy2002 Finished scap sync-world: testwikis to 1.46.0-wmf.10 refs T408280 (duration: 44m 16s) [04:47:37] T408280: 1.46.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T408280 [05:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T0500) [05:09:09] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:24:09] FIRING: KubernetesCalicoDown: ml-serve2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-mlserve&var-instance=ml-serve2004.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:34:09] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:21:21] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 5/6 UP : OSPFv3: 5/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:21:23] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:22:17] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 6/6 UP : OSPFv3: 6/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:23:23] RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:29:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T0700) [07:00:05] marostegui, Amir1, and federico3: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Primary database switchover deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T0700). [07:19:58] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#11494756 (10MoritzMuehlenhoff) [07:26:30] (03PS1) 10Muehlenhoff: doc: Update PHP spec file to what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 [07:26:55] (03CR) 10Muehlenhoff: doc: Remove obsolete spec test (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1223145 (owner: 10Muehlenhoff) [07:27:12] (03Abandoned) 10Muehlenhoff: doc: Remove obsolete spec test [puppet] - 10https://gerrit.wikimedia.org/r/1223145 (owner: 10Muehlenhoff) [07:28:24] (03CR) 10CI reject: [V:04-1] doc: Update PHP spec file to what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 (owner: 10Muehlenhoff) [07:53:00] (03PS1) 10Muehlenhoff: Remove obsolete keys for PCC 5 instances [puppet] - 10https://gerrit.wikimedia.org/r/1223532 (https://phabricator.wikimedia.org/T367399) [07:53:30] (03CR) 10Muehlenhoff: "Made https://gerrit.wikimedia.org/r/c/operations/puppet/+/1223532 for this" [puppet] - 10https://gerrit.wikimedia.org/r/1075187 (https://phabricator.wikimedia.org/T367399) (owner: 10Muehlenhoff) [07:54:41] !log installing bash updates from Bookworm point release [07:54:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:00:04] Amir1, Urbanecm, and awight: UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T0800). Please do the needful. [08:00:05] No Gerrit patches in the queue for this window AFAICS. [08:06:24] (03CR) 10Hashar: [C:03+2] Disable banner for the 2025 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1223167 (owner: 10Hashar) [08:07:05] (03Merged) 10jenkins-bot: Disable banner for the 2025 developer survey [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1223167 (owner: 10Hashar) [08:08:15] !log hashar@deploy2002 Started deploy [gerrit/gerrit@c323101]: Disable banner for the 2025 developer survey [08:08:27] !log hashar@deploy2002 Finished deploy [gerrit/gerrit@c323101]: Disable banner for the 2025 developer survey (duration: 00m 11s) [08:12:13] (03CR) 10Slyngshede: [C:03+1] "Minor nit, looks good otherwise." [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [08:12:15] !log installing distro-info-data updates [08:12:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:22:12] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#11494814 (10MoritzMuehlenhoff) [08:37:00] (03CR) 10Dzahn: [C:03+2] Bump buildkitd to wmf-v0.26.3 [puppet] - 10https://gerrit.wikimedia.org/r/1223247 (https://phabricator.wikimedia.org/T412869) (owner: 10Ahmon Dancy) [08:46:02] (03PS1) 10Kevin Bazira: ml-services: update embeddings model-server image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223629 (https://phabricator.wikimedia.org/T412338) [08:52:46] (03PS2) 10Muehlenhoff: doc: Update PHP spec file to what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 [08:53:28] (03CR) 10CI reject: [V:04-1] doc: Update PHP spec file to what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 (owner: 10Muehlenhoff) [08:56:14] (03PS3) 10Muehlenhoff: doc: Update PHP spec file to what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 [09:00:13] (03PS4) 10Muehlenhoff: doc: Update PHP spec file to test what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 [09:07:55] !log installing e2fsprogs updates from Bookworm point release [09:07:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:14:00] !log push pfw policies - T413833 [09:14:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:15:44] (03CR) 10Ozge: [C:03+2] ml-services: update embeddings model-server image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223629 (https://phabricator.wikimedia.org/T412338) (owner: 10Kevin Bazira) [09:16:52] (03PS1) 10Kosta Harlan: QuickSurveys: Enable coverage percentages for safety survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223630 (https://phabricator.wikimedia.org/T413022) [09:16:52] 10SRE-swift-storage: PDF does not exist - https://phabricator.wikimedia.org/T413733#11494859 (10MatthewVernon) @Wargo I mean, that's a thread from 2013, so I don't think it's of any help to us now. The path in swift is determined by the name (see [[ https://wikitech.wikimedia.org/wiki/Swift/How_To#Find_an_o... [09:17:51] (03Merged) 10jenkins-bot: ml-services: update embeddings model-server image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223629 (https://phabricator.wikimedia.org/T412338) (owner: 10Kevin Bazira) [09:38:37] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#11494927 (10MoritzMuehlenhoff) [09:40:18] !log kevinbazira@deploy2002 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:12:36] (03CR) 10Majavah: [C:03+1] Remove obsolete keys for PCC 5 instances [puppet] - 10https://gerrit.wikimedia.org/r/1223532 (https://phabricator.wikimedia.org/T367399) (owner: 10Muehlenhoff) [10:19:49] !log mvernon@cumin2002 START - Cookbook sre.swift.remove-ghost-objects from container wikipedia-commons-local-public.f7 in codfw [10:22:16] !log mvernon@cumin2002 END (FAIL) - Cookbook sre.swift.remove-ghost-objects (exit_code=99) from container wikipedia-commons-local-public.f7 in codfw [10:25:25] (03CR) 10Cathal Mooney: [C:03+1] "LGTM!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1220311 (https://phabricator.wikimedia.org/T407991) (owner: 10Elukey) [10:26:15] 10SRE-SLO, 10Citoid, 10VisualEditor, 06Editing-team (Tracking): Seperate SLO for requests made from Citoid Extension, possible wmf deployed extension only, vs bots etc. - https://phabricator.wikimedia.org/T345627#11495075 (10Mvolz) 05Open→03Resolved [10:27:19] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'configure' for AS: 208915 [10:27:50] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 208915 [10:28:11] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'configure' for AS: 400566 [10:28:28] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 400566 [10:30:28] jouncebot: nowandnext [10:30:28] No deployments scheduled for the next 0 hour(s) and 29 minute(s) [10:30:28] In 0 hour(s) and 29 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1100) [10:30:28] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'configure' for AS: 18734 [10:30:33] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 18734 [10:30:43] I will sync a config patch, uless someone else is deploying now [10:31:53] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'clear' for AS: 55818 [10:31:53] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223630 (https://phabricator.wikimedia.org/T413022) (owner: 10Kosta Harlan) [10:32:40] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 55818 [10:32:44] (03Merged) 10jenkins-bot: QuickSurveys: Enable coverage percentages for safety survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223630 (https://phabricator.wikimedia.org/T413022) (owner: 10Kosta Harlan) [10:33:38] !log kharlan@deploy2002 Started scap sync-world: Backport for [[gerrit:1223630|QuickSurveys: Enable coverage percentages for safety survey (T413022)]] [10:33:40] T413022: First test, then launch the 2026 Community Safety survey - https://phabricator.wikimedia.org/T413022 [10:35:49] (03PS1) 10Kosta Harlan: IPReputation: Define data provider, URL and developer mode config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223635 (https://phabricator.wikimedia.org/T410615) [10:35:51] !log kharlan@deploy2002 kharlan: Backport for [[gerrit:1223630|QuickSurveys: Enable coverage percentages for safety survey (T413022)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [10:35:52] (03PS2) 10Hnowlan: thumbor: limit SVGs based on original file format, not output [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1212191 (https://phabricator.wikimedia.org/T411076) [10:36:15] (03PS2) 10Kosta Harlan: IPReputation: Define data provider, URL and developer mode config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223635 (https://phabricator.wikimedia.org/T410615) [10:36:59] !log kharlan@deploy2002 kharlan: Continuing with sync [10:40:20] !log installing aom bugfix updates from Bookworm point release [10:40:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:42:55] (03PS1) 10Kosta Harlan: IPReputation: Enable OpenSearch IPoid provider on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223636 (https://phabricator.wikimedia.org/T410615) [10:43:05] !log kharlan@deploy2002 Finished scap sync-world: Backport for [[gerrit:1223630|QuickSurveys: Enable coverage percentages for safety survey (T413022)]] (duration: 09m 27s) [10:43:08] T413022: First test, then launch the 2026 Community Safety survey - https://phabricator.wikimedia.org/T413022 [10:46:30] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'email' for AS: 3223 [10:47:27] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3223 [10:57:18] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#11495149 (10MoritzMuehlenhoff) [10:58:07] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'clear' for AS: 400566 [10:58:20] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 400566 [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1100) [11:00:48] (03CR) 10STran: IPReputation: Define data provider, URL and developer mode config (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223635 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:07:14] I'm going to deploy an apache config change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1223188 if you're all done kostajh [11:12:07] claime: yes I'm done [11:12:38] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [11:13:10] (03PS3) 10Kosta Harlan: IPReputation: Define data provider, URL and developer mode config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223635 (https://phabricator.wikimedia.org/T410615) [11:13:11] (03PS2) 10Kosta Harlan: IPReputation: Enable OpenSearch IPoid provider on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223636 (https://phabricator.wikimedia.org/T410615) [11:13:18] (03CR) 10Kosta Harlan: IPReputation: Define data provider, URL and developer mode config (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223635 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:13:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, January 06 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223205 (https://phabricator.wikimedia.org/T413773) (owner: 10STran) [11:13:49] (03PS1) 10Mvolz: Update zotero [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223642 [11:16:43] (03PS3) 10Kosta Harlan: (WIP) IPReputation: Enable OpenSearch IPoid provider on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223636 (https://phabricator.wikimedia.org/T410615) [11:17:06] (03CR) 10STran: (WIP) IPReputation: Enable OpenSearch IPoid provider on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223636 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:17:17] (03CR) 10Kosta Harlan: (WIP) IPReputation: Enable OpenSearch IPoid provider on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223636 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:17:28] (03PS5) 10Mvolz: Remove deprecated parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1204813 (https://phabricator.wikimedia.org/T361576) [11:17:32] (03PS6) 10Mvolz: Remove deprecated parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1204813 (https://phabricator.wikimedia.org/T361576) [11:17:34] (03CR) 10STran: (WIP) IPReputation: Enable OpenSearch IPoid provider on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223636 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:17:38] (03CR) 10CI reject: [V:04-1] (WIP) IPReputation: Enable OpenSearch IPoid provider on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223636 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:17:56] (03CR) 10STran: [C:03+1] IPReputation: Define data provider, URL and developer mode config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223635 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:18:37] (03PS7) 10Mvolz: Remove deprecated parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1204813 (https://phabricator.wikimedia.org/T361576) [11:18:54] (03CR) 10STran: [C:03+1] IPReputation: Define data provider, URL and developer mode config (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223635 (https://phabricator.wikimedia.org/T410615) (owner: 10Kosta Harlan) [11:19:27] (03CR) 10Clément Goubert: [C:03+2] apache: Don't redirect RestSandbox on wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1223188 (https://phabricator.wikimedia.org/T396807) (owner: 10Clément Goubert) [11:19:29] (03PS7) 10Mvolz: Remove deprecated parameter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1204813 (https://phabricator.wikimedia.org/T361576) [11:20:59] !log Deploying apache config change T396807 - 1223188 [11:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:21:02] T396807: Reroute /api/rest_v1 documentation to REST Sandbox - https://phabricator.wikimedia.org/T396807 [11:22:38] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [11:31:22] !log cgoubert@deploy2002 Started scap sync-world: 1223188: apache: Don't redirect RestSandbox on wikimedia.org [11:32:27] !log cgoubert@deploy2002 cgoubert: 1223188: apache: Don't redirect RestSandbox on wikimedia.org synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:35:01] (03CR) 10Ladsgroup: [C:03+1] thumbor: limit SVGs based on original file format, not output [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1212191 (https://phabricator.wikimedia.org/T411076) (owner: 10Hnowlan) [11:41:31] (03PS1) 10Clément Goubert: Revert "apache: Don't redirect RestSandbox on wikimedia.org" [puppet] - 10https://gerrit.wikimedia.org/r/1223643 (https://phabricator.wikimedia.org/T396807) [11:43:20] (03CR) 10CI reject: [V:04-1] Revert "apache: Don't redirect RestSandbox on wikimedia.org" [puppet] - 10https://gerrit.wikimedia.org/r/1223643 (https://phabricator.wikimedia.org/T396807) (owner: 10Clément Goubert) [11:43:35] (03PS2) 10Clément Goubert: Revert "apache: Don't redirect RestSandbox on wikimedia.org" [puppet] - 10https://gerrit.wikimedia.org/r/1223643 (https://phabricator.wikimedia.org/T396807) [11:45:52] (03CR) 10Clément Goubert: [C:03+2] Revert "apache: Don't redirect RestSandbox on wikimedia.org" [puppet] - 10https://gerrit.wikimedia.org/r/1223643 (https://phabricator.wikimedia.org/T396807) (owner: 10Clément Goubert) [11:57:10] !log cgoubert@deploy2002 Started scap sync-world: 1223643: Revert "apache: Don't redirect RestSandbox on wikimedia.org" [11:58:11] !log ladsgroup@deploy2002:~$ mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 11 echo-subscriptions-email-page-review (T406724) [11:58:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:58:14] T406724: Clean up watchlist and user properties of users if they don't log in for certain time - https://phabricator.wikimedia.org/T406724 [11:58:14] !log cgoubert@deploy2002 cgoubert: 1223643: Revert "apache: Don't redirect RestSandbox on wikimedia.org" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:58:42] !log cgoubert@deploy2002 cgoubert: Continuing with sync [11:59:48] !log cgoubert@deploy2002 Finished scap sync-world: 1223643: Revert "apache: Don't redirect RestSandbox on wikimedia.org" (duration: 03m 34s) [12:03:56] (03PS1) 10Btullis: Configure the contents of /etc/kyuubi/conf for the kyuubi toolbox pod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223646 (https://phabricator.wikimedia.org/T410017) [12:04:56] jouncebot: nowandnext [12:04:57] No deployments scheduled for the next 0 hour(s) and 55 minute(s) [12:04:57] In 0 hour(s) and 55 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1300) [12:05:45] claime: can I deploy something 👉 👈 [12:08:48] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1222827 (https://phabricator.wikimedia.org/T413031) (owner: 10Ladsgroup) [12:09:45] (03Merged) 10jenkins-bot: Reduce VP9 transcode resolution steps [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1222827 (https://phabricator.wikimedia.org/T413031) (owner: 10Ladsgroup) [12:10:19] !log ladsgroup@deploy2002 Started scap sync-world: Backport for [[gerrit:1222827|Reduce VP9 transcode resolution steps (T413031)]] [12:10:21] T413031: Reduce TimedMediaHandler VP9 transcode resolution steps - https://phabricator.wikimedia.org/T413031 [12:12:47] !log ladsgroup@deploy2002 ladsgroup: Backport for [[gerrit:1222827|Reduce VP9 transcode resolution steps (T413031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:14:11] !log ladsgroup@deploy2002 ladsgroup: Continuing with sync [12:14:26] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'email' for AS: 20940 [12:17:56] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 20940 [12:18:05] (03CR) 10Aklapper: [V:03+2 C:03+2] "Applies cleanly on top of git master locally, apart form the usual whitespace warnings" [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1222809 (owner: 10Pppery) [12:18:21] !log ladsgroup@deploy2002 Finished scap sync-world: Backport for [[gerrit:1222827|Reduce VP9 transcode resolution steps (T413031)]] (duration: 08m 02s) [12:18:23] T413031: Reduce TimedMediaHandler VP9 transcode resolution steps - https://phabricator.wikimedia.org/T413031 [12:23:38] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [12:28:01] (03CR) 10Andrew Bogott: [C:03+2] wikitech-static: associate with an elastic IP stored in AWS [dns] - 10https://gerrit.wikimedia.org/r/1218333 (https://phabricator.wikimedia.org/T376400) (owner: 10Andrew Bogott) [12:28:40] !log andrew@dns1004 START - running authdns-update [12:28:46] (03PS1) 10JMeybohm: Remove to be migrated ipblock sources fetch_external_*_nets.py [puppet] - 10https://gerrit.wikimedia.org/r/1223648 (https://phabricator.wikimedia.org/T412805) [12:28:48] (03PS1) 10JMeybohm: hiddenparma: Temporarily disable ipblock_source_no_ipblock_exists policy [puppet] - 10https://gerrit.wikimedia.org/r/1223649 (https://phabricator.wikimedia.org/T412805) [12:28:50] (03PS1) 10JMeybohm: Revert "hiddenparma: Temporarily disable ipblock_source_no_ipblock_exists policy" [puppet] - 10https://gerrit.wikimedia.org/r/1223650 (https://phabricator.wikimedia.org/T412805) [12:29:44] !log andrew@dns1004 END - running authdns-update [12:31:09] (03CR) 10CI reject: [V:04-1] hiddenparma: Temporarily disable ipblock_source_no_ipblock_exists policy [puppet] - 10https://gerrit.wikimedia.org/r/1223649 (https://phabricator.wikimedia.org/T412805) (owner: 10JMeybohm) [12:33:38] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [12:34:06] PROBLEM - Host wikitech-static.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [12:51:59] (03CR) 10Btullis: [C:03+2] Configure the contents of /etc/kyuubi/conf for the kyuubi toolbox pod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223646 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [12:52:51] !log uploaded Bird 2.18-1~wmf13u1 to component/bird-routed-ganeti for trixie-wikimedia T413740 [12:52:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:52:54] T413740: Backport and test Bird 2.18 - https://phabricator.wikimedia.org/T413740 [12:53:42] (03Merged) 10jenkins-bot: Configure the contents of /etc/kyuubi/conf for the kyuubi toolbox pod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223646 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [12:56:41] (03PS1) 10Clément Goubert: rest-gateway: Switch redis backend to 6380 instance [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223651 (https://phabricator.wikimedia.org/T413876) [12:58:28] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply [12:58:37] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply [13:00:06] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1300) [13:17:36] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1215186 (https://phabricator.wikimedia.org/T411775) (owner: 10Slyngshede) [13:20:32] (03PS1) 10Clément Goubert: rest-gateway: Move REST API Sandbox to mw-rest-php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223659 (https://phabricator.wikimedia.org/T396807) [13:21:21] jouncebot: nowandnext [13:21:21] For the next 0 hour(s) and 38 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1300) [13:21:21] In 0 hour(s) and 38 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1400) [13:23:10] (03PS1) 10DCausse: cirrus: allow title natural sort [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223660 (https://phabricator.wikimedia.org/T40403) [13:23:32] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, 06Traffic: Propose a new set of standard thumbnail sizes - https://phabricator.wikimedia.org/T412971#11495789 (10Ladsgroup) I've added those sizes to https://www.mediawiki.org/w/index.php?title=Common_thumbnail_sizes&diff=prev&oldid=8130399 [13:24:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, January 06 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223660 (https://phabricator.wikimedia.org/T40403) (owner: 10DCausse) [13:35:25] (03PS1) 10Btullis: Update the volumes and volumemounts of the kyuubi toolbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223661 (https://phabricator.wikimedia.org/T410017) [13:36:55] (03PS2) 10Clément Goubert: rest-gateway: Move REST API Sandbox to mw-rest-php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223659 (https://phabricator.wikimedia.org/T396807) [13:39:44] (03CR) 10Btullis: [C:03+2] Update the volumes and volumemounts of the kyuubi toolbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223661 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [13:41:25] (03Merged) 10jenkins-bot: Update the volumes and volumemounts of the kyuubi toolbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223661 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [13:46:12] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply [13:49:38] (03CR) 10Daniel Kinzler: [C:03+2] smokepy: send http requests in parallel [deployment-charts] - 10https://gerrit.wikimedia.org/r/1219188 (owner: 10Daniel Kinzler) [13:49:40] (03CR) 10Daniel Kinzler: [C:03+2] rest-gateway: improve structure of end-to-end tests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1219222 (https://phabricator.wikimedia.org/T413179) (owner: 10Daniel Kinzler) [13:51:29] (03Merged) 10jenkins-bot: smokepy: send http requests in parallel [deployment-charts] - 10https://gerrit.wikimedia.org/r/1219188 (owner: 10Daniel Kinzler) [13:51:53] (03Merged) 10jenkins-bot: rest-gateway: improve structure of end-to-end tests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1219222 (https://phabricator.wikimedia.org/T413179) (owner: 10Daniel Kinzler) [13:52:52] !log daniel@deploy2002 helmfile [staging] START helmfile.d/services/rest-gateway: apply [13:53:02] !log daniel@deploy2002 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [13:55:07] (03PS1) 10Btullis: Update the configmap names for spark-support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223663 (https://phabricator.wikimedia.org/T410017) [13:55:46] !log andrew@cumin2002 START - Cookbook sre.hosts.reimage for host cloudvirtlocal1003.eqiad.wmnet with OS trixie [13:56:26] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply [14:00:05] Lucas_WMDE, Urbanecm, and TheresNoTime: OwO what's this, a deployment window?? UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1400). nyaa~ [14:00:05] Tran and dcausse: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:08] o/ [14:00:11] o/ [14:01:48] I can deploy [14:02:17] I can also deploy my own. Doesn't seem like there's a deployer around so I can get started. [14:02:32] Tran: sure please go ahead [14:03:52] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223205 (https://phabricator.wikimedia.org/T413773) (owner: 10STran) [14:04:24] o/ [14:04:32] go ahead ^^ [14:04:41] * Lucas_WMDE meows back at jouncebot [14:04:48] (03Merged) 10jenkins-bot: Deploy IRS to pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223205 (https://phabricator.wikimedia.org/T413773) (owner: 10STran) [14:05:17] !log stran@deploy2002 Started scap sync-world: Backport for [[gerrit:1223205|Deploy IRS to pilot wikis (T413773 T413774 T413775 T413776 T413777)]] [14:05:27] link ALL the phabricator tasks [14:05:28] T413773: Deploy Incident Reporting System to ptwiki - https://phabricator.wikimedia.org/T413773 [14:05:28] T413774: Deploy Incident Reporting System to idwiki - https://phabricator.wikimedia.org/T413774 [14:05:28] T413775: Deploy Incident Reporting System to trwiki - https://phabricator.wikimedia.org/T413775 [14:05:29] T413776: Deploy Incident Reporting System to bnwiki - https://phabricator.wikimedia.org/T413776 [14:05:29] T413777: Deploy Incident Reporting System to azwiki - https://phabricator.wikimedia.org/T413777 [14:07:00] at least I didn't also link the epic :p [14:07:24] !log stran@deploy2002 stran: Backport for [[gerrit:1223205|Deploy IRS to pilot wikis (T413773 T413774 T413775 T413776 T413777)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:07:28] testing now [14:11:20] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, 06Traffic: Propose a new set of standard thumbnail sizes - https://phabricator.wikimedia.org/T412971#11496076 (10Ladsgroup) I'd say we should move the discussion about pre-generation to another ticket since it's a bit offtopic but in the... [14:11:25] (03CR) 10Btullis: [C:03+2] Update the configmap names for spark-support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223663 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [14:12:35] (03CR) 10Hashar: [C:03+1] doc: Update PHP spec file to test what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 (owner: 10Muehlenhoff) [14:12:42] !log andrew@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirtlocal1003.eqiad.wmnet with reason: host reimage [14:12:55] Hm...Does anyone know why idwiki wouldn't show the fallback en translation for a string? The others look good to me so I'm inclined to let it through as the others look okay and the primary user-facing UI also looks like it works on all wikis. [14:13:00] !log andrew@cumin2002 END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudvirtlocal1003.eqiad.wmnet with reason: host reimage [14:13:16] (03Merged) 10jenkins-bot: Update the configmap names for spark-support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223663 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [14:13:31] do you have an example? [14:14:40] (03PS1) 10Urbanecm: growthexperiments: Add pcmwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223666 (https://phabricator.wikimedia.org/T409480) [14:15:08] jk, cacheing issue (although this is my first time visiting the page)? Reloaded after testing all the others and it works as expected now. I'm going to proceed. [14:15:13] !log stran@deploy2002 stran: Continuing with sync [14:15:34] Tran: local override existing at `MediaWiki:XXX` on the wiki might also change things [14:16:37] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply [14:16:41] urbanecm: It looked like the text you see when the string hasn't been defined, eg `` [14:19:18] !log stran@deploy2002 Finished scap sync-world: Backport for [[gerrit:1223205|Deploy IRS to pilot wikis (T413773 T413774 T413775 T413776 T413777)]] (duration: 14m 01s) [14:19:27] T413773: Deploy Incident Reporting System to ptwiki - https://phabricator.wikimedia.org/T413773 [14:19:27] T413774: Deploy Incident Reporting System to idwiki - https://phabricator.wikimedia.org/T413774 [14:19:28] T413775: Deploy Incident Reporting System to trwiki - https://phabricator.wikimedia.org/T413775 [14:19:28] T413776: Deploy Incident Reporting System to bnwiki - https://phabricator.wikimedia.org/T413776 [14:19:28] T413777: Deploy Incident Reporting System to azwiki - https://phabricator.wikimedia.org/T413777 [14:19:39] dcausse: I'm done if you want to deploy yours [14:19:54] Tran: thanks, will do [14:20:17] (03PS1) 10Btullis: Fix the name of the kerberos-client-configuration configmap [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223668 (https://phabricator.wikimedia.org/T410017) [14:20:25] (03PS2) 10Btullis: Fix the name of the kerberos-client-configuration configmap [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223668 (https://phabricator.wikimedia.org/T410017) [14:20:26] (03CR) 10CI reject: [V:04-1] Fix the name of the kerberos-client-configuration configmap [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223668 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [14:21:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223660 (https://phabricator.wikimedia.org/T40403) (owner: 10DCausse) [14:21:48] (03Merged) 10jenkins-bot: cirrus: allow title natural sort [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223660 (https://phabricator.wikimedia.org/T40403) (owner: 10DCausse) [14:22:20] !log dcausse@deploy2002 Started scap sync-world: Backport for [[gerrit:1223660|cirrus: allow title natural sort (T40403)]] [14:22:23] T40403: Sortable search results - https://phabricator.wikimedia.org/T40403 [14:22:25] (03CR) 10Btullis: [C:03+2] Fix the name of the kerberos-client-configuration configmap [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223668 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [14:24:05] (03PS3) 10Silvan Heintze: Report progress of Wikibase entity dumps [dumps] - 10https://gerrit.wikimedia.org/r/1219837 (https://phabricator.wikimedia.org/T408423) [14:24:09] (03Merged) 10jenkins-bot: Fix the name of the kerberos-client-configuration configmap [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223668 (https://phabricator.wikimedia.org/T410017) (owner: 10Btullis) [14:24:09] (03CR) 10Muehlenhoff: [C:03+2] doc: Update PHP spec file to test what's actually used in production [puppet] - 10https://gerrit.wikimedia.org/r/1223530 (owner: 10Muehlenhoff) [14:24:28] !log dcausse@deploy2002 dcausse: Backport for [[gerrit:1223660|cirrus: allow title natural sort (T40403)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:26:51] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply [14:28:06] (03CR) 10Silvan Heintze: "Thanks for the review" [dumps] - 10https://gerrit.wikimedia.org/r/1219837 (https://phabricator.wikimedia.org/T408423) (owner: 10Silvan Heintze) [14:28:51] * Lucas_WMDE is intrigued but can’t find the title sort yet on mwdebug [14:30:07] Lucas_WMDE: there's no UI yet :/ [14:30:12] ah, ok ^^ [14:30:12] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply [14:30:23] mystery feature ✨ [14:30:24] !log btullis@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply [14:30:50] yes... you have to append &sort=title_natural_asc to your Special:Search url [14:31:05] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on restbase1035 - https://phabricator.wikimedia.org/T413678#11496127 (10Eevans) >>! In T413678#11494323, @Jclark-ctr wrote: > Service request with Dell 220855023 Presumably this means //weeks//, not //days//; I will decommission this node. [14:32:01] that works :o [14:32:07] cool stuff [14:32:25] hopefully we could get that exposed to the AdvancedSearch UI at some point (T403775 would be the ticket I guess) [14:32:26] T403775: New search option: Sort results by page name - https://phabricator.wikimedia.org/T403775 [14:32:45] testing another couple wikis and will continue the sync [14:33:10] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on restbase1035 - https://phabricator.wikimedia.org/T413678#11496162 (10Jclark-ctr) @Eevans The part has been ordered. They requested additional shipping information last night at 7:00 PM Eastern, so it should ship today and arrive on-site tomorrow afternoon. [14:35:46] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on restbase1035 - https://phabricator.wikimedia.org/T413678#11496188 (10Eevans) >>! In T413678#11496162, @Jclark-ctr wrote: > @Eevans The part has been ordered. They requested additional shipping information last night at 7:00 PM Eastern, so it should ship today an... [14:38:15] !log andrew@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirtlocal1003.eqiad.wmnet with OS trixie [14:39:53] (03PS2) 10Muehlenhoff: ci/php: Remove support for buster [puppet] - 10https://gerrit.wikimedia.org/r/1219875 [14:40:26] !log dcausse@deploy2002 dcausse: Continuing with sync [14:40:55] (03PS1) 10Ayounsi: Add astein RO user to network devices [homer/public] - 10https://gerrit.wikimedia.org/r/1223671 (https://phabricator.wikimedia.org/T413826) [14:41:52] (03PS3) 10Majavah: spec: Stop running tests on buster [puppet] - 10https://gerrit.wikimedia.org/r/1219149 [14:42:37] (03PS4) 10Majavah: spec: Stop running tests on buster [puppet] - 10https://gerrit.wikimedia.org/r/1219149 [14:42:39] (03CR) 10CI reject: [V:04-1] spec: Stop running tests on buster [puppet] - 10https://gerrit.wikimedia.org/r/1219149 (owner: 10Majavah) [14:44:30] !log dcausse@deploy2002 Finished scap sync-world: Backport for [[gerrit:1223660|cirrus: allow title natural sort (T40403)]] (duration: 22m 10s) [14:44:33] T40403: Sortable search results - https://phabricator.wikimedia.org/T40403 [14:45:27] (03CR) 10Ssingh: "Looks good for the bits that I "own"; how are we doing the review for this? Just +1ing and specifying individually or should someone revie" [puppet] - 10https://gerrit.wikimedia.org/r/1219149 (owner: 10Majavah) [14:45:34] ok I'm done [14:46:10] !log closing the UTC afternoon backport window [14:46:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:47:56] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: Cleaning up Puppet and Netbox VLAN sub-ints on edge sites - https://phabricator.wikimedia.org/T410411#11496209 (10ssingh) a:03ssingh [14:49:57] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: Cleaning up Puppet and Netbox VLAN sub-ints on edge sites - https://phabricator.wikimedia.org/T410411#11496214 (10ssingh) Is there anything to be aware of about the order of this? Should we just merge the above patch and then s... [14:50:45] !log andrew@cumin2002 START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS trixie [14:53:26] 06SRE, 06cloud-services-team, 10Wikimedia-Mailing-lists: aborrero@wikimedia.org still subscribed to ops@lists.wikimedia.org - https://phabricator.wikimedia.org/T413883#11496217 (10Ladsgroup) I'm not seeing the email address in ops list. Maybe someone removed it in the mean time. [14:55:10] (03CR) 10Herron: [C:03+2] admin: herron: add yubikey ssh key [puppet] - 10https://gerrit.wikimedia.org/r/1219625 (owner: 10Herron) [14:58:20] (03PS1) 10Dreamy Jazz: Write new for CheckUser user agent table migration on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223673 (https://phabricator.wikimedia.org/T361196) [15:00:00] (03PS1) 10Dreamy Jazz: Write new for CheckUser user agent table migration on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223674 (https://phabricator.wikimedia.org/T361196) [15:00:02] (03PS1) 10Dreamy Jazz: Write new for CheckUser user agent table migration everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223675 (https://phabricator.wikimedia.org/T361196) [15:00:05] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1500) [15:00:39] (03PS2) 10Dreamy Jazz: Write new for CheckUser user agent table migration on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223674 (https://phabricator.wikimedia.org/T361196) [15:00:46] (03PS2) 10Dreamy Jazz: Write new for CheckUser user agent table migration everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223675 (https://phabricator.wikimedia.org/T361196) [15:01:23] (03PS2) 10Dreamy Jazz: Write new for CheckUser user agent table migration on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223673 (https://phabricator.wikimedia.org/T361196) [15:01:33] (03PS3) 10Dreamy Jazz: Write new for CheckUser user agent table migration on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223674 (https://phabricator.wikimedia.org/T361196) [15:01:33] (03PS3) 10Dreamy Jazz: Write new for CheckUser user agent table migration everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223675 (https://phabricator.wikimedia.org/T361196) [15:05:52] !log andrew@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage [15:06:10] !log andrew@cumin2002 END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage [15:06:47] (03CR) 10Cathal Mooney: [C:03+1] hiera: lvs/interfaces: remove VLAN sub-ints for edges [puppet] - 10https://gerrit.wikimedia.org/r/1207180 (https://phabricator.wikimedia.org/T410411) (owner: 10Ssingh) [15:09:07] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: Cleaning up Puppet and Netbox VLAN sub-ints on edge sites - https://phabricator.wikimedia.org/T410411#11496261 (10cmooney) >>! In T410411#11496209, @ssingh wrote: > Is there anything to be aware of about the order of this? Shou... [15:10:12] (03CR) 10Muehlenhoff: "Fixed via https://gerrit.wikimedia.org/r/c/operations/puppet/+/1223530" [puppet] - 10https://gerrit.wikimedia.org/r/1219875 (owner: 10Muehlenhoff) [15:13:50] (03PS1) 10Muehlenhoff: etcd: Remove obsolete check [puppet] - 10https://gerrit.wikimedia.org/r/1223676 [15:15:27] (03PS1) 10Muehlenhoff: beta::mediawiki_packages: Remove support for buster [puppet] - 10https://gerrit.wikimedia.org/r/1223677 [15:17:35] 06SRE, 07SRE-Unowned, 06serviceops-radar, 10wikitech.wikimedia.org, 13Patch-For-Review: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#11496297 (10taavi) >>! In T376400#11345384, @Andrew wrote: >> Sure, for example the first image on http://ec2-54-81-201-239.compute-1.amazonaws.com/... [15:20:02] (03PS1) 10Slyngshede: Update to CAS version 7.3.1 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1223679 [15:28:54] !log andrew@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet1005.eqiad.wmnet with OS trixie [15:29:53] (03CR) 10Cyndywikime: [C:03+1] growthexperiments: Add pcmwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223666 (https://phabricator.wikimedia.org/T409480) (owner: 10Urbanecm) [15:29:55] (03CR) 10Jakob: [C:03+1] "LGTM, thank you!" [dumps] - 10https://gerrit.wikimedia.org/r/1219837 (https://phabricator.wikimedia.org/T408423) (owner: 10Silvan Heintze) [15:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1530) [15:30:50] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223676 (owner: 10Muehlenhoff) [16:00:05] jelto, arnoldokoth, mutante, and arnaudb: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) SRE Collaboration Services office hours deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1600). [16:04:56] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:28:11] !log andrew@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage [16:28:37] !log andrew@cumin2002 END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage [16:33:15] !log sukhe@cp1100:~$ sudo ats-backend-restart [16:33:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:34:27] (03PS2) 10Clément Goubert: rest-gateway: Switch redis backend to 6380 instance [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223651 (https://phabricator.wikimedia.org/T413876) [16:44:32] (03PS1) 10Awight: Configure new stream mediawiki.wmde_page_summary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223688 (https://phabricator.wikimedia.org/T413891) [16:49:02] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Configure new stream mediawiki.wmde_page_summary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223688 (https://phabricator.wikimedia.org/T413891) (owner: 10Awight) [16:50:11] !log andrew@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet1006.eqiad.wmnet with OS trixie [16:55:05] (03PS2) 10Urbanecm: growthexperiments: Add pcmwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223666 (https://phabricator.wikimedia.org/T409480) [16:55:10] (03CR) 10Urbanecm: [C:03+2] growthexperiments: Add pcmwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223666 (https://phabricator.wikimedia.org/T409480) (owner: 10Urbanecm) [16:56:07] (03Merged) 10jenkins-bot: growthexperiments: Add pcmwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223666 (https://phabricator.wikimedia.org/T409480) (owner: 10Urbanecm) [16:57:27] !log urbanecm@deploy2002 Started scap sync-world: Backport for [[gerrit:1223666|growthexperiments: Add pcmwiki (T409480)]] [16:57:30] T409480: Enable GrowthExperiments on a new wiki (pcmwiki) and document the process - https://phabricator.wikimedia.org/T409480 [16:59:16] (03PS9) 10CDanis: P:cache haproxy support tagging residential proxies [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [16:59:20] (03CR) 10CDanis: P:cache haproxy support tagging residential proxies (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [16:59:22] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [16:59:38] !log urbanecm@deploy2002 urbanecm: Backport for [[gerrit:1223666|growthexperiments: Add pcmwiki (T409480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:00:05] jhathaway and rzl: OwO what's this, a deployment window?? Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1700). nyaa~ [17:00:05] No Gerrit patches in the queue for this window AFAICS. [17:03:43] !log urbanecm@deploy2002 urbanecm: Continuing with sync [17:07:54] !log urbanecm@deploy2002 Finished scap sync-world: Backport for [[gerrit:1223666|growthexperiments: Add pcmwiki (T409480)]] (duration: 10m 27s) [17:07:58] T409480: Enable GrowthExperiments on a new wiki (pcmwiki) and document the process - https://phabricator.wikimedia.org/T409480 [17:09:22] (03PS10) 10CDanis: P:cache haproxy support tagging residential proxies [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [17:09:23] (03PS1) 10CDanis: haproxy: res proxy: trial on cp6006 [puppet] - 10https://gerrit.wikimedia.org/r/1223692 [17:09:43] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223692 (owner: 10CDanis) [17:09:46] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [17:15:38] (03PS11) 10CDanis: P:cache haproxy support tagging residential proxies [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [17:15:38] (03PS2) 10CDanis: haproxy: res proxy: trial on cp6006 [puppet] - 10https://gerrit.wikimedia.org/r/1223692 [17:15:47] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [17:17:42] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests, 06Security-Team, 07SecTeam-Processed: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11497096 (10sbassett) Cleaned up a few additional Phab ACLs and projects. [17:41:41] (03CR) 10Jdlrobson: [C:03+1] "Looks optimized to me? Am I missing something?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1219865 (https://phabricator.wikimedia.org/T413217) (owner: 10Aude) [17:56:50] !log 💙cdanis@cumin1003.eqiad.wmnet ~ 🕐☕ sudo cumin 'A:cp' 'disable-puppet "cdanis deploy I9b2f6edc6b7"' [17:56:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:57:51] (03CR) 10CDanis: [C:03+2] haproxy: res proxy: trial on cp6006 [puppet] - 10https://gerrit.wikimedia.org/r/1223692 (owner: 10CDanis) [17:57:53] (03CR) 10CDanis: [C:03+2] P:cache haproxy support tagging residential proxies [puppet] - 10https://gerrit.wikimedia.org/r/1219882 (owner: 10Slyngshede) [17:58:35] (03CR) 10Jforrester: "This was very intentionally left in the repo to lock behaviour between users. Why did you remove it?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1221678 (owner: 10Zabe) [18:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1800) [18:09:20] (03PS1) 10CDanis: haproxy: fix: the shared Lua always loads both mmdb [puppet] - 10https://gerrit.wikimedia.org/r/1223701 [18:09:50] (03CR) 10CI reject: [V:04-1] haproxy: fix: the shared Lua always loads both mmdb [puppet] - 10https://gerrit.wikimedia.org/r/1223701 (owner: 10CDanis) [18:10:44] (03PS2) 10CDanis: haproxy: fix: the shared Lua always loads both mmdb [puppet] - 10https://gerrit.wikimedia.org/r/1223701 [18:11:13] (03CR) 10CI reject: [V:04-1] haproxy: fix: the shared Lua always loads both mmdb [puppet] - 10https://gerrit.wikimedia.org/r/1223701 (owner: 10CDanis) [18:12:04] (03PS3) 10CDanis: haproxy: fix: the shared Lua always loads both mmdb [puppet] - 10https://gerrit.wikimedia.org/r/1223701 [18:12:44] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223701 (owner: 10CDanis) [18:13:49] (03CR) 10CDanis: [C:03+2] haproxy: fix: the shared Lua always loads both mmdb [puppet] - 10https://gerrit.wikimedia.org/r/1223701 (owner: 10CDanis) [18:13:58] (03CR) 10Ssingh: [C:03+1] haproxy: fix: the shared Lua always loads both mmdb [puppet] - 10https://gerrit.wikimedia.org/r/1223701 (owner: 10CDanis) [18:22:34] !log 💙cdanis@cumin1003.eqiad.wmnet ~ 🕜☕ sudo cumin 'A:cp' 'enable-puppet "cdanis deploy I9b2f6edc6b7"' [18:22:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:47:29] (03PS1) 10Andrew Bogott: external_monitoring: don't choke on hosts that don't resolve for both ipv4 and ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/1223702 [18:47:36] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223702 (owner: 10Andrew Bogott) [18:49:16] (03CR) 10CI reject: [V:04-1] external_monitoring: don't choke on hosts that don't resolve for both ipv4 and ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/1223702 (owner: 10Andrew Bogott) [18:51:12] (03PS2) 10Andrew Bogott: external_monitoring: don't choke on hosts that don't resolve both ipv4 and ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/1223702 [18:51:56] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223702 (owner: 10Andrew Bogott) [18:57:05] (03CR) 10Andrea Denisse: [C:03+1] external_monitoring: don't choke on hosts that don't resolve both ipv4 and ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/1223702 (owner: 10Andrew Bogott) [18:59:42] (03CR) 10Andrew Bogott: [C:03+2] external_monitoring: don't choke on hosts that don't resolve both ipv4 and ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/1223702 (owner: 10Andrew Bogott) [19:00:05] dduvall and dancy: Time to snap out of that daydream and deploy MediaWiki train - Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T1900). [19:00:20] o/ [19:00:35] o/ [19:03:08] PROBLEM - mailman list info ssl expiry on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [19:03:54] (03PS1) 10Andrew Bogott: Remove wikitech-static from external_monitoring.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1223704 [19:04:02] RECOVERY - mailman list info ssl expiry on lists1004 is OK: OK - Certificate lists.wikimedia.org will expire on Sat 04 Apr 2026 07:22:16 PM GMT +0000. https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [19:06:44] (03PS1) 10TrainBranchBot: group0 to 1.46.0-wmf.10 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223705 (https://phabricator.wikimedia.org/T408280) [19:06:46] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy2002" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223705 (https://phabricator.wikimedia.org/T408280) (owner: 10TrainBranchBot) [19:07:17] (03CR) 10Ssingh: [C:03+1] Remove wikitech-static from external_monitoring.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1223704 (owner: 10Andrew Bogott) [19:07:22] (03PS2) 10Andrew Bogott: Remove wikitech-static from external_monitoring.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1223704 [19:07:32] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223704 (owner: 10Andrew Bogott) [19:07:35] (03Merged) 10jenkins-bot: group0 to 1.46.0-wmf.10 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1223705 (https://phabricator.wikimedia.org/T408280) (owner: 10TrainBranchBot) [19:09:17] (03CR) 10Ssingh: Remove wikitech-static from external_monitoring.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1223704 (owner: 10Andrew Bogott) [19:10:16] (03CR) 10Andrea Denisse: [C:03+1] "LGTM thanks!!" [puppet] - 10https://gerrit.wikimedia.org/r/1223704 (owner: 10Andrew Bogott) [19:12:41] (03CR) 10Andrew Bogott: [C:03+2] Remove wikitech-static from external_monitoring.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1223704 (owner: 10Andrew Bogott) [19:13:47] !log dduvall@deploy2002 rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.10 refs T408280 [19:13:50] T408280: 1.46.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T408280 [19:25:08] (03CR) 10Thcipriani: [C:03+1] "Confirmed my new key is working, good to remove the old one! Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1220410 (https://phabricator.wikimedia.org/T413416) (owner: 10Thcipriani) [19:47:27] (03PS2) 10Jgreen: nsca_frack.cfg.erb deprecate check_endpoints service and pay-lvs hostgroup [puppet] - 10https://gerrit.wikimedia.org/r/1202827 (https://phabricator.wikimedia.org/T367370) [19:52:06] (03CR) 10Dwisehaupt: [C:03+1] nsca_frack.cfg.erb deprecate check_endpoints service and pay-lvs hostgroup [puppet] - 10https://gerrit.wikimedia.org/r/1202827 (https://phabricator.wikimedia.org/T367370) (owner: 10Jgreen) [19:54:57] is there anyone around who can review+merge this ^^^ gerrit commit? [20:08:17] 10ops-codfw, 06SRE, 06DC-Ops, 10fundraising-tech-ops: Q2:rack/setup/install franio2004 - https://phabricator.wikimedia.org/T405981#11497740 (10Jgreen) a:05Jgreen→03None [20:19:04] (03PS1) 10Jdlrobson: Update VE core submodule to master (24389ad60) [extensions/VisualEditor] (wmf/1.46.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1223715 [20:50:36] (03PS1) 10CDanis: haproxy: proxy mmdb: extend canary to cp6009 (text) [puppet] - 10https://gerrit.wikimedia.org/r/1223719 [20:50:51] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223719 (owner: 10CDanis) [20:51:05] (03CR) 10CI reject: [V:04-1] haproxy: proxy mmdb: extend canary to cp6009 (text) [puppet] - 10https://gerrit.wikimedia.org/r/1223719 (owner: 10CDanis) [20:51:28] (03PS2) 10CDanis: haproxy: proxy mmdb: extend canary to cp6009 (text) [puppet] - 10https://gerrit.wikimedia.org/r/1223719 [20:51:46] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1223719 (owner: 10CDanis) [20:53:45] (03CR) 10CDanis: [C:03+2] haproxy: proxy mmdb: extend canary to cp6009 (text) [puppet] - 10https://gerrit.wikimedia.org/r/1223719 (owner: 10CDanis) [21:00:05] RoanKattouw, Urbanecm, TheresNoTime, kindrobot, and cjming: UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T2100). Please do the needful. [21:00:05] Superpes: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:08:07] (03PS1) 10Ebernhardson: dumps: Repoint cirrus dumps to new location [puppet] - 10https://gerrit.wikimedia.org/r/1223722 (https://phabricator.wikimedia.org/T366248) [21:10:51] (03PS1) 10Daniel Kinzler: Makefiles: strip trailing whitespace from parameters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223723 [21:16:17] Superpes: do you need a deployer? [21:31:26] (03PS1) 10Bking: OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) [21:33:16] (03CR) 10CI reject: [V:04-1] OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) (owner: 10Bking) [21:35:05] (03PS2) 10Bking: OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) [21:36:31] (03CR) 10CI reject: [V:04-1] OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) (owner: 10Bking) [21:42:48] (03PS3) 10Bking: OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) [22:00:04] Deploy window Web Team deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260106T2200) [22:05:24] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [22:08:27] o/ about to start a deploy in a bit. Lemme know if anything is in flight that I should lknow abiut [22:09:16] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [22:09:43] all quiet afaik [22:10:05] (03PS1) 10Jdlrobson: Update VE core submodule to master (f53cf0905) [extensions/VisualEditor] (wmf/1.46.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1223716 (https://phabricator.wikimedia.org/T413356) [22:10:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2002 using scap backport" [extensions/VisualEditor] (wmf/1.46.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1223715 (owner: 10Jdlrobson) [22:10:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2002 using scap backport" [extensions/VisualEditor] (wmf/1.46.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1223716 (https://phabricator.wikimedia.org/T413356) (owner: 10Jdlrobson) [22:12:43] (03Merged) 10jenkins-bot: Update VE core submodule to master (24389ad60) [extensions/VisualEditor] (wmf/1.46.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1223715 (owner: 10Jdlrobson) [22:12:44] (03Merged) 10jenkins-bot: Update VE core submodule to master (f53cf0905) [extensions/VisualEditor] (wmf/1.46.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1223716 (https://phabricator.wikimedia.org/T413356) (owner: 10Jdlrobson) [22:13:18] !log jdlrobson@deploy2002 Started scap sync-world: Backport for [[gerrit:1223715|Update VE core submodule to master (24389ad60)]], [[gerrit:1223716|Update VE core submodule to master (f53cf0905) (T413356)]] [22:13:22] T413356: On Wikimedia Commons: Uncaught TypeError: can't access property "getSelection", surface is null - https://phabricator.wikimedia.org/T413356 [22:15:44] !log jdlrobson@deploy2002 jdlrobson: Backport for [[gerrit:1223715|Update VE core submodule to master (24389ad60)]], [[gerrit:1223716|Update VE core submodule to master (f53cf0905) (T413356)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:19:24] !log jdlrobson@deploy2002 jdlrobson: Continuing with sync [22:23:35] !log jdlrobson@deploy2002 Finished scap sync-world: Backport for [[gerrit:1223715|Update VE core submodule to master (24389ad60)]], [[gerrit:1223716|Update VE core submodule to master (f53cf0905) (T413356)]] (duration: 10m 17s) [22:23:38] T413356: On Wikimedia Commons: Uncaught TypeError: can't access property "getSelection", surface is null - https://phabricator.wikimedia.org/T413356 [22:28:13] (03PS4) 10Bking: OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) [22:29:05] (03CR) 10Ryan Kemper: [C:03+1] OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) (owner: 10Bking) [22:30:09] (03CR) 10Bking: [C:03+2] OpenSearch on K8s: add curated alerts from official mixin [alerts] - 10https://gerrit.wikimedia.org/r/1223727 (https://phabricator.wikimedia.org/T408640) (owner: 10Bking) [22:33:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1219865 (https://phabricator.wikimedia.org/T413217) (owner: 10Aude) [22:34:38] (03Merged) 10jenkins-bot: Add wordmark logo to beta cluster for Minerva (mobile) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1219865 (https://phabricator.wikimedia.org/T413217) (owner: 10Aude) [22:35:10] !log jdlrobson@deploy2002 Started scap sync-world: Backport for [[gerrit:1219865|Add wordmark logo to beta cluster for Minerva (mobile) (T413217)]] [22:35:14] T413217: Add wordmark logo on beta cluster for Minerva (mobile) - https://phabricator.wikimedia.org/T413217 [22:37:31] !log jdlrobson@deploy2002 jdlrobson, aude: Backport for [[gerrit:1219865|Add wordmark logo to beta cluster for Minerva (mobile) (T413217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:43:02] (done! releasing the conch!) [23:29:17] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, 06Traffic: Propose a new set of standard thumbnail sizes - https://phabricator.wikimedia.org/T412971#11498230 (10AntiCompositeNumber) Special:NewFiles doesn't appear to be as bad as it was a few years ago, but I do think it would still be... [23:55:47] (03CR) 10Aaron Schulz: rest-gateway: Move REST API Sandbox to mw-rest-php (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1223659 (https://phabricator.wikimedia.org/T396807) (owner: 10Clément Goubert)