[00:46:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#11892483 (10Papaul) [02:07:56] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 5 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11892594 (10EvenTwist41) I understand the need to stop generating arbitrary thumbnail sizes, but was it really necessary to break exi... [02:26:18] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11892598 (10Papaul) All the servers in rack 23 are online and ready for re-image. I tested the re-image on cp4038 and completed with no issues after @ayou... [04:37:22] FIRING: [3x] HaproxyKafkaExporterDown: HaproxyKafka on cp4040 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [04:37:22] FIRING: [4x] HaproxyKafkaExporterDown: HaproxyKafka on cp4046 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [04:38:11] FIRING: [7x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp4040:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [06:48:13] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11892851 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `ganeti4006.ulsfo.wmnet` - ganeti4006.ulsfo.wmnet (**FAIL**) - Down... [07:08:07] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11892868 (10Bbbub) I am sorry to say that this issue continues to be unresolved for our iOS / Android app. There was an improvement after the in... [07:14:48] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11892877 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `ganeti4008.ulsfo.wmnet` - ganeti4008.ulsfo.wmn... [08:34:44] 06Traffic, 06collaboration-services, 10Continuous-Integration-Infrastructure, 10doc.wikimedia.org: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11893263 (10hashar) We had the cache set to one hour since content o... [08:37:37] FIRING: [3x] HaproxyKafkaExporterDown: HaproxyKafka on cp4040 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [08:37:37] FIRING: [4x] HaproxyKafkaExporterDown: HaproxyKafka on cp4046 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [08:38:11] FIRING: [7x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp4040:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [08:58:50] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893348 (10SLyngshede-WMF) a:03SLyngshede-WMF [09:06:05] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11893363 (10Nylki) >>! In T413570#11892868, @Bbbub wrote: > I am sorry to say that this issue continues to be unresolved for our iOS / Android a... [09:15:31] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11893382 (10Bbbub) Thank you confirming! Did you get any actionable insights from correspondence or experimentation, like maybe adding a user-ha... [09:15:44] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893384 (10SLyngshede-WMF) Re-image command: ` sudo cookbook sre.hosts.reimage --os trixie -t T424686 cp4040 ` [09:17:35] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893389 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4040.ulsfo.wmnet with OS trixie [09:43:22] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11893478 (10Nylki) >>! In T413570#11893382, @Bbbub wrote: > Thank you confirming! Did you get any actionable insights from correspondence or exp... [10:11:09] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893585 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4040.ulsfo.wmnet with OS trixie compl... [10:16:54] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893594 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4042.ulsfo.wmnet with OS trixie [10:17:49] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893603 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4044.ulsfo.wmnet with OS trixie [10:18:18] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893604 (10SLyngshede-WMF) [10:33:24] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893638 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti4006.ulsfo.wmnet with OS bookworm [11:10:47] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893727 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4042.ulsfo.wmnet with OS trixie compl... [11:14:52] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893731 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4044.ulsfo.wmnet with OS trixie compl... [11:19:10] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893732 (10SLyngshede-WMF) [11:19:56] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893733 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4048.ulsfo.wmnet with OS trixie [11:20:26] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893734 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie [11:37:12] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893749 (10SLyngshede-WMF) cp4046 refused to be reimaged and seems to be missing some configuration in Netbox from what I can tell. I've pinged @Pap... [11:42:31] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893752 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti4006.ulsfo.wmnet with OS bookworm comple... [12:16:49] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893843 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie compl... [12:18:08] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893847 (10SLyngshede-WMF) Same issue for cp4052: ` slyngshede@cumin1003:~$ sudo cookbook sre.hosts.reimage --os trixie -t T424686 cp4052 ==> ATTE... [12:19:16] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893858 (10SLyngshede-WMF) [12:20:32] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893861 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4048.ulsfo.wmnet with OS trixie compl... [12:37:37] FIRING: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp4046 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [12:38:26] FIRING: [2x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp4046:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [12:40:34] 10netops, 06Traffic, 06Infrastructure-Foundations: POPs LVS : remove public vlan trunking - https://phabricator.wikimedia.org/T367732#11893920 (10ayounsi) a:05ssingh→03Papaul I think that cable got properly removed yesterday and this task can be closed. @papaul do you confirm? [12:46:32] Hey team, we (ml team) are in the process of exposing an additional port for an existing LVS service. We are basically adding GRPC capability to ml inference services on our ml k8s clusters alongside REST. I prepared a sequence of 3 changes starting with this one: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1283745 [12:46:32] Does anybody have time to chime in and make sure it looks good? :) [12:50:15] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11893969 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host lvs4008.ulsfo.wmnet with OS trixie [13:12:48] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894094 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host lvs4010.ulsfo.wmnet with OS trixie [13:19:40] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894116 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4046.ulsfo.wmnet with OS trixie [13:36:24] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894186 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti4008.ulsfo.wmnet with OS bookworm [13:55:20] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894298 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4052.ulsfo.wmnet with OS trixie [14:08:43] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894442 (10MoritzMuehlenhoff) [14:11:40] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894462 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4046.ulsfo.wmnet with OS trixie compl... [14:16:25] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894499 (10SLyngshede-WMF) [14:21:12] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894546 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti4008.ulsfo.wmnet with OS bookworm comple... [14:41:38] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894698 (10MoritzMuehlenhoff) [14:54:00] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894781 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4052.ulsfo.wmnet with OS trixie compl... [14:54:43] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11894786 (10SLyngshede-WMF) [15:00:11] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11894839 (10daniel) >>! In T413570#11892868, @Bbbub wrote: > I am sorry to say that this issue continues to be unresolved for our iOS / Android... [15:14:26] FIRING: SystemdUnitFailed: ipip-multiqueue-optimizer.service on lvs4008:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:15:49] FIRING: PuppetFailure: Puppet has failed on lvs4008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:35:36] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11895179 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS bookworm [15:37:20] that was absolutely the wrong ticket but yeah [15:38:28] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11895204 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host lvs4008.ulsfo.wmnet with OS bookworm [15:40:49] FIRING: PuppetFailure: Puppet has failed on lvs4010:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:44:26] FIRING: SystemdUnitFailed: ipip-multiqueue-optimizer.service on lvs4010:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:47:17] would have downtimed [15:47:41] but reimaging will start so leaving it [16:28:53] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11895400 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host lvs4008.ulsfo.wmnet with OS bookworm completed: - lvs4008 (**PASS*... [16:37:11] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11895415 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS bookworm completed: - dns4004 (**WARN**) - Downtimed on Icinga/Alertmanager - //Unable... [16:40:23] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11895455 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host lvs4008.ulsfo.wmnet with OS trixie completed: - lvs4008 (**WA... [16:51:47] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11895505 (10ssingh) ^ The above is not a trixie reimage but a bookworm reimage for the ulsfo work in T424686. Please disregard. [17:02:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11895570 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a4b7dc3f-da06-4cb4-8580-9dac41f4da23) set by sukhe@cumin1003 for 3 days, 0:00... [17:10:28] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS serving an older revision of some overwritten files - https://phabricator.wikimedia.org/T425216#11895597 (10ssingh) >>! In T425216#11887650, @TheDJ wrote: > I believe there is a 24 hourly script that checks cross dc cons... [17:22:25] sukhe: do you know if anyone is trying reimages in ulsfo right now? [17:22:49] we need to reboot the switch in rack 23 to upgrade and potentially overcome some issues that were seen earlier in the week [17:24:01] topranks: no current reimages in progress [17:24:36] sukhe: ok great, I'll kick it off now, will ping back here when it's done [17:24:39] thanks! [17:27:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11895651 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cc7686ab-d152-4291-9303-296008017c88) set by cmooney@cumin1003 for 1:00:00 on... [18:19:14] sukhe: and traffic-opsen.... all done with the switch in rack 23. it gave me a few surprises so took a little longer than I'd expected [18:26:58] topranks: thanks! [18:29:51] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11895844 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host lvs4010.ulsfo.wmnet with OS bookworm [18:32:00] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11895869 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie [18:32:33] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11895870 (10ssingh) [18:50:40] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11895904 (10Nylki) Hi @daniel, thanks for the update :) [18:57:09] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11895915 (10Bbbub) Thank you for the update, I'll be happy to report back if I observe any changes after tomorrow. What do you mean by > This... [19:15:03] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11895979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host lvs4010.ulsfo.wmnet with OS bookworm completed: - lvs4010 (**PASS*... [19:23:36] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11896006 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie completed: - cp4050 (**PASS**)... [19:31:07] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11896023 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host lvs4010.ulsfo.wmnet with OS trixie completed: - lvs4010 (**WA... [19:44:03] 10Domains, 07HTTPS, 10DNS, 06SRE, 06Traffic-Icebox: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071#11896068 (10Novem_Linguae) Should this be closed in favor of {T205378}? Seems like pretty soon browsers and servers will support technology that co... [19:46:09] 07HTTPS, 06Traffic, 06SRE, 06Traffic-Icebox, 07Upstream: Support Encrypted Client Hello (ECH) on Wikimedia servers - https://phabricator.wikimedia.org/T205378#11896070 (10Novem_Linguae) [22:14:58] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS serving an older revision of some overwritten files - https://phabricator.wikimedia.org/T425216#11896466 (10AlexisJazz) >>! In T425216#11895597, @ssingh wrote: >>>! In T425216#11887650, @TheDJ wrote: >> I believe there i... [22:48:29] 06Traffic, 06SRE: "Nahui Ollin" is enwiki's #1 article. Never heard of it? That's the problem - https://phabricator.wikimedia.org/T425600 (10AlexisJazz) 03NEW [23:22:45] 06Traffic, 10Pageviews-Anomaly, 06SRE: "Nahui Ollin" is enwiki's #1 article. Never heard of it? That's the problem - https://phabricator.wikimedia.org/T425600#11896663 (10AntiCompositeNumber)