[00:16:06] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11896736 (10ssingh) [00:33:54] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS serving an older revision of some overwritten files - https://phabricator.wikimedia.org/T425216#11896751 (10ssingh) Yeah, no, I was wrong and I misunderstood the problem. I misread that the actual image also differs acro... [01:50:03] 06Traffic, 10Pageviews-Anomaly, 06SRE: "Nahui Ollin" is enwiki's #1 article. Never heard of it? That's the problem - https://phabricator.wikimedia.org/T425600#11896822 (10Nemoralis) Mobile percentage is 0.5% [02:11:18] 06Traffic, 10Pageviews-Anomaly, 06SRE: "Nahui Ollin" is enwiki's #1 article. Never heard of it? That's the problem - https://phabricator.wikimedia.org/T425600#11896825 (10AlexisJazz) I wonder if T420833 was a test run for this. [02:14:38] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS serving an older revision of some overwritten files - https://phabricator.wikimedia.org/T425216#11896827 (10AlexisJazz) >>! In T425216#11896751, @ssingh wrote: > Yeah, no, I was wrong and I misunderstood the problem. I m... [02:20:07] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS and others serving older revisions of overwritten files - https://phabricator.wikimedia.org/T425216#11896828 (10AlexisJazz) [02:31:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11896831 (10Papaul) [05:33:49] 06Traffic, 10Pageviews-Anomaly, 06SRE: "Nahui Ollin" is enwiki's #1 article. Never heard of it? That's the problem - https://phabricator.wikimedia.org/T425600#11896993 (10Bugreporter2) Is it a hoax? [05:38:20] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11896996 (10daniel) >>! In T413570#11895915, @Bbbub wrote: > Thank you for the update, I'll be happy to report back if I observe any changes aft... [05:52:20] 06Traffic, 10Pageviews-Anomaly, 06SRE: "Nahui Ollin" is enwiki's #1 article. Never heard of it? That's the problem - https://phabricator.wikimedia.org/T425600#11897035 (10AlexisJazz) >>! In T425600#11896993, @Bugreporter2 wrote: > Is it a hoax? Doesn't look like a hoax. One of the sources is "An Inglorious... [05:55:50] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (CommonsFinder iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11897040 (10daniel) [06:45:58] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (CommonsFinder iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11897051 (10Bbbub) Thank you for the quick response! I am also not connected to CommonsFinder, our app just happens to have encoun... [06:46:28] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897052 (10ops-monitoring-bot) VM ncredir4003.ulsfo.wmnet switching disk type to drbd [06:48:23] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897053 (10ops-monitoring-bot) VM ncredir4003.ulsfo.wmnet switching disk type to drbd [07:10:53] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (CommonsFinder iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11897097 (10Nylki) >>! In T413570#11896996, @daniel wrote: > With all this said: I don't see blocked requests with "CommonsFinder"... [07:12:48] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (native mobile apps, eg. CommonsFinder iOS app) - https://phabricator.wikimedia.org/T413570#11897099 (10Nylki) [07:15:06] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons in non-WMF native apps treated as bot-traffic - https://phabricator.wikimedia.org/T413570#11897103 (10Nylki) [07:15:36] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#11897104 (10Nylki) [07:30:29] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897115 (10ops-monitoring-bot) VM netflow4003.ulsfo.wmnet switching disk type to drbd [08:14:39] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897242 (10ops-monitoring-bot) VM ncredir4004.ulsfo.wmnet switching disk type to drbd [08:15:18] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897243 (10MoritzMuehlenhoff) [08:29:43] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897289 (10ops-monitoring-bot) VM prometheus4003.ulsfo.wmnet switching disk type to drbd [09:19:51] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897459 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast4006.wikimedia.org with OS trixie [09:32:48] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897502 (10ops-monitoring-bot) VM hcaptcha-proxy4003.wikimedia.org switching disk type to drbd [10:11:00] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11897596 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast4006.wikimedia.org with OS trixie completed: - bast4006 (**PASS*... [15:49:37] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS and others serving older revisions of overwritten files - https://phabricator.wikimedia.org/T425216#11899449 (10ssingh) ` sukhe@cumin1003:~$ sudo cumin "A:cp and not P{cp2041* or cp2042*}" "curl -s https://upload.wikimed... [19:46:34] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11900420 (10Epidosis) [23:07:00] 06Traffic: images are not loading for some users (on the us west coast?) - https://phabricator.wikimedia.org/T425670#11901162 (10Nil_NZ) Issue was no longer present when I woke up this morning NZT (~08:00 UTC)