[00:37:19] 06Traffic: images are not loading for some users (on the us west coast?) - https://phabricator.wikimedia.org/T425670#11901270 (10ForeverFlying) I'm on the East Coast and I think I have this issue, specifically when trying to replace invalid SVGs with valid SVGs on Wikimedia Commons. [00:57:32] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11901274 (10Yirba) @Epidosis Hi, what makes you say that QS 3.0 respects maxlag? I checked the source code and see no reference to maxlag. Maxlag has been abnormally hig... [05:49:55] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11901401 (10Mahir256) >>! In T421642#11793464, @Mahir256 wrote: > ...tools not respecting maxlag should be directed at @Arcstur in the former case... Pinging @ACorrea-W... [07:46:28] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11901486 (10karapayneWMDE) [07:48:07] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11901495 (10Yirba) The issue has got progressively worse over the past several hours, and max lag is now at around 40 minutes: {F80259637} I can't say with certainty th... [08:26:35] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#11901593 (10Bbbub) Thank you! Please let us know if this is also tracked or documented elsewhere so that we don't miss any developments [09:08:42] 06Traffic: images are not loading for some users (on the us west coast?) - https://phabricator.wikimedia.org/T425670#11901736 (10Aklapper) Again: What does happen? If you try to open one of those images themselves (not the `File:` page of an image), which error code do you get? [10:47:01] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11901977 (10Yirba) I wonder if someone did something; max lag seems to have returned to normal levels since around 10:00 UTC. [10:57:57] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network QoS: use the 'CS1' DSCP code point for low-priority instead of AF41 - https://phabricator.wikimedia.org/T424640#11902014 (10cmooney) Just an update here. The patch to map CS1 into low-priority has been rolled out across the network... [11:26:03] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11902069 (10Xqt) >>! In T421642#11901977, @Yirba wrote: > I wonder if someone did something; max lag seems to have returned to normal levels since around 10:00 UTC. I fe... [11:41:05] 10netops, 06Infrastructure-Foundations, 06SRE: Network QoS: use the 'CS1' DSCP code point for low-priority instead of AF41 - https://phabricator.wikimedia.org/T424640#11902144 (10cmooney) So it seems the reason for this is some ferm complexity. When puppet signals a 'refresh' to it it asks ferm itself to re... [11:44:34] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11902162 (10Lydia_Pintscher) See also T425758. [12:05:12] 06Traffic, 06SRE: wiki.openstreetmap.org Commons thumbs rate limit allowance - https://phabricator.wikimedia.org/T423570#11902254 (10Fabfur) a:05jcrespo→03Fabfur [12:39:25] 06Traffic, 06MediaWiki-Platform-Team: Error 429 for search queries and images in older browsers - https://phabricator.wikimedia.org/T425763#11902407 (10A_smart_kitten) [tagging MWP as well as Traffic as it [[https://gerrit.wikimedia.org/g/operations/deployment-charts/+/de52332be12c5cc5f40de783b33afd850e9cee1b/... [12:41:21] 06Traffic, 06MediaWiki-Platform-Team: Error 429 for search queries and images in older browsers - https://phabricator.wikimedia.org/T425763#11902414 (10Aklapper) > Open Wikipedia or Wiktionary in an old version of any browser Hi, what does that mean //exactly//? "old" and "any" make it hard to reproduce somet... [12:45:46] 06Traffic, 06SRE: wiki.openstreetmap.org Commons thumbs rate limit allowance - https://phabricator.wikimedia.org/T423570#11902427 (10jcrespo) I sent an email to Grant with additional data I can share, as allowed by staff SREs. [13:18:34] 06Traffic, 06SRE: wiki.openstreetmap.org Commons thumbs rate limit allowance - https://phabricator.wikimedia.org/T423570#11902575 (10mxn) >>! In T423570#11902251, @Fabfur wrote: > If not possible at the moment I suggest to modify the configuration of QuickInstantCommons to increase `descriptionCacheExpiry` and... [13:25:59] 06Traffic, 06SRE: wiki.openstreetmap.org Commons thumbs rate limit allowance - https://phabricator.wikimedia.org/T423570#11902578 (10Fabfur) >>! In T423570#11902575, @mxn wrote: >>>! In T423570#11902251, @Fabfur wrote: >> If not possible at the moment I suggest to modify the configuration of QuickInstantCommon... [13:41:18] 06Traffic, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Radar), 07OKR-Work: rest-gateway: configure comprehensive set of cg-nat IPs - https://phabricator.wikimedia.org/T424827#11902628 (10daniel) 05Open→03Resolved {F80292455} [14:12:39] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11902723 (10Arcstur) Hello, everyone, I'll share here some info regarding QS3 so you can help me understand if we are or not respecting it... I'll split it into parts.... [14:16:02] 06Traffic, 07OKR-Work, 10Test Kitchen (Experiment Platform Sprint 23): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11902724 (10Sfaci) While we discuss separately the privacy considerations that were raised during last meeting... [14:19:16] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11902733 (10Arcstur) I updated what I described above on Jan 24, seeing as the graph here shows the increase after 2026-03-25 15:00 UTC I wonder if they are really conne... [14:55:02] 06Traffic, 10Apertium, 06SRE: LVS IP for apertium.svc.eqiad.wmnet ends up in CheckUser XFF data - https://phabricator.wikimedia.org/T425783 (10Reedy) 03NEW [17:35:40] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11903249 (10Epidosis) Maxlag dropped from 51 minutes (!) to a normal ca. 5 seconds at 10 UTC today, but already shortly after 11 UTC it restarted a significant growth an... [18:56:30] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: BFD broken with default homer configuration - https://phabricator.wikimedia.org/T425813 (10cmooney) 03NEW p:05Triage→03High [21:20:20] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS and others serving older revisions of overwritten files - https://phabricator.wikimedia.org/T425216#11904016 (10AlexisJazz) I just noticed that for https://commons.wikimedia.org/wiki/File:Paul_de_Munnik_(2017).jpg this i...