[01:23:41] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11506021 (10Papaul) ` Hi Papaul, During testing in our lab we noticed that SPT (Spanning Tree Protocol) packets are being counted as “in-error” packe... [06:23:15] 06Traffic, 06SRE: Wiki Education Dashboard being rate-limited for OAuth login and token fetching - https://phabricator.wikimedia.org/T414114#11506171 (10Joe) @Ragesoss I see you still get blocked from time to time; I will add an exception, per https://wikitech.wikimedia.org/wiki/Robot_policy#What_to_do_if_thes... [07:03:24] 06Traffic, 06SRE: Wiki Education Dashboard being rate-limited for OAuth login and token fetching - https://phabricator.wikimedia.org/T414114#11506217 (10Joe) 05Open→03Resolved p:05Triage→03High a:03Joe Exception added. I allowed a generous amount of requests; please let us know if you still run i... [09:22:11] 06Traffic, 10Beta-Cluster-Infrastructure: CDN edge HAProxy config broken by missing `/usr/share/GeoIP/proxy.mmdb` data file - https://phabricator.wikimedia.org/T414111#11506475 (10SLyngshede-WMF) a:03SLyngshede-WMF [09:30:10] 06Traffic, 10Beta-Cluster-Infrastructure: CDN edge HAProxy config broken by missing `/usr/share/GeoIP/proxy.mmdb` data file - https://phabricator.wikimedia.org/T414111#11506506 (10SLyngshede-WMF) @ssingh The issue is in the Lua code that opens the database file, but fails. It's a little weird, because the code... [09:40:44] 06Traffic, 10Beta-Cluster-Infrastructure: CDN edge HAProxy config broken by missing `/usr/share/GeoIP/proxy.mmdb` data file - https://phabricator.wikimedia.org/T414111#11506535 (10SLyngshede-WMF) Quick test revels that the error handling does NOTHING if the fail doesn't exist. [09:51:21] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11506551 (10cmooney) As you would expect the unmanaged mgmt switches do send STP frames ` A:cmooney@lswtest-d8-eqiad# bash network-instance mgmt tcpdu... [10:51:18] 06Traffic, 10MediaWiki-Action-API, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11506731 (10Xqt) [11:13:33] 06Traffic, 10MediaWiki-Action-API, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11506781 (10Xqt) [11:19:09] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11506805 (10taavi) [11:32:19] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11506876 (10Tgr) What user agent are you using? [11:44:33] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11506948 (10Joe) I'm not sure having your CI depend on external resources is a good policy; I encourage you to change that long-term, but anyways, we don't want to... [11:57:54] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11506993 (10Xqt) >>! In T414173#11506876, @Tgr wrote: > What user agent are you using? `pwb.py version` for a sample test taks gives ` Pywikibot: [https] wikimedi... [12:08:48] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507067 (10Xqt) I also get that blocker for my normal bot during running redirect.py script: ` >>> Talk:Licentiate (Pontifical Degree) <<< Links to: [[en:Talk... [12:31:14] 06Traffic, 06serviceops, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work, 13Patch-For-Review: api-gateway helm chart: rest routes should return retry-after when a rate limit applies. - https://phabricator.wikimedia.org/T405636#11507215 (10daniel) 05Open→03In progress [12:31:50] 06Traffic, 06serviceops, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work, 13Patch-For-Review: api-gateway helm chart: rest routes should return retry-after when a rate limit applies. - https://phabricator.wikimedia.org/T405636#11507235 (10daniel) a:05Hokwelum→03daniel [12:40:59] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507270 (10Joe) This user agent is not compliat with our user-agent policy: https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy... [13:32:06] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507457 (10Xqt) >>! In T414173#11507270, @Joe wrote: > This user agent is not compliat with our user-agent policy: > > https://foundation.wikimedia.org/wiki/Poli... [13:42:42] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507482 (10Fabfur) >>! In T414173#11507457, @Xqt wrote: >>>! In T414173#11507270, @Joe wrote: >> This user agent is not compliat with our user-agent policy: >> >... [13:48:07] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507521 (10Xqt) [13:55:53] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507526 (10Xqt) @Fabfur: Currently we have this UA: `version (wikipedia:de; User:Xqtest) Pywikibot/11.0.0.dev10 (g20136) requests/2.32.5 Python/3.13.0.final.0` W... [14:11:16] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507562 (10Fabfur) >>! In T414173#11507526, @Xqt wrote: > @Fabfur: Currently we have this UA: > `version (wikipedia:de; User:Xqtest) Pywikibot/11.0.0.dev10 (g201... [14:17:54] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507588 (10JAnD) >>! In T414173#11507562, @Fabfur wrote: > I think this is better, if you want to add an email as contact you can do it right after the URL, separ... [14:21:41] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507597 (10Fabfur) >>! In T414173#11507588, @JAnD wrote: >>>! In T414173#11507562, @Fabfur wrote: >> I think this is better, if you want to add an email as contac... [14:35:08] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507634 (10Xqt) @Joe, @Tgr: Could you please consider postponing the newly introduced restriction until the Pywikibot User-Agent has been updated? As far as I kno... [15:07:24] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507718 (10revi) >>! In T414173#11507526, @Xqt wrote: > @Fabfur: Currently we have this UA: > `version (wikipedia:de; User:Xqtest) Pywikibot/11.0.0.dev10 (g20136... [15:14:18] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507752 (10Joe) >>! In T414173#11507634, @Xqt wrote: > @Joe, @Tgr: Could you please consider postponing the newly introduced restriction until the Pywikibot User-... [15:18:58] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11507766 (10Papaul) ` Thanks for confirming. We recommend blocking those packets, if possible, using dst mac 01:80:c2:00:00:00 on the mgmt switch, whi... [15:19:36] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11507767 (10Joe) To clarify: - Users on toolsforge or cloud VPS are exempt from the limit - I only see about 5% of all requests with UA containing `User:XXX`... [15:20:10] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507771 (10Jgiannelos) [15:20:42] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507776 (10Jgiannelos) @Wikiwan can you provide some examples of failing requests/responses? [15:20:55] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507777 (10Wikiwan) Yes, we believe this is the cause but we don't understand why - we are using this user agent for years [15:23:51] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507796 (10Jgiannelos) Without some steps to reproduce the issue it would be difficult to debug this on our end [15:29:23] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507805 (10Wikiwan) Potential reason is a wild AI scraping bots attacks we are dealing with the last couple of months, we blocked them all 2 weeks ago but that p... [15:31:28] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507815 (10Wikiwan) Very easy top reproduce, any uncached article would resolve in 429 which we than redirect automatically to the original Wikipedia article, fo... [15:33:14] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507828 (10Joe) 05Open→03Invalid @Wikiwan I guess you got some message with the 429 responses, right? And those messages didn't suggest to come open a ta... [15:36:49] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507846 (10Wikiwan) Hi Joe, I have, and was suggested to open this ticket [15:44:51] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507898 (10Jgiannelos) @Wikiwan I suggested to file a ticket so folks from other teams who are responsible for that part of the stack can provide more inform... [15:51:48] 06Traffic, 06Content-Transform-Team, 10Page Content Service: Wikiwand - mobile-html end point blocked - https://phabricator.wikimedia.org/T414204#11507955 (10Joe) >>! In T414204#11507846, @Wikiwan wrote: > Hi Joe, > I have, and was suggested to open this ticket No, the block provides an email to contac... [16:04:24] I guess you started enforcing stricter UA-policy quite recently? [16:05:23] (Just confirming before contacting discord.com about their opengraph image fetching proxy being noncompliant) [16:06:10] revi: yea, tell people to email the email address shown in the error message, if they get a 429 [16:07:00] ACK :-) [16:07:00] revi: eh, I mean.. only if they are already following the robot policy [16:07:04] https://wikitech.wikimedia.org/wiki/Robot_policy [16:07:09] Probably not I guess [16:07:33] Otherwise they wouldn't be blocked [16:07:39] revi: ok, so first send them to https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy [16:07:56] Yup, both I am aware of :-) [16:18:17] Uhm yeah corporate autoresponder so fun https://usercontent.irccloud-cdn.com/file/vGLD8jCB/IMG_0136.PNG [16:20:06] <_joe_> revi: thanks, but also, we can for sure unblock discord [16:20:19] <_joe_> it's to show thumbnails to humans I guess? [16:20:24] Yeah I think? [16:20:50] <_joe_> revi: do you have any idea what UA they use? [16:21:04] That one I have no idea tbh [16:21:16] <_joe_> discordbot I guess [16:23:11] https://commons.wikimedia.org/wiki/Special:Redirect/page/177976798 should have some entries, about 10 minutes ago via some IP in Korea and then from discord [16:23:36] (If you want to confirm) [16:24:25] <_joe_> uhm they are mostly blocked by a rule created on cache-upload since december [16:24:44] <_joe_> you're talking about images right? [16:24:56] Yup [16:25:26] <_joe_> yeah [16:25:31] <_joe_> ok it's easy to fix then [16:25:33] Poops instead of the image https://usercontent.irccloud-cdn.com/file/hMJpe4B6/IMG_0932.PNG [16:25:45] <_joe_> let me try [16:26:43] <_joe_> but yes, this has been happening since dec 24th [16:27:02] Heh interesting, didn't notice this until today [16:27:54] <_joe_> it's happening to about 1 in 2 requests from them [16:28:19] <_joe_> ok, in ~ 2/3 minutes, they should be unblocked [16:28:24] <_joe_> same for slack actually lol [16:28:32] Lol [16:28:45] Nobody at wmf slack noticed this would be interesting [16:29:09] <_joe_> I did notice [16:29:14] Heh [16:29:21] <_joe_> but was too busy to solve it [16:29:32] <_joe_> this was the result of SREs trying desperately to survive the festivities [16:29:45] <_joe_> which we failed to do, we were paged constantly through that period [16:29:48] (Not that I ever had access to wmfslack — I left before the slack era :-p) [16:29:56] Yeah end-of-year season is a hell anyway [16:30:01] <_joe_> revi: we still miss you though <3 [16:30:20] <_joe_> can you check with another image on discord? [16:30:25] Lemme seee [16:30:56] <_joe_> data tells me it's unblocked [16:32:12] Yeah lgtm https://usercontent.irccloud-cdn.com/file/dC5V8I80/IMG_0933.PNG [17:10:55] 06Traffic, 06SRE, 13Patch-For-Review: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11508207 (10Fabfur) We're now allowing this new type of contact information in User-Agent string, this change should be propagated shortly. P... [17:19:26] 06Traffic, 07Essential-Work, 05MW-1.46-notes (1.46.0-wmf.5; 2025-12-02), 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 17)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11508242 (10Sfaci) >>! In T407570#11505162,... [18:35:50] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11508591 (10cmooney) If we configured the mgmt switches we'd just have spanning-tree portfast or something on access interfaces, so they wouldn't be s... [18:56:17] 06Traffic, 10AutoWikiBrowser: 429 "too many requests" while requesting "what transcludes page" for many templates in AWB - https://phabricator.wikimedia.org/T414214#11508648 (10Reedy) [19:14:36] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11508669 (10Papaul) I think the quick fix here is for us to go with your option (2) exclude any interface called "mgmt0" for the time being and when... [20:07:04] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11508763 (10SerDIDG) Maybe it's somehow related. I use [[https://github.com/siddharthvp/mwn|siddharthvp/mwn]] for deploying my gadget. But a couple of days ago my... [20:19:54] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11508769 (10taavi) >>! In T414173#11508763, @SerDIDG wrote: > Maybe it's somehow related. I use [[https://github.com/siddharthvp/mwn|siddharthvp/mwn]] for deployin... [20:56:12] 06Traffic, 10AutoWikiBrowser: 429 "too many requests" while requesting "what transcludes page" for many templates in AWB - https://phabricator.wikimedia.org/T414214#11508813 (10Reedy) This is probably going to need some behaviour changes on the AWB side, and a followup release...