[05:20:08] 06Traffic: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11676432 (10BrokenImages1234) [05:25:35] 06Traffic: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11676434 (10BrokenImages1234) @ssingh Have just tried it in Brave and still got the same error: ` Failed to load resource: the server responded with a status of 429 () ` [06:21:43] 10netops, 06Infrastructure-Foundations, 06SRE: Update esams network pop diagrams - https://phabricator.wikimedia.org/T368084#11676461 (10ayounsi) LGTM thanks ! [08:28:25] 06Traffic: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11676571 (10JanTH) At first I thought this was just me using an exotic browser and getting misclassified as a bot, but I'm encountering the same error even in Edge, no third-party cookie blocking as far as I'm aware.... [11:51:13] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#11677162 (10JMeybohm) [12:27:32] 06Traffic, 13Patch-For-Review: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11677267 (10matmarex) Copying my comment from : I figured out where to find the data about... [12:42:20] 06Traffic, 13Patch-For-Review: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11677338 (10Ladsgroup) Note that we are being hammered by scrapers. On non-standard sizes, we are basically issuing 12 429s for each 200 (majority of the 429s are with zero or negative browser sc... [13:05:32] 06Traffic, 13Patch-For-Review: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11677384 (10Nemo_bis) I don't know if the chart above can be broken down by size, but this is not just about non-standard sizes. For example https://en.wikipedia.org/wiki/Road_signs_in_Finland ne... [13:33:19] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11677459 (10ABran-WMF) 05Stalled→03In progress [14:28:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11677654 (10Papaul) [15:44:37] 10Wikimedia-Apache-configuration, 06ServiceOps new, 06SRE, 10Wikibase GraphQL, and 2 others: Create a rewrite for the GraphQL endpoint on wikidata.org - https://phabricator.wikimedia.org/T417026#11678061 (10Blake) Hey folks, would it be possible to get some more detail about what assistance is required her... [16:18:42] 10Wikimedia-Apache-configuration, 06ServiceOps new, 06SRE, 10Wikibase GraphQL, and 2 others: Create a rewrite for the GraphQL endpoint on wikidata.org - https://phabricator.wikimedia.org/T417026#11678196 (10Silvan_WMDE) Thanks for getting back on this. Yes, we need help with applying the rule, please: does... [16:43:15] 06Traffic, 06SRE: Image Rate Limiting Issues For Future Audiences Project - https://phabricator.wikimedia.org/T418377#11678305 (10derenrich) given the recently announced increase in rate limiting i think this ticket is more urgent [16:55:03] 06Traffic, 06SRE: Image Rate Limiting Issues For Future Audiences Project - https://phabricator.wikimedia.org/T418377#11678360 (10CDanis) > The app involves fetching a large number of apps to present to the user. I'd definitely like to unblock development, but, I don't want to set us up for future trouble he... [16:57:27] hey Traffic folks! Could someone take a quick look at the following change? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1240197/comments/3e8760ca_4d0eb77e We're aiming to test it first on gerrit-spare and then gerrit-replica on Monday. (there is a bit more detail on the test in the linked comment) [16:57:27] With the new config applied, we will then remove that line and the next: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/trafficserver/backend.yaml#135 [16:57:27] to check whether anything is still triggering the bug described in https://phabricator.wikimedia.org/T417998 [16:59:05] arnaudb: will try to but maybe later [17:00:17] thanks sukhe [17:04:47] 10Wikimedia-Apache-configuration, 06ServiceOps new, 06SRE, 10Wikibase GraphQL, and 2 others: Create a rewrite for the GraphQL endpoint on wikidata.org - https://phabricator.wikimedia.org/T417026#11678377 (10Blake) p:05Triage→03Medium [17:08:44] 10Wikimedia-Apache-configuration, 06ServiceOps new, 06SRE, 10Wikibase GraphQL, and 2 others: Create a rewrite for the GraphQL endpoint on wikidata.org - https://phabricator.wikimedia.org/T417026#11678384 (10Clement_Goubert) Hi :) Depending on exactly what you want this rewrite to do, it may be that an ap... [17:10:49] 06Traffic: HAProxy 3.0 fires http-after-response rules twice on 500s resulting on KC session state - https://phabricator.wikimedia.org/T419149 (10Vgutierrez) 03NEW [17:10:55] 06Traffic: HAProxy 3.0 fires http-after-response rules twice on 500s resulting on KC session state - https://phabricator.wikimedia.org/T419149#11678398 (10Vgutierrez) p:05Triage→03High [17:12:09] 06Traffic: HAProxy 3.0 fires http-after-response rules twice on 500s resulting on KC session state - https://phabricator.wikimedia.org/T419149#11678399 (10Vgutierrez) [17:31:29] 06Traffic, 06SRE: Image Rate Limiting Issues For Future Audiences Project - https://phabricator.wikimedia.org/T418377#11678447 (10derenrich) thanks for the reply. i think first it would be helpful to know what the current limits are. i've been given conflicting information. not being able to enforce the limit... [18:12:42] 06Traffic, 13Patch-For-Review: HAProxy 3.0 fires http-after-response rules twice on 500s resulting on KC session state - https://phabricator.wikimedia.org/T419149#11678567 (10Vgutierrez) 05Open→03Resolved [18:49:36] 06Traffic, 06MW-Interfaces-Team, 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06MediaWiki-Platform-Team (Radar), 07OKR-Work: haproxy: capture x-wmf-* headers in webrequest data set - https://phabricator.wikimedia.org/T417864#11678731 (10daniel) With T417780 deployed, you should be seeing dat... [18:56:29] 06Traffic, 06SRE: Image Rate Limiting Issues For Future Audiences Project - https://phabricator.wikimedia.org/T418377#11678748 (10CDanis) Apologies for the conflicting information, that's partially my fault. But all the limits we've been discussing so far have been the limits we apply to bot (non-human) traff... [19:29:21] 06Traffic, 06MW-Interfaces-Team, 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06MediaWiki-Platform-Team (Radar), 07OKR-Work: haproxy: capture x-wmf-* headers in webrequest data set - https://phabricator.wikimedia.org/T417864#11678858 (10Fabfur) Hi @daniel I can confirm I see the `x_is_browse... [19:31:50] 06Traffic, 06SRE: Image Rate Limiting Issues For Future Audiences Project - https://phabricator.wikimedia.org/T418377#11678861 (10HSwan-WMF) Hey Chris, Perhaps we can talk live about this. I'm concerned about you mentioning that there will be no version of the limits that can facilitate an acceptable UX. I t... [20:19:08] 06Traffic, 06MW-Interfaces-Team, 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06MediaWiki-Platform-Team (Radar), 07OKR-Work: haproxy: capture x-wmf-* headers in webrequest data set - https://phabricator.wikimedia.org/T417864#11679050 (10daniel) 05Open→03Resolved >>! In T417864#116788... [20:21:23] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11679068 (10daniel) 05In progress→03Resolved Confirmed... [20:22:02] 06Traffic, 06MW-Interfaces-Team, 06MediaWiki-Platform-Team (Radar), 07OKR-Work: haproxy: strip x-wmf-* headers from responses - https://phabricator.wikimedia.org/T417781#11679071 (10daniel) 05Open→03Resolved a:03daniel Done for x-wmf-ratelimit-class for now. Will revisit for a more general solution. [20:25:17] !issync [20:25:18] Syncing #wikimedia-traffic (requested by JJMC89) [20:25:19] No updates for #wikimedia-traffic [23:02:04] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11679713 (10bd808) p:05Triage→03High `lang=shell-session, counterexample bd808@deployment-cache-upload08.deployment-prep.eqiad...