[08:54:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: mr1-codfw: add second uplink to lsw1-a2-codfw - https://phabricator.wikimedia.org/T410717#11491000 (10ayounsi) Nice, you can create the /31 prefix in netbox, similar to the other link : https://netbox.wikimedia.org/ipam/prefixes/890/ [09:40:38] 10netops, 06FR-donorrelations, 06Fundraising-Backlog, 10fundraising-tech-ops, and 2 others: Routing issue from payments hosts to 62.146.255.205 - https://phabricator.wikimedia.org/T413382#11491119 (10ayounsi) Looks like that IP is not part of the `payments-processors` address-set so it's not being NATed.... [09:41:17] 10netops, 06FR-donorrelations, 06Fundraising-Backlog, 10fundraising-tech-ops, and 2 others: Routing issue from payments hosts to 62.146.255.205 - https://phabricator.wikimedia.org/T413382#11491120 (10ayounsi) a:03Dwisehaupt [09:56:46] 06Traffic, 06MediaWiki-Platform-Team (Radar), 07SecTeam-Processed, 07Security: SUL Integration for eventyay (Wikimania virtual event platform) - https://phabricator.wikimedia.org/T378157#11491169 (10Dzahn) 05Resolved→03Open [11:30:26] 06Traffic, 10Beta-Cluster-Infrastructure: Add allowlist to make poking holes in abuse_networks:blocked_nets:networks easier - https://phabricator.wikimedia.org/T393481#11491418 (10TheresNoTime) Drive-by comment that the `./whoisit.sh` utility mentioned in https://wikitech.wikimedia.org/wiki/Nova_Resource:Deplo... [13:04:11] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons since a few days (iOS app) - https://phabricator.wikimedia.org/T413570#11491805 (10TheDJ) > The response body is empty when this happens That seems unlikely to me. Maybe your framework is one of those that doesn't handle response bodie... [13:17:14] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons since a few days (iOS app) - https://phabricator.wikimedia.org/T413570#11491836 (10Nylki) >>! In T413570#11491805, @TheDJ wrote: >> The response body is empty when this happens > > That seems unlikely to me. Maybe your framework is one... [13:17:52] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons since a few days (iOS app) - https://phabricator.wikimedia.org/T413570#11491837 (10TheDJ) For @Traffic, there have been quite a few 429 reports over the last two weeks. In discord, on Commons VP etc. might be worth checking if the last... [13:26:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp6012:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=drmrs%20prometheus/ops&var-instance=cp6012&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [13:31:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp6012:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=drmrs%20prometheus/ops&var-instance=cp6012&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [13:43:43] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons since a few days (iOS app) - https://phabricator.wikimedia.org/T413570#11492001 (10Joe) @TheDJ the 429s are probably due to the massive scraping activity SREs have had to deal with over the festivities. @Nylki I can ensure that when yo... [15:05:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: msw1-b6-codfw down - https://phabricator.wikimedia.org/T413715#11492257 (10Jhancock.wm) the msw didn't just go down. it died. getting a replacement out of storage. [15:24:59] 10netops, 06Infrastructure-Foundations: asw1-b12-drmrs stopped reporting metrics - https://phabricator.wikimedia.org/T413181#11492343 (10LSobanski) p:05Triage→03Medium a:03ayounsi [15:25:20] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons since a few days (iOS app) - https://phabricator.wikimedia.org/T413570#11492346 (10Nylki) hi @Joe and @TheDJ, Thanks for sticking with me! I created a minimal iOS sample app to reproduce the issue, this time with a body. If you like, yo... [15:55:19] @fabfur: just checking back about https://gerrit.wikimedia.org/r/c/operations/puppet/+/1218817 have you had a chance to look? [15:56:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: msw1-b6-codfw down - https://phabricator.wikimedia.org/T413715#11492530 (10Jhancock.wm) switch has been replaced. all alerts have cleared. netbox updated. closing other tickets. [15:56:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: msw1-b6-codfw down - https://phabricator.wikimedia.org/T413715#11492532 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [16:16:15] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11492636 (10MatthewVernon) [16:22:06] 06Traffic, 06Data-Persistence, 10MediaViewer, 10Page-Previews, and 2 others: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11492653 (10MatthewVernon) 05Open→03Resolved [16:23:22] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.7 Standardize thumbnail sizes - https://phabricator.wikimedia.org/T408062#11492666 (10MatthewVernon) [16:23:33] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11492673 (10MatthewVernon) 05Open→03Resolved [16:43:04] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Propose a new set of standard thumbnail sizes - https://phabricator.wikimedia.org/T412971#11492739 (10MatthewVernon) [16:43:33] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Propose a new set of standard thumbnail sizes - https://phabricator.wikimedia.org/T412971#11492740 (10MatthewVernon) In the light of the above discussion, I now propose only pre-generating the one size (that the user will want... [16:43:49] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons since a few days (iOS app) - https://phabricator.wikimedia.org/T413570#11492742 (10SuperHamster) Hi there. Chiming in as another external tool developer who also noticed the 429 issue. I develop [[ https://commons.gallery | https://comm... [17:24:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: mr1-codfw: add second uplink to lsw1-a2-codfw - https://phabricator.wikimedia.org/T410717#11492952 (10Papaul) 05Open→03Resolved Configuration done ` lsw1-a3-codfw# run show route receive-protocol bgp 10.192.254.15 inet.0: 99... [17:27:43] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11492967 (10Papaul) I sent a Follow up email to Nokia to ask if there were any updates. [17:30:46] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons since a few days (iOS app) - https://phabricator.wikimedia.org/T413570#11492984 (10SuperHamster) Did a bit more experimenting: If I go to https://commons.gallery in my local Firefox and run this in the console: ` fetch("https://upload... [18:00:04] 06Traffic, 10Beta-Cluster-Infrastructure: Add allowlist to make poking holes in abuse_networks:blocked_nets:networks easier - https://phabricator.wikimedia.org/T393481#11493167 (10bd808) >>! In T393481#11491418, @TheresNoTime wrote: > Drive-by comment that the `./whoisit.sh` utility mentioned in https://wikite... [18:45:26] 10netops, 06FR-donorrelations, 06Fundraising-Backlog, 10fundraising-tech-ops, and 2 others: Routing issue from payments hosts to 62.146.255.205 - https://phabricator.wikimedia.org/T413382#11493488 (10Dwisehaupt) @ayounsi I have a question for clarification. When we made the move to ipset based rules in ipt... [21:02:00] 10netops, 06Infrastructure-Foundations: access request - read-only access to pfw's for Avishua Stein (astein) - https://phabricator.wikimedia.org/T413826 (10AStein-WMF) 03NEW [23:53:46] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11494278 (10Papaul) ` Joao Passo (Nokia) 2:21 PM (3 hours ago) to me, supportservices@nokiacom Hello Papaul, I’ve requested an update to the R&D tea...