[00:40:09] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11679938 (10bd808) I can't find a Phab task for this, but there is are [[https://prometheus-alerts.wmcloud.org/?q=%40state%3Dactiv... [01:09:56] 06Traffic, 10MediaWiki-Revision-deletion, 06MW-Interfaces-Team: 404 error when using action=raw on an admin-level hidden revision - https://phabricator.wikimedia.org/T351688#11679999 (10Krinkle) `action=raw` is the backend for `importScript` and `importScriptURI` which is the officially supported way to load... [04:36:51] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#11680410 (10cmooney) Thinking it through, I think this process could be used to "drain" //ssw1-d1// if we wanted to attempt this without any impact to hosts. * Set //c... [04:41:30] 06Traffic: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11680413 (10BrokenImages1234) I don't get rate-limited anymore in Tor and Brave but still do in Firefox. Is it because I use an old version of it? [09:36:23] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11680717 (10elukey) Very weird: ` elukey@pki-intermediate:~$ openssl s_client pki-intermediate.pki.eqiad1.wikimedia.cloud:443 | o... [09:44:35] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11680743 (10elukey) Ok found the problem: ` elukey@pki-intermediate:~$ openssl s_client -connect pki-intermediate.pki.eqiad1.wiki... [10:15:36] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11680831 (10elukey) Ok the issue is: ` root@pki-puppetserver-1:/etc/puppet/puppetserver/ca# ls -l total 36 -rw-r--r-- 1 puppet pu... [13:09:11] 06Traffic: Images randomly fail to load (Error 429) - https://phabricator.wikimedia.org/T418323#11681573 (10Aklapper) Many factors come into play; one of them is using old browser versions. Please update if possible. [14:05:16] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026-03-06 - 2026-03-27), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11681951 (10Gehel) [14:28:37] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 3 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11682215 (10TheDJ) [14:29:02] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 3 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11682219 (10TheDJ) [14:41:42] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 3 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11682308 (10MatthewVernon) @Ladsgroup they're only a tiny number of files, but XCF will probably likewise need addressing? [15:05:14] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11682485 (10Andrew) @elukey I doubt that I'll be of much help here. If the key isn't present on the puppetserver itself then it is... [15:42:34] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 3 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11682595 (10Aklapper) [16:30:59] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11682744 (10Volans) This host is still marked as Active in Netbox but disappeared from PuppetDB, if it's still broken please fix its status in Netbox. [16:35:29] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11682781 (10bd808) >>! In T419099#11682485, @Andrew wrote: > I also don't really understand what's going on with the cross-project... [16:36:18] 06Traffic: Upgrade to HAProxy 3.0 on cache (bullseye) hosts - https://phabricator.wikimedia.org/T417253#11682785 (10Vgutierrez) 05Open→03Resolved [16:38:26] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263 (10A_smart_kitten) 03NEW [16:39:44] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11682806 (10elukey) @Andrew I asked since from the logs you created the puppetserver vm keeping the old certs in place, so I was w... [16:59:43] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11682901 (10A_smart_kitten) [17:01:23] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11682914 (10sbassett) Do we need anything beyond api.flickr.com? Ideally, we'd like... [17:03:23] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11682929 (10A_smart_kitten) (I defer to other folks on that question, I do not have e... [17:08:52] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11682946 (10ShakespeareFan00) Another question : Can the CSP limit the requestable m... [17:33:05] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11682990 (10RobH) Set to failed. Ordering of replacement part will take place today or tomorrow, sorting the terms. [19:46:39] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11683424 (10sbassett) We are now supporting `api.flickr.com` within the enforcing CSP... [19:57:15] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11683447 (10A_smart_kitten) FWICS it seems to now work for me (I didn't fully upload... [19:57:37] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11683449 (10taavi) So for longer term: given that this functionality from an extensio... [20:35:07] 06Traffic, 10UploadWizard, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07Regression: UploadWizard on Wikimedia Commons no longer able to import files from Flickr - https://phabricator.wikimedia.org/T419263#11683585 (10A_smart_kitten) >>! In T419263#11683447, @A_smart_kitten wrote: > I [[htt...