[03:51:45] 06Traffic, 06SRE: "Backend fetch failed" on edit save - https://phabricator.wikimedia.org/T382790#11011741 (10BCornwall) Hi, @MGChecker! I apologize for the long delay in getting back to you on this. Would you say that this is still an issue since you opened the task? [03:54:53] 06Traffic, 06Commons, 10MediaWiki-Uploading, 06SRE: HTTP 503 error when uploading images on Wikimedia Commons - https://phabricator.wikimedia.org/T383274#11011752 (10BCornwall) @Underbar_dk Since some time has passed, have you observed similar difficulties with various sites on IPv6? Or would you say that... [03:57:10] 06Traffic, 06Commons, 06SRE: Backend fetch failed - https://phabricator.wikimedia.org/T383013#11011754 (10BCornwall) Hi, @Jeff_G! I apologize for the delay in getting to you on this - Would you say this was a transient issue or a persistent one? [04:00:27] 10Domains, 06Traffic: Park pay-for-edit and scam domains - https://phabricator.wikimedia.org/T380334#11011773 (10BCornwall) 05Open→03Resolved An earlier decision codified our desire to redirect to the "don't pay for edit domains" blog post and the domains have been redirected. [04:01:14] 06Traffic: Improve geo-maps file - https://phabricator.wikimedia.org/T380651#11011776 (10BCornwall) 05Open→03Invalid This ticket is quite old, so I'll close it as stale. Feel free to reopen if it develops more! [06:53:55] FIRING: [3x] MaxConntrack: Max conntrack at 99.32% on ncredir3004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [06:58:55] RESOLVED: [5x] MaxConntrack: Max conntrack at 96.08% on ncredir3004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [07:13:03] 06Traffic, 06Commons, 10MediaWiki-Uploading, 06SRE: HTTP 503 error when uploading images on Wikimedia Commons - https://phabricator.wikimedia.org/T383274#11011982 (10Underbar_dk) I have not seen similar problems in other sites, but I have not had the opportunity to test with Commons either, unfortunately. [09:08:02] 10Acme-chief, 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11012384 (10Vgutierrez) yes, I'm gonna add a systemd timer to take care... [09:32:05] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11012447 (10cmooney) Arelion came back to say they no longer see CRC errrors on their side: ` Please note we are not detecting errors in our interface on Dallas... [15:07:57] 06Traffic, 10MediaWiki-Platform-Team (Radar), 07SecTeam-Processed, 07Security: SUL Integration for eventyay (Wikimania virtual event platform) - https://phabricator.wikimedia.org/T378157#11013815 (10Tgr) HTTP 429 is some sort of Varnish-level rate limiting. [15:42:45] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11013994 (10cmooney) Not sure how to progress this one. Still see zero packet loss over the link, even running for a longer period (5 mins this time): ` cmooney... [15:43:58] 06Traffic, 06Experimentation Lab, 07Regression, 07xLab: Cookie “WMF-Uniq” has been rejected because it is in a cross-site context - https://phabricator.wikimedia.org/T395958#11013998 (10Milimetric) p:05Triage→03Medium @phuedx: got time to check in on where this is? @Vgutierrez: how's it going, how did... [15:45:48] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11014023 (10ssingh) > Arelion want to close the ticket as they see no issue. I asked that they don't. Perhaps for now we just leave eqsin depooled and the circ... [15:56:47] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11014121 (10cmooney) >>! In T399221#11014023, @ssingh wrote: > I think leaving eqsin depooled given that it is off-peak there and observing this for a few hours... [16:03:59] 10Acme-chief, 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11014150 (10bd808) Testing the new timer service out on deployment-acme-... [16:34:15] 10Acme-chief, 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11014274 (10Vgutierrez) I like the idea of using two passes instead of `... [17:08:06] 10Acme-chief, 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11014450 (10ssingh) >>! In T399419#11014274, @Vgutierrez wrote: > I like...