[00:59:30] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11292126 (10Krinkle) Misc notes from the remaining audits across Wiktionary, Wikisource, Commons, Wikidata and Wik...
[00:59:41] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11292127 (10Krinkle) 05Open→03Resolved
[01:01:06] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11292134 (10Krinkle)
[04:13:06] <wikibugs>	 06Traffic, 10observability: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826 (10tstarling) 03NEW
[04:17:20] <wikibugs>	 06Traffic, 06SRE, 06MediaWiki-Platform-Team (Radar): Have CDN edge set the `X-Request-Id` header for incoming external requests - https://phabricator.wikimedia.org/T221976#11292290 (10tstarling) {T407826} may be related.
[04:19:47] <wikibugs>	 06Traffic, 10observability: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11292292 (10tstarling)
[04:55:25] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11292306 (10RolandUnger) @Kinkle, es.wikivoyage.org: The code of https://es.wikivoyage.org/wiki/MediaWiki:Mobile.j...
[05:20:45] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11292317 (10RolandUnger) @Krinkle, de.wikivoyage.org: The "Karte" button is created by a Javascript and consists o...
[07:02:43] <jinxer-wm>	 FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3070&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[07:07:43] <jinxer-wm>	 FIRING: [21x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[07:17:43] <jinxer-wm>	 RESOLVED: [21x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[07:27:52] <wikibugs>	 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: PeeringBGPDown (instance cr3-eqsin:9804) - https://phabricator.wikimedia.org/T407833 (10LSobanski) 03NEW
[07:58:56] <wikibugs>	 06Traffic, 10bot-traffic-requests: Global block exception for AddDesc app - https://phabricator.wikimedia.org/T407706#11292633 (10Joe) Hi, I'm not sure this task is tagged with the right tags.  These addresses are part of google cloud compute, which we don not blanket-block at the CDN. We rather block a few ve...
[08:43:02] <wikibugs>	 06Traffic, 10bot-traffic-requests: Global block exception for AddDesc app - https://phabricator.wikimedia.org/T407706#11292764 (10Joe) For instance, I see some of the IP ranges from GCP are part of a [[ https://meta.wikimedia.org/wiki/Special:Log?type=gblblock&page=35.203.128.0/18 | global on-wiki block ]], so...
[08:48:20] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Nokia: add new switches in eqiad/codfw to monitoring and make 'active' - https://phabricator.wikimedia.org/T405558#11292788 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d70417af-8325-49e7-a880-7a0cd37bd2d2) set by cmo...
[09:16:38] <wikibugs>	 06Traffic, 10observability: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11292897 (10Vgutierrez) This is pretty weird, according to [RFC 9562](https://www.rfc-editor.org/rfc/rfc9562) for UUID v4, the third block should always start with a `4`.  A quick check suggests tha...