[04:09:21] <wikibugs>	 06Traffic: Images randomly fail to load - https://phabricator.wikimedia.org/T418323#11666449 (10BrokenImages1234) ` GET https://upload.wikimedia.org/wikipedia/commons/thumb/4/49/Kaiten-zushi_005.jpg/500px-Kaiten-zushi_005.jpg NS_BINDING_ABORTED  Status 429 Version HTTP/2 Transferred 939 B (0 B size) Referrer Pol...
[06:51:51] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11666554 (10ayounsi) From JTAC after sharing with them our gNMIc config : > I was able to replicate the issue in the lab. I've also shared the same update with engineering....
[07:19:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11666586 (10ops-monitoring-bot) Draining ganeti1051.eqiad.wmnet of running VMs
[07:43:51] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11666621 (10JMeybohm)
[08:10:44] <wikibugs>	 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11666664 (10elukey) I am finally able to query a week worth of IPs from webrequest and dump a txt file on t...
[08:25:31] <moritzm>	 vgutierrez, XioNoX: sorry, got distracted yesterday, since lvs1013 is just an experimental host, shall I simply powercycle it? then we can see if it's some NIC issue which solves itself on reboot. the server is from 2017 after all
[08:27:00] <XioNoX>	 moritzm: what's the issue?
[08:28:51] <volans>	 that the server is from 2017 :D 
[08:29:15] <XioNoX>	 true, right now I'd say, why not decom it?
[08:29:20] <moritzm>	 XioNoX: from yesterday: https://paste.debian.net/hidden/7b3b268e
[08:29:39] <XioNoX>	 oh wow I have a short memory :)
[08:29:40] <moritzm>	 TTBOMK these are used for some Liberica/Katran tests still
[08:30:06] <moritzm>	 XioNoX: I'll send you a Tiktok message next time :-)
[08:30:12] <XioNoX>	 hahahaha
[08:30:33] <XioNoX>	 yeah maybe decom it and re-use a less old decomed host if testing is still needed
[08:32:31] <moritzm>	 let's wait for Valentin to chime in, he's probably the last one to use it for tests
[08:32:48] <XioNoX>	 for sure
[08:40:45] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11666770 (10JMeybohm)
[08:44:07] <vgutierrez>	 yes please go ahead moritzm 
[08:47:44] <moritzm>	 doing that now
[08:52:48] <moritzm>	 all fine after the powercycle
[08:53:20] <wikibugs>	 06Traffic: Images randomly fail to load - https://phabricator.wikimedia.org/T418323#11666816 (10ihurbain) The whole thing, bar a couple of redacted elements (for another image on officewiki):  ` GET   scheme  https host  upload.wikimedia.org filename  /wikipedia/commons/thumb/8/86/WMF_homepage_banner_image_01.pn...
[09:35:32] <wikibugs>	 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11666985 (10ABran-WMF) >>! In T417998#11639695, @ABran-WMF wrote: > This is currently blocked...
[09:36:21] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11666990 (10MoritzMuehlenhoff)
[09:54:34] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: MediaViewer (and the commons file page) should serve WebP originals not thumbnails of equivalent size - https://phabricator.wikimedia.org/T418745#11667054 (10MatthewVernon) Perhaps instead for the odd non-web format (which seem...
[09:57:16] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11667074 (10BTullis)
[11:54:53] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: MediaViewer (and the commons file page) should serve WebP originals not thumbnails of equivalent size - https://phabricator.wikimedia.org/T418745#11667612 (10Ladsgroup) >>! In T418745#11665357, @Tacsipacsi wrote: > TIFFs (e.g....
[14:20:28] <brouberol>	 vgutierrez: qq about ATS backend caching. If i'm visiting https://turnilo-next.w.o and still getting the infamous landing page, does that refresh the 24h cache on the cp host I landed on?
[14:20:52] <vgutierrez>	 24h is the max TTL that the CDN will allow
[14:21:30] <brouberol>	 so once a page is cached, a new hit does not reset the cache time then
[14:22:14] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11668230 (10Ladsgroup) Top "file formats" for the non-standard sizes with enwiki as referrer are as follows: ` spark-sql (default)> s...
[14:22:49] <brouberol>	 if that's indeed the case, the turnilo-next.w.o page has been cached by ATS for more than 24h, and I'm still getting redirected to the landing page sometimes, and sometimes to the idp-test auth page (which is expected)
[14:23:02] <brouberol>	 and then for each asset (css, js, etc) it's a roll of the dice
[14:23:37] <brouberol>	 s/has been cached by ATS for more than 24h/was first cached in ATS > 24h ago/
[14:24:29] <brouberol>	 https://wikitech.wikimedia.org/wiki/Kafka_HTTP_purging#One-off_purge seems to only work for wiki pages according to c.laime
[14:24:41] <brouberol>	 so, I'm wondering what my options are atm
[14:27:09] <brouberol>	 I can see the change going through puppet roughly 24h ago https://puppetboard.wikimedia.org/report/cp7015.magru.wmnet/544038a0f5516590224008140e531c7755824a68
[14:28:28] <brouberol>	 what's even more surprising to me is that uncached URLs (assuming we differentiate URLs in the cache based on query args) such as https://turnilo-next.wikimedia.org/?abcd=1 still redirect me to the landing page
[14:29:30] <cdanis>	 brouberol: is caching even enabled in ATS for that backend?
[14:30:03] <cdanis>	 oh huh it is caching: 'normal'
[14:30:10] <brouberol>	 it should be yes, according to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247013
[14:30:34] <cdanis>	 I don't think what claime told you is true
[14:31:32] <brouberol>	 for additional datapoints, 
[14:31:33] <brouberol>	 brouberol@cp6009:~$ curl https://turnilo-next.discovery.wmnet:30443 # shows idp redirection
[14:31:33] <brouberol>	 <script src="portal/wikimedia.org/assets/js/index-56d76e0e59.js"></script></html>brouberol@cp6009:~$ curl -s -v -H 'Host: turnilo-next.wikimedia.org' http://turnilo-next.wikimedia.org:3128 # shows the landing page html
[14:31:55] <brouberol>	 sorry, disregard the <script>, bad copy paste
[14:33:45] <cdanis>	 try a purge imo
[14:34:43] <brouberol>	 I have tried `echo 'https://turnilo-next.wikimedia.org' | mwscript-k8s --attach -- purgeList.php`, with http and https, with and without trailing slash
[14:39:12] <vgutierrez>	 hmmm let me see
[14:41:30] <vgutierrez>	 so cache in drmrs is still poisoned
[14:41:52] <volans>	 is it possible something is misconfigured somewhere so that the cache gets re-poisoned?
[14:42:18] <brouberol>	 FWIW, I can see the following config line in all cp hosts
[14:42:22] <brouberol>	 "/etc/trafficserver/remap.config:map http://turnilo-next.wikimedia.org https://turnilo-next.discovery.wmnet:30443"
[14:44:28] <vgutierrez>	 so what it got from mw.eqiad includes `-   RespHeader     cache-control: s-maxage=3600, must-revalidate, max-age=0   `
[14:45:14] <brouberol>	 I ran `cumin 'cp7*' 'curl -s https://turnilo-next.discovery.wmnet:30443 2>&1 | grep idp-test'` for each DC, and it worked 100% of the time, so the issue really seem to be scoped to ATS, somehow 
[14:45:42] <brouberol>	 (or to how I (mis)configured caching)
[14:48:25] <vgutierrez>	 https://www.irccloud.com/pastebin/WR3SYE1z/
[14:48:44] <vgutierrez>	 we got 41 ATS instances returning the expected thing and 22 poluted
[14:50:03] <brouberol>	 huh, and it's not just tied to specific zones. Say, If i had smashed the ~like~ refresh button all day, my requests should have gone to drmrs
[14:50:26] <brouberol>	 which would have polluted the cache of hosts in a a specific pop
[14:50:28] <vgutierrez>	 always to the same host in drmrs
[14:50:36] <brouberol>	 right, even more so
[14:50:53] <brouberol>	 (it's heartwarming to think I have a place of my own in Marseille)
[14:51:24] <brouberol>	 this is where I'll fully admit that I don't understand ATS enough to be able to have good intuitions
[14:51:51] <brouberol>	 I'm happy to wait an extra 24h without trying to connect
[14:54:53] <brouberol>	 naive q: is there a way to inspect/purge cached URLs on specific hosts, by eg popping a key in a redis/memcached, or is it fully in-memory?
[14:55:25] <vgutierrez>	 ATS is a hybrid in-memory/on disk cache daemon
[14:56:49] <vgutierrez>	 so the PURGE is working
[14:57:08] <vgutierrez>	 Date:2026-03-03 Time:14:56:33 ConnAttempts:0 ConnReuse:244 TTFetchHeaders:0 ClientTTFB:1 CacheReadTime:0 CacheWriteTime:0 TotalSMTime:1 TotalPluginTime:0 ActivePluginTime:0 TotalTime:1 OriginServer:mw-web-ro.discovery.wmnet OriginServerTime:1 CacheResultCode:TCP_MISS CacheWriteResult:FIN ReqMethod:GET RespStatus:200 OriginStatus:200 ReqURL:http://turnilo-next.wikimedia.org/ ReqHeader:User-Agent:curl/7.74.0 
[14:57:08] <vgutierrez>	 ReqHeader:Host:turnilo-next.wikimedia.org ReqHeader:X-Client-IP:- ReqHeader:Cookie:- BerespHeader:Set-Cookie:- BerespHeader:Cache-Control:s-maxage=3600, must-revalidate, max-age=0 BerespHeader:Connection:- RespHeader:X-Cache-Int:cp2027 miss RespHeader:Backend-Timing:D=233 t=1772549793246266
[14:57:18] <vgutierrez>	 but for some reason is hitting mw-web-ro
[14:57:33] <vgutierrez>	 ok...
[14:57:56] <vgutierrez>	 Mar 03 09:53:37 cp2027 trafficserver[1549]: [Mar  3 09:53:37.032] [ET_TASK 0] ERROR: [ReverseProxy] failed to add remap rule at /etc/trafficserver/remap.config line 41: failed to create instance for plugin /usr/lib/trafficserver/modules/conf_remap.so: Unknown plugin error
[14:58:35] <brouberol>	 ooh
[14:58:52] <brouberol>	 I take it that L41 is unrelated to turnilo-next ?
[14:59:18] <vgutierrez>	 yeah... it's gerrit-replica 
[15:00:33] <brouberol>	 cc arnaudb fyi
[15:00:45] <arnaudb>	 ack, reverting the patch right now
[15:00:47] <vgutierrez>	 let me create a CR really quick...
[15:01:04] <vgutierrez>	 so who decided to disable @pparam=proxy.config.http.server_session_sharing.match=none?
[15:01:22] <vgutierrez>	 arnaudb: stop please
[15:01:27] <arnaudb>	 ack
[15:02:25] <vgutierrez>	 so.. I see an additional # commenting out '@pparam=proxy.config.http.server_session_sharing.match=none' for gerrit-replica and gerrit-spare 
[15:02:29] <vgutierrez>	 is that intentional or a typo?
[15:03:17] <arnaudb>	 it's a replication of the workaround in the first move of gerrit behind CDN, looking for the original CR
[15:03:25] <vgutierrez>	 so that's a typo, thanks
[15:04:36] <arnaudb>	 yes it is a typo indeed
[15:06:45] <vgutierrez>	 ack, running puppet on A:cp
[15:08:06] <brouberol>	 question for later: why were things failing on *some* hosts and not all of them? My theory is that these patches were merged 15 min apart, so some hosts had the time to take the turnilo redirect into account and then failed on the gerrit one, and some tried to apply both at the same time and failed, due to gerrit
[15:08:23] <brouberol>	 due to the puppet splay
[15:09:59] <vgutierrez>	 yep
[15:10:01] <vgutierrez>	 timing issue
[15:15:43] <brouberol>	 vgutierrez: re-running your cumin command, I now get 100% "OK"
[15:15:53] <vgutierrez>	 pure luck
[15:15:58] <vgutierrez>	 puppet hasn't finished on A:cp
[15:16:25] <brouberol>	 sometimes, I get to get lucky
[15:19:23] <wikibugs>	 06Traffic: Images randomly fail to load - https://phabricator.wikimedia.org/T418323#11668554 (10Ladsgroup) >>! In T418323#11666816, @ihurbain wrote: > The whole thing, bar a couple of redacted elements (for another image on officewiki):  That is a different thing (non-standard thumb size on private wikis) which...
[15:36:49] <claime>	 cdanis: I read the php wrong, it does indeed send a PURGE request for non-wiki urls (cc brouberol )
[15:37:06] <brouberol>	 tx
[15:38:48] <cdanis>	 was pretty sure i had used it to purge trace.wikimedia.org once 😅
[15:40:01] <vgutierrez>	 yeah.. it was getting purged as expected
[15:40:11] <vgutierrez>	 no doubt about that
[16:57:09] <wikibugs>	 06Traffic: Images randomly fail to load - https://phabricator.wikimedia.org/T418323#11669169 (10ihurbain) Oops, sorry for hijacking the thread, then (want me to put that somewhere else?)  It seems to have fixed the main page (which is the one I was getting that from this morning), but not the contact list, for w...
[17:34:19] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE: Network drop errors with new codfw cp hosts - https://phabricator.wikimedia.org/T418527#11669357 (10BCornwall) `perf` shows a lot of the drops related to ats and purged:  ` [ET_NET 72]  747843 [025] 71562.924664: skb:kfree_skb: skbaddr=0xff4203b3c053fa00 rx_sk=(nil)...
[17:59:41] <wikibugs>	 06Traffic, 07OKR-Work, 06Test Kitchen (Experiment Platform Sprint 20): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11669479 (10ssingh) >>! In T407570#11665964, @Sfaci wrote: > Hi again @ssingh! >  > After discussing a bit the...
[19:28:28] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE: Network drop errors with new codfw cp hosts - https://phabricator.wikimedia.org/T418527#11670094 (10Papaul) @BCornwall I did some testing today to eliminate the fact that it is the Network card that is causing the issue by putting in another 10G network card in cp20...
[20:00:43] <jinxer-wm>	 FIRING: HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp6009 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=drmrs&var-instance=cp6009 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages
[20:02:51] <sukhe>	 hmmm
[20:03:30] <sukhe>	 Feb 11 03:44:08 cp6009 haproxykafka[3916544]: {"level":"error","error":"ssl://kafka-jumbo1017.eqiad.wmnet:9093/1017: Disconnected (af>
[20:03:32] <sukhe>	 fun
[20:03:42] <brett>	 disconnected AF
[20:03:44] <brett>	 :P
[20:03:55] <sukhe>	 it's other hosts there as well
[20:05:09] <sukhe>	 nothing after Feb 11
[20:05:15] <sukhe>	 but the graph indicates sometihng is up for sure
[20:05:43] <jinxer-wm>	 RESOLVED: HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp6009 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=drmrs&var-instance=cp6009 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages
[20:14:48] <sukhe>	 ok well
[20:34:09] <jinxer-wm>	 FIRING: [10x] LVSHighCPU: The host lvs1018:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1018 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU
[20:39:09] <jinxer-wm>	 RESOLVED: [10x] LVSHighCPU: The host lvs1018:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1018 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU
[20:43:07] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: MediaViewer (and the commons file page) should serve WebP originals not thumbnails of equivalent size - https://phabricator.wikimedia.org/T418745#11670534 (10Krinkle) >>! In T418745#11667612, @Ladsgroup wrote: >>>! In T418745#...
[20:55:20] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE: Network drop errors with new codfw cp hosts - https://phabricator.wikimedia.org/T418527#11670617 (10BCornwall) We reimaged cp7008 to see if the other hardware would also be affected. [[ https://grafana-rw.wikimedia.org/d/000000377/host-overview?forceLogin=true&from=...
[21:44:00] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 07Wikimedia-Incident: 503 Service Unavailable No server is available to handle this request. - https://phabricator.wikimedia.org/T418392#11670845 (10AlexisJazz) On 28 February at 13:16 there was another bump in 5xx haproxy errors paired  with...
[21:44:27] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE: Network drop errors with new codfw cp hosts - https://phabricator.wikimedia.org/T418527#11670848 (10BCornwall) 05Open→03Resolved a:03BCornwall @BBlack discovered the source of the issue: LLDP packets are being dropped by lldpd. See T418941 for a follow-up....
[22:31:49] <wikibugs>	 06Traffic: Images randomly fail to load - https://phabricator.wikimedia.org/T418323#11671049 (10Ladsgroup) If it's now a standard size then it's another rate limit and I don't know what's going on with those :( sorry.
[22:45:06] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: MediaViewer (and the commons file page) should serve WebP originals not thumbnails of equivalent size - https://phabricator.wikimedia.org/T418745#11671101 (10Ladsgroup) I like the idea. +2'ed the patch. Will backport all three...
[22:54:06] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: MediaViewer (and the commons file page) should serve WebP originals not thumbnails of equivalent size - https://phabricator.wikimedia.org/T418745#11671125 (10Ladsgroup) First, fixing the ones that are already explicitly settin...
[23:33:48] <wikibugs>	 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), and 2 others: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11671361 (10Tgr) Deployed again. Resulted in a large but short...