[03:01:01] nothing to report from americas on-call [08:19:34] hello oncallers! I just moved all the maps traffic to codfw (on the new stack), we want to test if we have hopefully fixed some performance issues [08:52:44] elukey: plz use our nicks, at least mine :) [08:53:11] I'm not gonna configure my irc client to ping me on `oncallers` :) [08:56:58] vgutierrez: I just wanted to leave a track in case anything came up, not necessarily disturb you before coffee :) [08:57:26] if I run something dangerous I'll ping you directly :) [08:57:38] you should call 911 if it's 10:53 CEST and I'm running without caffeine on blood [08:57:52] even at 10:19 ;P [08:59:56] sometimes it is fine to take it easy and have a slower start if you worked 18 hours the previous day [09:08:57] elukey: you say it like 18h isn't the average [09:09:38] vgutierrez: wishful thinking [13:23:44] I want to note something- the "PyBal IPVS diff check" icinga check seems to have a parsing issue. Not sure if worth fixing it but FYI [13:26:23] what kind of parsing issue? [13:37:29] parsing issue? [13:37:45] jynus: ^^ could you elaborate please? [13:44:18] ('CRITICAL: Mismatch between IPVS and PyBal [13:44:28] it returns that, probably not intended [13:45:00] and that is probably because icinga is very picky about what characters it allows (no pipes and other stuff that gets intepreted specially) [13:45:23] but again, probably not worth spending time on that, as long as you are aware [13:45:31] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=lvs2013&service=PyBal+IPVS+diff+check [13:46:31] CRITICAL: Mismatch between IPVS and PyBal [13:46:31] ', "Services in IPVS but unknown to PyBal: set(['10.2.1.93:80', '10.2.1.94:80'])") [13:46:38] if there is time to spent it may probably be better on alertmanager [13:46:55] just wanted to make sure you were aware [13:47:20] jynus: it also returns the above information. do you mean the alert description itself? [13:47:31] the status information [13:47:41] yeah, this is a known alert and the alertmanager work is probably not worth it IMO because we will soon move over to liberica (eventually) [13:47:52] ok, thanls, that's all [13:48:32] this is wdqs-internal-main [13:48:35] I also brought it up because checks return weird stuff and there is ongoing work for a translator [13:49:01] which should be reliable against weird outputs, but thats for obs, not your team [13:49:33] I mean we can spend time on this but really, as soon as the k8s blocker goes away, vg will work on making Liberica everywhere including eqiad/codfw [13:49:39] so probably not worth the effort right now [13:49:42] indeed, I agree [14:53:40] moritzm: would you rather have us reimage urldownloaders to remove the hcatpcha bits or simply run Puppet? [14:53:43] cc Raine ^ [14:57:07] no need, I'll be reimaging them to bookworm anyway in the coming months, after merging your patch let's simply remove the nginx debs and call it a way [14:57:16] s/way/day [14:57:28] ok thanks [14:58:46] * Raine forgot to pass that message along, sorry [15:22:11] Hey folks! Please don't use cumin1002 anymore, the right host is cumin1003 :) 1002 will be deleted hopefully soon! [15:28:57] we run backup from cumin1002 [15:40:09] We still have to solve: https://phabricator.wikimedia.org/T393692 [15:42:08] that's independent [15:42:47] the issue with nft doesn't depend on where it's run, but to were it is sent [16:00:30] but I will now give both top priority [20:20:48] hey, is there an SRE online who I can quickly PM for a second opinion on something (for lack of knowing who else to ask)? [20:22:17] Amir1: do you mind if I PM you? (apologies for picking on you, I just noticed the banner says you're on clinic duty :p) [20:23:42] Sure. It's a bit late for me but I see what I can do [20:25:05] thx :) PMed