[11:05:28] 06Traffic: Consider using the alternate chain of Google Trust Services certificates - https://phabricator.wikimedia.org/T398596#10999853 (10Vgutierrez) @CDanis our current measurement payload doesn't provide the hostname of the cp host that handled the measure requests, it is feasible to fetch it from the x-cach... [12:56:56] cdanis: re https://phabricator.wikimedia.org/T398596#10999853, do we have any kind of CI to test if javascript code is still working as expected? [13:39:24] it works in ::1, I'll send a MR your way :D [13:40:52] <_joe_> sukhe: please lmk when you reenable puppet [13:41:42] _joe_: ok, should be soon. I see your change. [13:41:52] <_joe_> nod [13:52:57] <_joe_> sukhe: if you reenable puppet, please let me know, I have to monitor some stuff [13:53:29] _joe_: sorry, I misread the above. I thought you want me to ping you _after_ [13:53:38] yes, I have re-enabled it, in the process of [13:53:44] <_joe_> ok that makes sense [13:54:03] -b11, cp[2027-2042] should be done [13:55:12] <_joe_> sukhe: np, I just saw data in turnilo I couldn't explain otherwise [13:56:52] cdanis: after https://gitlab.wikimedia.org/repos/sre/probenet/-/merge_requests/1 I guess I'll have to submit a CR to mediawiki/extensions/WikimediaEvents repo [14:07:27] vgutierrez: Correct, the probenet repo itself is more a curiosity [14:07:35] no CI btw :( [14:08:02] ack, I'll send the CR there in case somebody gets it reviewed before you're back [14:08:15] sorry about the pings and enjoy your PTO *<3 [14:09:00] no worries <3 [14:10:11] _joe_: all done. [14:10:23] <_joe_> sukhe: thanks [14:10:32] I have to roll out one more change. I will wait for you since it's not urgent. [14:15:09] 06Traffic: Googlebot Commons 429 throttling - https://phabricator.wikimedia.org/T398668#11000539 (10Joe) We have doubled the allowed rps rate per ip we allow to googlebot. Tentatively resolving, although I fear it might stumble into other rules. [14:23:16] 06Traffic: provide x-analytics data for http frontend in haproxy - https://phabricator.wikimedia.org/T399167#11000610 (10Fabfur) 05Open→03Resolved [14:27:54] win 44 [14:27:59] ugh [14:28:11] _joe_: any more changes from your end? [14:28:53] <_joe_> sukhe: nope [14:28:58] thanks [15:02:35] 06Traffic: Disable OCSP for GTS issued certificates - https://phabricator.wikimedia.org/T399079#11000857 (10ssingh) 05Open→03Resolved [15:02:51] 06Traffic, 13Patch-For-Review: Remove OCSP monitoring and related bits - https://phabricator.wikimedia.org/T399114#11000859 (10ssingh) 05In progress→03Resolved a:03ssingh [16:27:36] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11001359 (10RobH) [18:20:05] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11001862 (10cmooney) Just an update on this, we unfortunately did not have a spare optic of the right kind so dc-ops are ordering one with expedited delivery. W... [18:29:14] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11001879 (10VRiley-WMF) Hey @BCornwall I have swapped the cables, would you be able to test this again? (I was going to try to reimage it, but didn't know what version of bullseye to put on it, I wa... [19:43:04] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002108 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bullseye [20:07:46] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002158 (10BCornwall) Looks like that worked, it's booting PXE now. Thanks! [20:08:33] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002159 (10VRiley-WMF) Awesome! is this okay to close out? [20:08:39] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002160 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bullseye executed with errors: - lvs1017 (**FAIL**) - Removed f... [20:09:03] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002162 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bookworm [20:49:47] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002294 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bookworm completed: - lvs1017 (**PASS**) - Removed from Puppet... [20:51:09] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002298 (10BCornwall) [20:51:50] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11002300 (10BCornwall) 05In progress→03Resolved We're all set. Thank you for all your help, @VRiley-WMF! [20:54:04] 06Traffic, 10Liberica: Test/benchmark Mellanox NICs for LVS usage - https://phabricator.wikimedia.org/T399502 (10BCornwall) 03NEW [20:54:13] 06Traffic, 10Liberica: Test/benchmark Mellanox NICs for LVS usage - https://phabricator.wikimedia.org/T399502#11002322 (10BCornwall) p:05Triage→03Medium [23:03:52] 06Traffic, 06SRE: Traffic cache daemon restart scripts need some rework - https://phabricator.wikimedia.org/T346640#11002716 (10BCornwall) Do we use these scripts anywhere or are they just for operator convenience? i.e. do we still need them? This also begs the question: Do we want this sort of operation to o...