[00:46:05] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#11892483 (10Papaul)
[02:26:18] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11892598 (10Papaul) All the servers in rack 23 are online and ready for re-image. I tested the re-image on cp4038 and completed with no issues after @ayou...
[07:06:56] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10Observability-Logging: Allow IT Services to view inbound email logs - https://phabricator.wikimedia.org/T419906#11892865 (10MoritzMuehlenhoff) From the technical angle the solution with cloudproxy seems feasible, and aws-sigv4-proxy also has limited dependencies which...
[07:16:45] <elukey>	 o/
[07:16:59] <elukey>	 pki1002 works nicely, I checked logs for the past hours and I don't see any error etc..
[07:17:19] <elukey>	 I'll keep them monitored, and I'd say next week we reimage pki2002 as well?
[07:17:59] <moritzm>	 sounds good, given that pki1001 is still around we can always simply fall back to it in case of errors
[07:18:27] <moritzm>	 in that sense the inactivity of cfssl upstream is both a blessing and a curse :-)
[07:21:53] <elukey>	 we can simply depool pki1002 in case of issues, I filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1283552 to prep pki1001 for decom
[07:22:01] <elukey>	 we can probably do it on Monday
[07:23:05] <elukey>	 I pinged upstream for my patch but no reply
[07:23:14] <elukey>	 I see a few new commits though
[10:15:42] <wikibugs>	 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, and 3 others: Replace Spamassassin with Rspam for VRTS on Postfix - https://phabricator.wikimedia.org/T402260#11893589 (10ABran-WMF) Small update after another round of Pontoon testing. A few things changed while testing the patch: * Rs...
[11:39:36] <effie>	 moritzm:  (or anyone privy to idp details)
[11:39:56] <effie>	 we seem to use 1 redis server for both idp instances 
[11:40:01] <effie>	 (idp.yaml) 
[11:40:07] <effie>	 is that correct ?
[11:40:18] <effie>	 (both=eqiad & codfw)
[11:41:58] <moritzm>	 that's correct, yes. otherwise we wouldn't be able to switch the IDP between sites w/o losing the current users sessions
[11:42:48] <effie>	 context is that we are trying out redis8 and trixie, but those servers are on codfw 
[11:43:27] <moritzm>	 for testing we could configure idp-test2005 to use the new Redis 8 node
[11:45:45] <effie>	 that would be great, would you be interested to move the prod one too after the test has been completed?
[11:45:56] <effie>	 again on codfw 
[11:46:09] <effie>	 if not that is ok, we can wait till we upgradet the eqiad ones
[11:46:57] <moritzm>	 sure, happy to be early adopters
[11:51:22] <effie>	 lol 
[11:51:30] <effie>	 moving on to another drama 
[11:52:03] <effie>	 https://www.irccloud.com/pastebin/9BvgX2Hf/
[11:52:34] <effie>	 what is odd is that, bjensen and I started reimaging about 12' before simon 
[11:53:00] <effie>	 and now we are blocked on the downtiming part 
[11:53:39] <effie>	 slyngs: is your reimaging progressing as expected?
[11:54:33] <slyngs>	 I think so, let me check
[11:55:07] <slyngs>	 Yup, all good, but I'm running two, so the wait was perhaps a little longer
[11:55:48] <effie>	 we have a concurrency of 1 in the downtime cookbook 
[11:55:54] <effie>	 and I think we hit the mark at the same time 
[11:56:23] <slyngs>	 Both of mine are done with downtime and running Puppet now
[11:58:11] <slyngs>	 Last lock was released at 11:42
[12:40:33] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Traffic: POPs LVS : remove public vlan trunking - https://phabricator.wikimedia.org/T367732#11893920 (10ayounsi) a:05ssingh→03Papaul I think that cable got properly removed yesterday and this task can be closed. @papaul do you confirm?
[16:45:06] <wikibugs>	 10Packaging, 06Abstract Wikipedia team, 10function-evaluator, 06Infrastructure-Foundations, 03Abstract Wikipedia Fix-It tasks: Package rustc from forky for wikimedia-bookworm so we can use it in an image like abstractwiki-rust - https://phabricator.wikimedia.org/T425341#11895465 (10Jdforrester-WMF) To di...
[17:02:28] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11895570 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a4b7dc3f-da06-4cb4-8580-9dac41f4da23) set by sukhe@cumin1003 for 3 days, 0:00...
[17:27:55] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11895651 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cc7686ab-d152-4291-9303-296008017c88) set by cmooney@cumin1003 for 1:00:00 on...