[00:20:00] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 06DC-Ops, and 2 others: eqiad: request for a decom'ed R440 - Config C - https://phabricator.wikimedia.org/T418544#11679905 (10VRiley-WMF) Re-labeled moss-fe1002 to contint1003 Racked it in B1 U36 CableID: 3720 Po...
[00:40:09] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11679938 (10bd808) I can't find a Phab task for this, but there is are [[https://prometheus-alerts.wmcloud.org/?q=%40state%3Dactiv...
[00:58:58] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11679953 (10Daimona) >>! In T419107#11678857, @Jdforrester-WMF wrote: > @daimona: Do we want...
[04:07:29] <wmcs-alerts>	 FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[05:01:58] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11680417 (10tstarling) Another failure after t...
[06:13:52] <wikibugs>	 (03PS1) 10Clare Ming: Remove Metrics Platform extension [integration/config] - 10https://gerrit.wikimedia.org/r/1248693 (https://phabricator.wikimedia.org/T417568)
[07:47:07] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11680548 (10ArthurTaylor) @Daimona thanks for...
[07:56:22] <hashar>	 bd808: thanks for the explanation about AI output not being copyrightable in the US (so far).  And yes disclosing is very much welcome.
[07:57:05] <hashar>	 sometime I feel it is on the same vein as using an IDE to refactor code instead of manually doing all the search & replace
[07:57:22] <hashar>	 or auto fixing lints/styles issues
[08:33:08] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11680590 (10ArthurTaylor) @tstarling I've trie...
[08:52:57] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11680621 (10ArthurTaylor) Okay - now I have an...
[09:06:47] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11680648 (10ArthurTaylor) ...and I can confirm...
[09:31:39] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11680707 (10ArthurTaylor) So I think I can say...
[09:36:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11680717 (10elukey) Very weird:  ` elukey@pki-intermediate:~$ openssl s_client pki-intermediate.pki.eqiad1.wikimedia.cloud:443 | o...
[09:44:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11680743 (10elukey) Ok found the problem:  ` elukey@pki-intermediate:~$ openssl s_client -connect pki-intermediate.pki.eqiad1.wiki...
[09:58:00] <wikibugs>	 10Continuous-Integration-Config, 06Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 07Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238#11680758 (10Xqt) >>! In T346238#11555772, @TheDJ wrote: > Seems this stalled upstream ?  It i...
[10:15:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11680831 (10elukey) Ok the issue is:  ` root@pki-puppetserver-1:/etc/puppet/puppetserver/ca# ls -l total 36 -rw-r--r-- 1 puppet pu...
[11:09:38] <wikibugs>	 (03CR) 10Arthur taylor: [C:03+1] "@hashar@free.fr Am I write in thinking this still isn't going to capture the runtime of individual split groups? Just the total time of th" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar)
[11:11:32] <wikibugs>	 (03PS1) 10Okerekechinweotito: Zuul: Add Okereke Chinweotito personal e-mail to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1248788
[11:33:47] <wikibugs>	 (03CR) 10Arthur taylor: [C:03+1] "ah. yeah. Just looked again at how that's implemented, and of course Quibble doesn't really know anything about the split groups, so there" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar)
[11:52:28] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I guess a similar report can be implemented in the PHP code that split and run the tests in parallel.  Bonus if that outputs some kind of " [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar)
[11:53:36] <wikibugs>	 (03CR) 10Arthur taylor: [C:03+1] "Was thinking the same. I can just create another artifact in PHP with the timing data for the splits." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar)
[12:09:46] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "`" [integration/config] - 10https://gerrit.wikimedia.org/r/1248409 (https://phabricator.wikimedia.org/T418797) (owner: 10Hashar)
[12:11:24] <wikibugs>	 (03Merged) 10jenkins-bot: jjb, Zuul: rename wikibase-selenium job for clarity [integration/config] - 10https://gerrit.wikimedia.org/r/1248409 (https://phabricator.wikimedia.org/T418797) (owner: 10Hashar)
[12:12:20] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248409/ "jjb, Zuul: rename wikibase-selenium job for clarity" | T418797
[12:12:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:12:23] <stashbot>	 T418797: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797
[12:15:45] <wikibugs>	 (03CR) 10Phedenskog: Wrap user commands with a Chronometer (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1235103 (owner: 10Hashar)
[12:15:48] <hashar>	 codders: for the quibble durations report, I think ultimately we want to push them to Prometheus and have them available via Grafana
[12:15:53] <hashar>	 instead of coding a custom tool for that
[12:16:28] <hashar>	 we have a task floating around to collect browser tests timing using Prometheus, though that one hasn't been pushed forward, that seems slithly related
[12:17:00] <hashar>	 the devil is defining the various labels we want to push for each metric, and I think the command name would need a prometheus friendly name :-]
[12:17:33] <hashar>	 I'll check with Peter Hedenskog from Test Platform team
[12:27:49] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "`" [integration/config] - 10https://gerrit.wikimedia.org/r/1248411 (https://phabricator.wikimedia.org/T418797) (owner: 10Hashar)
[12:27:59] <wikibugs>	 (03CR) 10CI reject: [V:04-1] jjb, Zuul: vary Wikibase Selenium for release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248411 (https://phabricator.wikimedia.org/T418797) (owner: 10Hashar)
[12:28:16] <wikibugs>	 (03PS3) 10Hashar: jjb, Zuul: vary Wikibase Selenium for release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248411 (https://phabricator.wikimedia.org/T418797)
[12:28:33] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb, Zuul: vary Wikibase Selenium for release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248411 (https://phabricator.wikimedia.org/T418797) (owner: 10Hashar)
[12:29:14] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Wrap user commands with a Chronometer [integration/quibble] - 10https://gerrit.wikimedia.org/r/1235103 (owner: 10Hashar)
[12:30:09] <wikibugs>	 (03Merged) 10jenkins-bot: jjb, Zuul: vary Wikibase Selenium for release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248411 (https://phabricator.wikimedia.org/T418797) (owner: 10Hashar)
[12:30:25] <hashar>	 Daimona: kostajh: hi, do you need Quibble to run `phpunit:config` or was that moved to mediawiki/core?   the later sounds better imho
[12:31:01] <hashar>	 that was proposed via https://gerrit.wikimedia.org/r/c/integration/quibble/+/1248400 and if the logic is now in MediaWiki, I guess that patch can be abandoned 
[12:32:02] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248411 "jjb, Zuul: vary Wikibase Selenium for release branches" | T418797
[12:32:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:32:05] <stashbot>	 T418797: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797
[12:37:21] <Daimona>	 Yeah, we should be good on that front now
[12:38:41] <hashar>	 Daimona: may you abandon https://gerrit.wikimedia.org/r/c/integration/quibble/+/1248400 so ? :] thx!
[12:43:30] <wikibugs>	 (03Merged) 10jenkins-bot: Wrap user commands with a Chronometer [integration/quibble] - 10https://gerrit.wikimedia.org/r/1235103 (owner: 10Hashar)
[12:47:18] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11681512 (10Daimona) >>! In T419107#11680545,...
[12:47:48] <wikibugs>	 (03CR) 10Daimona Eaytoy: "Yeah this is no longer needed" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248400 (https://phabricator.wikimedia.org/T419107) (owner: 10Kosta Harlan)
[12:47:54] <hashar>	 Daimona: merci!
[12:48:06] <Daimona>	 hashar: I can't, probably because I don't have +2 on integration/quibble
[12:48:11] <hashar>	 OH
[12:48:12] <hashar>	 bummer
[12:48:13] <hashar>	 :)
[12:48:15] <Daimona>	 (and it's not me who made the change)
[12:48:22] <hashar>	 yeah my bad sorry
[12:48:32] <hashar>	 my account is blessed with Administrator rights and thus I can abandon any change
[12:48:37] <Daimona>	 Np. I did confirm on the patch that is no longer needed if that helps :D
[12:48:41] <hashar>	 (and surely I should use a different account but that is a different story)
[12:48:51] <wikibugs>	 (03Abandoned) 10Hashar: commands: Run composer phpunit:config for parallel tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248400 (https://phabricator.wikimedia.org/T419107) (owner: 10Kosta Harlan)
[12:49:13] <hashar>	 I guess if we can get as much of the logic inside mediawiki/core, that makes it easier to amend/adjust
[12:57:10] <wikibugs>	 10Continuous-Integration-Config, 07ci-test-error: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797#11681541 (10hashar)
[12:59:27] <wikibugs>	 10Continuous-Integration-Config, 07ci-test-error: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797#11681542 (10hashar)
[13:19:22] <wikibugs>	 10Gerrit, 10Quibble: Gate pipeline tasks build fail ramdomly - https://phabricator.wikimedia.org/T419233 (10alex-mashin) 03NEW
[13:26:52] <wikibugs>	 10Gerrit, 10Quibble: Gate pipeline tasks build fail ramdomly - https://phabricator.wikimedia.org/T419233#11681614 (10taavi) →14Duplicate dup:03T419107
[13:26:53] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11681616 (10taavi)
[13:33:07] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11681643 (10Daimona) p:05Unbreak!→03High T...
[13:40:05] <wikibugs>	 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), and 2 others: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11681682 (10A_smart_kitten)
[13:51:24] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11681733 (10Jdforrester-WMF) (This is not abou...
[13:54:50] <A_smart_kitten>	 ah okay James_F :)
[13:55:44] <A_smart_kitten>	 should that be tagged with a different MW project though? i looked at it and saw some patches, the code for which seemed to imply issues with the code in MW itself (rather than e.g. necessarily Quibble), so I added the tag I thought best fit
[14:09:48] <wikibugs>	 (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1248389 (https://phabricator.wikimedia.org/T396709) (owner: 10Kosta Harlan)
[14:09:56] <wikibugs>	 (03CR) 10Jforrester: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1248389 (https://phabricator.wikimedia.org/T396709) (owner: 10Kosta Harlan)
[14:11:15] <wikibugs>	 10Continuous-Integration-Config, 07ci-test-error: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797#11682126 (10hashar) 05Open→03Resolved Dropping the gate-and-submit-pipeline and stripping the common job prefix gives: ` lang=yaml test:   - vendor-php83 test-1_43:...
[14:11:20] <James_F>	 A_smart_kitten: No. There's not really a project for "Everything about how MW uses PHPUnit is a disaster", sadly.
[14:11:50] <hashar>	 zabe: I have fixed the Wikibase job for REL branches and did a `recheck` of each of the changes. They seem to have passed \o/
[14:15:39] <zabe>	 Thanks! :)
[14:26:25] <A_smart_kitten>	 James_F: fair enough (so I suppose my second guess of `MediaWiki-General` would also not be fitting here)
[14:26:38] <wikibugs>	 (03PS1) 10Hashar: wm-checks-api: add tooltip to the CheckRun Run action [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1248835
[14:29:18] <James_F>	 A_smart_kitten: Yeah, agree that doesn't really fit either. :-(
[14:53:34] <wikibugs>	 (03PS1) 10Hashar: releasing: fix git log command [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248850
[14:57:33] <wikibugs>	 (03CR) 10Jforrester: [C:04-1] "This is a duplicate of your colleague's patch in I87e2f2f702add3671fa59d1fbf4e7d3a64466705. Let's not have multiple versions in parallel." [integration/config] - 10https://gerrit.wikimedia.org/r/1248693 (https://phabricator.wikimedia.org/T417568) (owner: 10Clare Ming)
[15:00:53] <wikibugs>	 (03PS9) 10Jforrester: Zuul: [mediawiki/extensions/MetricsPlatform] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci)
[15:00:53] <wikibugs>	 (03CR) 10Jforrester: Zuul: [mediawiki/extensions/MetricsPlatform] Archive (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci)
[15:00:53] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248853
[15:02:17] <wikibugs>	 (03Abandoned) 10Clare Ming: Remove Metrics Platform extension [integration/config] - 10https://gerrit.wikimedia.org/r/1248693 (https://phabricator.wikimedia.org/T417568) (owner: 10Clare Ming)
[15:02:37] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/MetricsPlatform] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci)
[15:03:14] <wikibugs>	 (03PS10) 10Jforrester: Zuul: [mediawiki/extensions/MetricsPlatform] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci)
[15:03:37] <wikibugs>	 (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci)
[15:05:13] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11682485 (10Andrew) @elukey I doubt that I'll be of much help here. If the key isn't present on the puppetserver itself then it is...
[15:08:42] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Dear Zuul, consider this a trigger message." [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci)
[15:10:02] <wikibugs>	 (03PS1) 10Hashar: release: Quibble 1.16.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248856 (https://phabricator.wikimedia.org/T417399)
[15:10:09] <wikibugs>	 (03PS1) 10Hashar: release: Start 1.16.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248857
[15:11:42] <James_F>	 Oh! Yeah, depends on the prod config things, meh.
[15:12:10] <wikibugs>	 (03CR) 10Jforrester: [C:03+1] "Oops, yes, this is waiting on the production config and puppet patches being deployed; leaving that to your team to arrange." [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci)
[15:30:17] <ebernhardson>	 I realize it's friday, but I'd like to deploy a mw-config patch switching a test cluster from http to https.  This is a misconfiguration, the normal clusters use envoy on localhost w/ http, but this one talks to the cluster directly and needs https. This cluster is only invoked via debug options, it's not part of any normal request flow. https://gerrit.wikimedia.org/r/1248858
[15:51:26] <hashar>	 ebernhardson: +1 I am happy to help.  But let SRE knows about it! :)
[15:53:03] <hashar>	 or maybe it is fine because that is just a test platform
[15:53:09] <hashar>	 anyway I +1ed
[15:53:57] <hashar>	 James_F: I have completed the wikibase-selenium job rename this morning :b
[15:54:05] <James_F>	 I saw. Thanks!
[15:54:16] <hashar>	 James_F: and I have prepared the changelog for a new quibble version, if you don't mind reviewing my broken english, I'll appreciate it https://gerrit.wikimedia.org/r/c/integration/quibble/+/1248856 :]
[15:54:36] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] releasing: fix git log command [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248850 (owner: 10Hashar)
[15:54:41] <ebernhardson>	 hashar: thanks!
[15:55:08] <wikibugs>	 10Gerrit: Gerrit not accesible for this user, - https://phabricator.wikimedia.org/T419261 (10ShakespeareFan00) 03NEW
[15:56:04] <James_F>	 hashar: Not 1.15.1? Eh. Sure, it looks fine.
[15:56:09] <wikibugs>	 10Gerrit: Gerrit not accesible for this user, - https://phabricator.wikimedia.org/T419261#11682674 (10ShakespeareFan00)
[15:56:20] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11682676 (10aude) We also have this problem wi...
[15:56:42] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11682679 (10aude)
[15:58:59] <hashar>	 James_F: this version has a new feature, reporting duration at end of build. So I felt it was worth a minor version bump
[15:59:16] <James_F>	 Yeah, I don't know if that's big enough, but sure, go for it.
[15:59:36] <hashar>	 <3
[15:59:56] <hashar>	 James_F: may you +1/+2 https://gerrit.wikimedia.org/r/c/integration/quibble/+/1248856/ ? :)
[16:00:04] <wikibugs>	 (03CR) 10Jforrester: [C:03+1] release: Quibble 1.16.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248856 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar)
[16:00:11] <wikibugs>	 (03CR) 10Hashar: [C:03+2] release: Quibble 1.16.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248856 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar)
[16:00:16] <wikibugs>	 (03CR) 10Hashar: [C:03+2] release: Start 1.16.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248857 (owner: 10Hashar)
[16:00:28] <hashar>	 thanks, I will do the tag  image update
[16:00:35] <hashar>	 and I guess switch the jobs next week
[16:01:10] <James_F>	 Not today?
[16:13:56] <wikibugs>	 (03Merged) 10jenkins-bot: releasing: fix git log command [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248850 (owner: 10Hashar)
[16:19:29] <wikibugs>	 (03Merged) 10jenkins-bot: release: Quibble 1.16.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248856 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar)
[16:19:33] <wikibugs>	 (03Merged) 10jenkins-bot: release: Start 1.16.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248857 (owner: 10Hashar)
[16:31:02] <hashar>	 !log Tag Quibble 1.16.0 @ 0b9db5fe3cabb2cec0b5d44e128bafa917b3b895 # T417399 T417409 T418461
[16:31:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:31:12] <stashbot>	 T417399: Quibble should emit a report of each time it took to run the steps/stages - https://phabricator.wikimedia.org/T417399
[16:31:14] <stashbot>	 T417409: Quibble ReportVersions takes 4 seconds! - https://phabricator.wikimedia.org/T417409
[16:31:15] <stashbot>	 T418461: --shell causes TypeError - https://phabricator.wikimedia.org/T418461
[16:31:43] <wikibugs>	 (03PS1) 10Hashar: dockerfiles: update Quibble to 1.16.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1248875
[16:31:59] <hashar>	 James_F: that `./utils/update-quibble` script looks so nice. Thx!
[16:32:04] <James_F>	 <3
[16:32:13] <James_F>	 hashar: Also see my WIP patch to make a bookworm quibble.
[16:32:28] <hashar>	 oh
[16:32:36] <hashar>	 I have completely lost track of how we align
[16:32:44] <James_F>	 It's stalled on two things  — one, dropping python2 (I'm pretty sure we don't need it any more), and two, a little bit of packaging from SRE.
[16:32:48] <hashar>	 I think the idea was to align with SRE underlying OS version and they had Bullseye for quite a while
[16:32:57] <James_F>	 Prod is on bookworm now, AIUI.
[16:33:13] <hashar>	 so we are lagging behind 
[16:33:22] <James_F>	 I shifted our php81 images off Wikimedia-PHP and to Sury-PHP so that we can switch
[16:33:24] <hashar>	 which packages would we miss?
[16:33:41] <James_F>	 (As SRE never bothered to package 81 for bookworm, rightly.)
[16:33:52] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Done by `./utils/update-quibble`" [integration/config] - 10https://gerrit.wikimedia.org/r/1248875 (owner: 10Hashar)
[16:34:08] <hashar>	 hmm
[16:34:12] <James_F>	 thirdparty/opensearch1
[16:34:17] <James_F>	 T418809
[16:34:17] <stashbot>	 T418809: Package thirdparty/opensearch1 for bookworm - https://phabricator.wikimedia.org/T418809
[16:34:20] <hashar>	 I am pretty sure I caught a few issues with the php8.1 (and again with php8.3) packaging
[16:34:35] <hashar>	 and I can't imagine us having run 8.1 from the stock Deb 
[16:35:06] <hashar>	 ah yeah for opensearch, I think the use case was to run some integration tests
[16:35:16] <hashar>	 in prod that is using a different set of hosts/debian version
[16:35:23] <hashar>	 but our CI image tries to match both environments
[16:35:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11682781 (10bd808) >>! In T419099#11682485, @Andrew wrote: > I also don't really understand what's going on with the cross-project...
[16:35:30] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.16.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1248875 (owner: 10Hashar)
[16:35:49] <James_F>	 I'm almost certain that the python2 use case (node-gyp) isn't live anywhere any more.
[16:36:06] <hashar>	 hopefully :\
[16:36:06] <James_F>	 And if it is, well, tough, those repos have to act like is 2010 and stop using python2.
[16:36:15] <hashar>	 2020!
[16:36:23] <hashar>	 but yeah I think it is fine
[16:36:35] <hashar>	 certainly after we dropped REL1_39 (which might still had some old/obsolete stuff)
[16:36:45] <James_F>	 It finally went EOL in 2019. But the switch-off-it was a decade before.
[16:36:47] <hashar>	 and we get node20+ so I assume everything got moved out already
[16:37:00] <James_F>	 Yeah, also I need to bump us to node22 or maybe 24.
[16:37:08] <James_F>	 20 goes EOL next month, I think.
[16:37:10] <hashar>	 !log Building Docker images for Quibble 1.16.0
[16:37:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:37:19] <hashar>	 joy
[16:37:25] <hashar>	 for opensearch, I can't remember which job uses it
[16:37:35] <hashar>	 but the git log probably has the details
[16:37:40] <James_F>	 Presumably something CirrusSearch related?
[16:37:43] <hashar>	 yeah
[16:37:50] <James_F>	 Anyway, time to move.
[16:37:54] <hashar>	 maybe to get rid of one of the browser bot they had
[16:39:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11682806 (10elukey) @Andrew I asked since from the logs you created the puppetserver vm keeping the old certs in place, so I was w...
[16:39:51] <hashar>	 ah https://phabricator.wikimedia.org/T386691
[16:40:04] <hashar>	 so Wikibase has some integration tests
[16:40:10] <hashar>	 and for CirrusSearch I don't know
[16:40:38] <hashar>	 https://codesearch.wmcloud.org/search/?q=QUIBBLE_OPENSEARCH&files=&excludeFiles=&repos=#Extension:CirrusSearch
[16:40:41] <hashar>	 they do as well (good!)
[16:48:15] <hashar>	 James_F: looks like logging servers are using Debian Bookworm so I guess we have built the packages already
[16:48:21] <hashar>	 but they are in a different component
[16:48:41] <hashar>	 so it might not be too complicated to have them published under bookworm thirdparty/opensearch81 
[16:50:29] <IKhitron>	 I suggest to delete and maybe even oversight https://commons.wikimedia.org/wiki/File:Woodpecker10.jpg.
[16:55:26] <wikibugs>	 (03PS1) 10Hashar: jjb: update jobs to Quibble 1.16.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1248880 (https://phabricator.wikimedia.org/T417399)
[16:56:25] <hashar>	 ebernhardson: do you need assistance for that operations/mediawiki-config patch? (http > https for the semanticsearch-test cluster)
[16:56:56] <hashar>	 I ponder whether I should update the jobs to Quibble 1.16.0 and then leave for the week-end, but that sounds bad practice :b
[16:58:07] <ebernhardson>	 hashar: no i can ship it, just in a meeting right now. Will ship after
[16:58:27] <hashar>	 :)
[16:58:34] <hashar>	 ebernhardson: ok :-]
[17:04:35] <wikibugs>	 (03CR) 10Hashar: [C:03+2] wm-checks-api: add tooltip to the CheckRun Run action [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1248835 (owner: 10Hashar)
[17:05:24] <wikibugs>	 (03Merged) 10jenkins-bot: wm-checks-api: add tooltip to the CheckRun Run action [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1248835 (owner: 10Hashar)
[17:07:27] <hashar>	 that one was easy
[17:14:57] <James_F>	 hashar: Yes, fingers crossed.
[17:15:26] <hashar>	 it worked! :b
[17:15:46] <hashar>	 well I am off
[17:15:52] <hashar>	 see you next week
[17:18:27] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11682965 (10DLynch) Conveniently this is only...
[17:22:36] <wikibugs>	 (03PS1) 10C. Scott Ananian: Fix visualdiff item URL links to use mobile/variant logic [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248890
[17:23:24] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Fix visualdiff item URL links to use mobile/variant logic [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248890 (owner: 10C. Scott Ananian)
[17:31:26] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb: Drop phpunit-coverage-…-publish for PHP 8.1/8.2, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1248502 (owner: 10Jforrester)
[17:33:00] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Drop phpunit-coverage-…-publish for PHP 8.1/8.2, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1248502 (owner: 10Jforrester)
[18:05:02] <wikibugs>	 (03PS1) 10C. Scott Ananian: Fix confidence report when some titles have not yet been processed [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248900
[18:05:33] <wikibugs>	 (03PS1) 10Jforrester: utils: Provide a script to find unused Jenkins Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1248901
[18:05:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Fix confidence report when some titles have not yet been processed [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248900 (owner: 10C. Scott Ananian)
[18:06:59] <wikibugs>	 (03CR) 10CI reject: [V:04-1] utils: Provide a script to find unused Jenkins Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1248901 (owner: 10Jforrester)
[18:15:03] <wikibugs>	 (03PS1) 10Jforrester: utils: Provide a script to find unused Docker images [integration/config] - 10https://gerrit.wikimedia.org/r/1248903
[18:16:28] <wikibugs>	 (03CR) 10CI reject: [V:04-1] utils: Provide a script to find unused Docker images [integration/config] - 10https://gerrit.wikimedia.org/r/1248903 (owner: 10Jforrester)
[18:29:50] <wikibugs>	 (03PS2) 10Jforrester: utils: Provide a script to find unused Jenkins Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1248901
[18:29:50] <wikibugs>	 (03PS2) 10Jforrester: utils: Provide a script to find unused Docker images [integration/config] - 10https://gerrit.wikimedia.org/r/1248903
[19:07:57] <wikibugs>	 (03open) 10zabe: repositories: Remove IPSet [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/124
[19:08:20] <wikibugs>	 (03merge) 10zabe: repositories: Remove IPSet [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/124
[19:13:08] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11683268 (10Dzahn) Imported jenkins into trixie-wikimedia.   ` [apt1002:/srv/wikimedia] $ sudo -i reprepro -C t...
[19:16:04] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11683281 (10Dillon) I can confirm this is also...
[19:16:22] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11683283 (10Dzahn) contint2003 now the basic jenkins installed via the new stub role "jenkins".    ` Notice: /S...
[19:18:28] <wikibugs>	 06Release-Engineering-Team, 10Citoid, 07Essential-Work, 07Technical-Debt: update nodejs22-slim image to 22.21.0 to support proxy env variables for outbound requests - https://phabricator.wikimedia.org/T416471#11683293 (10Jdforrester-WMF) Note that the Node 24 image will now be available from Monday; maybe...
[19:19:29] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11683297 (10Dillon)
[19:21:39] <wikibugs>	 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11683306 (10Jdforrester-WMF)
[19:21:48] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11683307 (10Daimona) I'm assuming the post-mer...
[19:36:18] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/services/geoshapes] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248916 (https://phabricator.wikimedia.org/T418372)
[19:38:58] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/geoshapes] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248916 (https://phabricator.wikimedia.org/T418372) (owner: 10Jforrester)
[19:40:31] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/geoshapes] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248916 (https://phabricator.wikimedia.org/T418372) (owner: 10Jforrester)
[19:46:47] <James_F>	 !log Zuul: [mediawiki/services/geoshapes] Mark as archived, for T418372
[19:46:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:46:50] <stashbot>	 T418372: Archive the Geoshapes service - https://phabricator.wikimedia.org/T418372
[20:43:39] <wikibugs>	 10Gerrit: Gerrit not accesible for this user, - https://phabricator.wikimedia.org/T419261#11683644 (10Aklapper) 05Open→03Declined > "115.33.0esr (32-bit"  Firefox ESR 115 support by Mozilla ends later this month (March 2026). Please upgrade to a more recent browser.
[21:09:38] <wikibugs>	 10Gerrit: Gerrit not accesible for this user, - https://phabricator.wikimedia.org/T419261#11683745 (10ShakespeareFan00) I'm sorry but that doesn't resolve the issue.  You are effectively telling me don't participate because I can't upgrade.  That is unacceptable,.
[21:15:39] <wikibugs>	 (03CR) 10Michael Große: "It seems this needs a bit more work, it caused this error on MinervaNeue:" [integration/config] - 10https://gerrit.wikimedia.org/r/1248412 (https://phabricator.wikimedia.org/T418053) (owner: 10EMcFarland)
[21:20:24] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11683758 (10Dzahn)
[21:39:17] <wikibugs>	 10Phabricator: Phabricator: Unable to establish a connection to any database host - https://phabricator.wikimedia.org/T419297 (10AlexisJazz) 03NEW
[21:46:30] <wikibugs>	 10Phabricator, 06collaboration-services: Phabricator: Unable to establish a connection to any database host - https://phabricator.wikimedia.org/T419297#11683839 (10Aklapper) I see that in the logs at `Mar 6, 2026 @ 21:35:22`, happens from time to time.
[22:02:29] <wmcs-alerts>	 FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[22:02:29] <wmcs-alerts>	 FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[22:04:21] <wikibugs>	 10Phabricator, 06collaboration-services: Phabricator: Unable to establish a connection to any database host - https://phabricator.wikimedia.org/T419297#11683870 (10A_smart_kitten) I got a similar error earlier, seems potentially similar to (non-public) {T408956} from late last year.
[22:37:35] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11683939 (10aude) I think this https://gerrit....
[23:54:53] <wikibugs>	 10Continuous-Integration-Infrastructure, 06collaboration-services: reserve 2 new public IPs for contint machines - https://phabricator.wikimedia.org/T418520#11684070 (10Dzahn) 05Open→03Resolved Hi,  we are not introducing a new service or use case for a public IP.  This is just a work around to solve a...