[02:00:57] (03CR) 10Subramanya Sastry: [C:03+2] Fix visualdiff item URL links to use mobile/variant logic [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248890 (owner: 10C. Scott Ananian) [02:01:25] (03CR) 10Subramanya Sastry: [C:03+2] Fix confidence report when some titles have not yet been processed [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248900 (owner: 10C. Scott Ananian) [02:01:35] (03Merged) 10jenkins-bot: Fix visualdiff item URL links to use mobile/variant logic [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248890 (owner: 10C. Scott Ananian) [02:02:05] (03Merged) 10jenkins-bot: Fix confidence report when some titles have not yet been processed [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1248900 (owner: 10C. Scott Ananian) [03:01:31] (03PS1) 10Subramanya Sastry: Non-MFE: Render images as transparent PNG to support better diffing [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1249079 [07:50:39] 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11686219 (10ArthurTaylor) > This still happens on CI: > htt... [07:53:23] (03PS1) 10Phedenskog: selenium: Add run selenium test plan [integration/quibble] - 10https://gerrit.wikimedia.org/r/1249093 (https://phabricator.wikimedia.org/T418743) [08:07:31] FIRING: [7x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:08:15] 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11686250 (10Viktoria_Hillerud_WMSE) We are having the same... [08:13:45] 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11686253 (10ArthurTaylor) Hi @Viktoria_Hillerud_WMSE, For... [08:14:54] (03CR) 10Kosta Harlan: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1248389 (https://phabricator.wikimedia.org/T396709) (owner: 10Kosta Harlan) [08:15:37] (03CR) 10Kosta Harlan: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1248389 (https://phabricator.wikimedia.org/T396709) (owner: 10Kosta Harlan) [08:15:43] 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11686255 (10Viktoria_Hillerud_WMSE) >>! In T419107#11686253... [08:52:52] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Create alerts using the new replication metrics exposed in Gerrit - https://phabricator.wikimedia.org/T418216#11686341 (10ABran-WMF) a:05ABran-WMF→03None [09:15:29] !log updating all CI Jenkins jobs using `./jjb-update` [09:15:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:24:53] !log Updating Quibble jobs to 1.16.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/1248880 | T417399 T417409 T418461 [09:24:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:24:59] T417399: Quibble should emit a report of each time it took to run the steps/stages - https://phabricator.wikimedia.org/T417399 [09:24:59] T417409: Quibble ReportVersions takes 4 seconds! - https://phabricator.wikimedia.org/T417409 [09:25:00] T418461: --shell causes TypeError - https://phabricator.wikimedia.org/T418461 [09:27:16] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11686396 (10WMDE-leszek) Hello, kindly asking to get @Virginie.caplet added as a project adming. She's leading UX and discovery initiatives at WMDE's Wikidata... [09:31:39] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11686400 (10Aklapper) @WMDE-leszek Added. In case @Virginie.caplet is staff it would be great to edit the [user profile](https://phabricator.wikimedia.org/p/Vi... [09:33:48] 10Gerrit, 06collaboration-services: Move Gerrit data to object storage - https://phabricator.wikimedia.org/T416972#11686430 (10ABran-WMF) Gerrit does not support object storage: https://gerrit.googlesource.com/plugins/lfs-storage-s3 the plugin has been deprecated for over 10 years. The documentation only ment... [09:34:51] 10Gerrit, 06collaboration-services: Gerrit topology evolution branch pick-up - https://phabricator.wikimedia.org/T414407#11686446 (10ABran-WMF) [09:34:55] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Explore solutions for Gerrit on Kubernetes - https://phabricator.wikimedia.org/T418364#11686448 (10ABran-WMF) →14Duplicate dup:03T414407 [09:35:54] 10Gerrit, 06collaboration-services: Move Gerrit data to CephFS - https://phabricator.wikimedia.org/T416972#11686456 (10ABran-WMF) [09:38:52] 10Gerrit, 06collaboration-services: Move Gerrit data to CephFS - https://phabricator.wikimedia.org/T416972#11686495 (10ABran-WMF) a:03ABran-WMF [09:40:01] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: remove mod_qos on Gerrit - https://phabricator.wikimedia.org/T417615#11686516 (10ABran-WMF) I'll merge the change and follow the [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1248843/comments/678156f2_a9f73fec | r... [09:45:27] 06Release-Engineering-Team, 10Quibble, 13Patch-For-Review: Quibble should emit a report of each time it took to run the steps/stages - https://phabricator.wikimedia.org/T417399#11686620 (10hashar) Released with Quibble 1.16.0. I have updated the CI jobs, they now output the ASCII table and have `quibble... [09:45:33] 06Release-Engineering-Team, 10Quibble, 13Patch-For-Review: Quibble should emit a report of each time it took to run the steps/stages - https://phabricator.wikimedia.org/T417399#11686622 (10hashar) 05Open→03Resolved [09:45:47] (03CR) 10Hashar: "I have deployed the jobs update." [integration/config] - 10https://gerrit.wikimedia.org/r/1248880 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [09:47:26] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: remove mod_qos on Gerrit - https://phabricator.wikimedia.org/T417615#11686676 (10ABran-WMF) [09:47:41] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11686688 (10hashar) [09:48:25] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: remove mod_qos on Gerrit - https://phabricator.wikimedia.org/T417615#11686695 (10ABran-WMF) After applying the change on `gerrit-spare`: ` arnaudb@gerrit2002:~ $ curl -sSL localhost:3903/metrics|rg -i http_status # HELP apa... [09:51:44] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: remove mod_qos on Gerrit - https://phabricator.wikimedia.org/T417615#11686719 (10ABran-WMF) `gerrit-replica` updated. I'll let the change sit for a while and move on to the primary instance. ` arnaudb@gerrit1003:~ $ curl -... [09:58:49] (03PS1) 10EMcFarland: Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen [integration/config] - 10https://gerrit.wikimedia.org/r/1249216 [10:08:56] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: remove mod_qos on Gerrit - https://phabricator.wikimedia.org/T417615#11686786 (10ABran-WMF) 05In progress→03Resolved the change has also been applied to the primary instance: ` arnaudb@gerrit2003:~ $ curl -sSL loca... [10:25:56] (03PS1) 10Phedenskog: Skip NpmInstall for core when not needed [integration/quibble] - 10https://gerrit.wikimedia.org/r/1249226 (https://phabricator.wikimedia.org/T418743) [10:40:43] !log gerrit: mediawiki/vendor: converted `es6` and `es710` branches to tags # T417804 [10:40:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:40:45] T417804: Delete some branches from mediawiki/vendor - https://phabricator.wikimedia.org/T417804 [10:41:27] 06Release-Engineering-Team, 10MediaWiki-Vendor: Delete some branches from mediawiki/vendor - https://phabricator.wikimedia.org/T417804#11686915 (10hashar) 05Open→03Resolved a:03hashar Via https://gerrit.wikimedia.org/r/admin/repos/mediawiki/vendor,tags , I have created the following tags: | Tag name... [10:50:49] 10Gerrit, 06collaboration-services: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11686941 (10ABran-WMF) a:05ABran-WMF→03hashar I'm handing you the task @hashar as it looks like all the remaining steps are yours! happy to help if needed, otherwise I'll follow up on... [11:01:07] 10Gerrit, 06collaboration-services, 07Sustainability (Incident Followup): Update user facing communication to reflect availability of Gerrit during maintenance - https://phabricator.wikimedia.org/T417135#11687015 (10ABran-WMF) a:05ABran-WMF→03None [11:04:00] (03CR) 10Hashar: [C:03+2] build: Updating mediawiki/mediawiki-codesniffer to 48.0.0 (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1185446 (owner: 10Libraryupgrader) [11:45:19] 10Gerrit, 06collaboration-services: Move Gerrit data to CephFS - https://phabricator.wikimedia.org/T416972#11687372 (10MatthewVernon) The Apus cluster does not currently support CephFS, I'm afraid. It wouldn't be straightforward to add support either - Apus does multi-site replication at the RGW/S3 level, the... [11:58:40] 10Gerrit, 06collaboration-services: Move Gerrit data to CephFS - https://phabricator.wikimedia.org/T416972#11687426 (10MatthewVernon) I see from T414407 you're thinking about moving gerrit to k8s. I do wonder if this is the sort of thing that k8s persistent volume claims are intended for? [12:47:25] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen [integration/config] - 10https://gerrit.wikimedia.org/r/1249216 (owner: 10EMcFarland) [12:47:53] (03CR) 10Jforrester: [C:03+2] "Per Hashar." [integration/config] - 10https://gerrit.wikimedia.org/r/1248880 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [12:48:29] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1248389 (https://phabricator.wikimedia.org/T396709) (owner: 10Kosta Harlan) [12:49:09] (03Merged) 10jenkins-bot: Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen [integration/config] - 10https://gerrit.wikimedia.org/r/1249216 (owner: 10EMcFarland) [12:49:45] (03Merged) 10jenkins-bot: jjb: update jobs to Quibble 1.16.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1248880 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [12:50:16] !log Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen [12:50:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:50:30] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1248389 (https://phabricator.wikimedia.org/T396709) (owner: 10Kosta Harlan) [12:50:40] !log Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency [12:50:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:51:19] hashar: Your input on https://gerrit.wikimedia.org/r/c/integration/config/+/1248901 and https://gerrit.wikimedia.org/r/c/integration/config/+/1248903 would be welcome! [12:56:18] (03CR) 10Jforrester: "FYI: If you use the utils/archive-repo.py script it'll add the trailing comment for traceability and not add errant whitespace. (I'll fix " [integration/config] - 10https://gerrit.wikimedia.org/r/1248968 (https://phabricator.wikimedia.org/T418008) (owner: 10Pppery) [12:59:57] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248968 (https://phabricator.wikimedia.org/T418008) (owner: 10Pppery) [12:59:57] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248969 (https://phabricator.wikimedia.org/T413585) (owner: 10Pppery) [12:59:57] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248970 (https://phabricator.wikimedia.org/T413586) (owner: 10Pppery) [12:59:57] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248971 (https://phabricator.wikimedia.org/T413587) (owner: 10Pppery) [12:59:58] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248972 (https://phabricator.wikimedia.org/T413588) (owner: 10Pppery) [13:00:01] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/QuickGV] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248974 (https://phabricator.wikimedia.org/T413348) (owner: 10Pppery) [13:00:05] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/Memento] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248975 (https://phabricator.wikimedia.org/T369991) (owner: 10Pppery) [13:00:09] (03PS2) 10Jforrester: Zuul; [mediawiki/extensions/WikiShare] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248983 (https://phabricator.wikimedia.org/T415389) (owner: 10Pppery) [13:00:34] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248968 (https://phabricator.wikimedia.org/T418008) (owner: 10Pppery) [13:02:37] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248968 (https://phabricator.wikimedia.org/T418008) (owner: 10Pppery) [13:04:37] !log Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived, for T418008 [13:04:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:04:39] T418008: Archive the SecurityApi extension - https://phabricator.wikimedia.org/T418008 [13:04:45] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248969 (https://phabricator.wikimedia.org/T413585) (owner: 10Pppery) [13:05:34] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248970 (https://phabricator.wikimedia.org/T413586) (owner: 10Pppery) [13:06:28] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248969 (https://phabricator.wikimedia.org/T413585) (owner: 10Pppery) [13:06:46] !log Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived, for T413585 [13:06:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:06:49] T413585: Archive the GoogleAdSense extension - https://phabricator.wikimedia.org/T413585 [13:07:02] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248970 (https://phabricator.wikimedia.org/T413586) (owner: 10Pppery) [13:07:12] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248971 (https://phabricator.wikimedia.org/T413587) (owner: 10Pppery) [13:07:24] !log Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived, for T413586 [13:07:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:07:26] T413586: Archive the Semantic Sifter extension - https://phabricator.wikimedia.org/T413586 [13:07:45] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1248971 (https://phabricator.wikimedia.org/T413587) (owner: 10Pppery) [13:07:55] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248972 (https://phabricator.wikimedia.org/T413588) (owner: 10Pppery) [13:08:11] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/QuickGV] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248974 (https://phabricator.wikimedia.org/T413348) (owner: 10Pppery) [13:08:41] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Memento] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248975 (https://phabricator.wikimedia.org/T369991) (owner: 10Pppery) [13:08:48] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248971 (https://phabricator.wikimedia.org/T413587) (owner: 10Pppery) [13:09:23] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248972 (https://phabricator.wikimedia.org/T413588) (owner: 10Pppery) [13:09:50] !log Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived, for T413587 [13:09:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:09:52] T413587: Archive the SidebarDonateBox extension - https://phabricator.wikimedia.org/T413587 [13:10:05] !log Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived, for T413588 [13:10:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:10:08] T413588: Archive the Semantic Image Input extension - https://phabricator.wikimedia.org/T413588 [13:10:16] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/QuickGV] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248974 (https://phabricator.wikimedia.org/T413348) (owner: 10Pppery) [13:10:46] !log Zuul: [mediawiki/extensions/QuickGV] Mark as archived, for T413348 [13:10:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:10:49] T413348: Archive the QuickGV extension from Gerrit - https://phabricator.wikimedia.org/T413348 [13:11:00] (03PS3) 10Jforrester: Zuul; [mediawiki/extensions/WikiShare] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248983 (https://phabricator.wikimedia.org/T413589) (owner: 10Pppery) [13:11:03] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Memento] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248975 (https://phabricator.wikimedia.org/T369991) (owner: 10Pppery) [13:11:06] (03CR) 10Jforrester: [C:03+2] Zuul; [mediawiki/extensions/WikiShare] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248983 (https://phabricator.wikimedia.org/T413589) (owner: 10Pppery) [13:11:58] !log Zuul: [mediawiki/extensions/Memento] Mark as archived, for T369991 [13:12:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:12:01] T369991: Archive the Memento extension's Gerrit repository - https://phabricator.wikimedia.org/T369991 [13:13:09] (03Merged) 10jenkins-bot: Zuul; [mediawiki/extensions/WikiShare] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1248983 (https://phabricator.wikimedia.org/T413589) (owner: 10Pppery) [13:13:56] !log Zuul: [mediawiki/extensions/WikiShare] Mark as archived, for T413589 [13:13:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:13:59] T413589: Archive the WikiShare extension - https://phabricator.wikimedia.org/T413589 [13:41:23] James_F: I did comment on one of the pathches about some logic being in docker-updates, but your script also have to extract the list of jobs from zuul so .. that is a slightly different thing :) [13:41:41] hashar: Oh, right. Yeah, it's a bit different. [13:41:53] Mostly I wanted to hunt down and kill the jobs we don't use. :-) [13:42:00] James_F: I am working on a migration this week, so feel free to self merge if the scripts work for you. We can always refactor them later on if they prove to be a problem or we are not at ease with the copypasta :b [13:42:05] Ack. [13:43:20] there is a `./jjb-list` helper at the root of the repo that lists jobs on Jenkins CI, though there are some additional jobs there that are not in jjb. I thinkt hey are mostly one off manual jobs I created for debugging stuff, I usually prefix them with `hashar-` [13:43:23] but there might be some more [13:43:44] JJB has support for deleting jobs it doesn't know about, but since we have jobs that are not defined in jjb... I never used it [13:43:48] anyway [13:44:30] and JJB defines some variants of jobs that are not triggered by Zuul cause the jjb projects generate a cartesian product of all the variants (phpflavor, db,packages-source etc) even when we don't use some of those combinations [14:25:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 06DC-Ops, and 2 others: eqiad: request for a decom'ed R440 - Config C - https://phabricator.wikimedia.org/T418544#11688018 (10VRiley-WMF) Hey @Dzahn for some reason the server had reverted back with it's IP addre... [14:30:54] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11688033 (10Daimona) >>! In T4191... [14:34:27] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 07ci-test-error (WMF-deployed Build Failure): mwext-phpunit-coverage-patch no longer runs any tests for modified files - https://phabricator.wikimedia.org/T419073#11688051 (10Daimona) **Clarification on the status of this task**: The cause has been i... [14:57:18] 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Split PrivateSettings into config and business logic - https://phabricator.wikimedia.org/T390773#11688161 (10Clement_Goubert) >>! In T390773#11685687, @Tgr wrote: > [...] > # Update [[https://codesearch.wmcloud.org/search/?q=Priv... [15:45:16] FIRING: [7x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:50:31] FIRING: [7x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:55:46] FIRING: [7x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:58:33] (03PS1) 10Pppery: Zuul: sort archived repo list [integration/config] - 10https://gerrit.wikimedia.org/r/1249340 [15:58:59] (03PS2) 10Pppery: Zuul: sort archived repo list [integration/config] - 10https://gerrit.wikimedia.org/r/1249340 [15:59:40] (03PS3) 10Pppery: Zuul: sort archived repo list [integration/config] - 10https://gerrit.wikimedia.org/r/1249340 [16:03:57] (03PS1) 10Pppery: Utils/archive_repo.py: Fix sorting order [integration/config] - 10https://gerrit.wikimedia.org/r/1249342 [16:04:18] (03CR) 10Pppery: "The reason I wasn't using it was I41feaa0cb41ec9ceafc3b4852232374d20d50215. But I should probably have just written a patch to fix that bu" [integration/config] - 10https://gerrit.wikimedia.org/r/1248968 (https://phabricator.wikimedia.org/T418008) (owner: 10Pppery) [16:05:48] (03CR) 10CI reject: [V:04-1] Utils/archive_repo.py: Fix sorting order [integration/config] - 10https://gerrit.wikimedia.org/r/1249342 (owner: 10Pppery) [16:11:31] FIRING: [7x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:12:00] (03PS2) 10Pppery: Utils/archive_repo.py: Fix sorting order [integration/config] - 10https://gerrit.wikimedia.org/r/1249342 [16:12:21] (03PS3) 10Pppery: Utils/archive_repo.py: Fix sorting order [integration/config] - 10https://gerrit.wikimedia.org/r/1249342 [16:12:31] FIRING: [7x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:12:37] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440 (10wmcs-alerts) 03NEW [16:16:46] RESOLVED: [7x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:17:57] 06Release-Engineering-Team (Radar), 10Ceph, 06ServiceOps new, 10SRE-swift-storage, and 2 others: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - https://phabricator.wikimedia.org/T412951#11688626 (10elukey) 05Open→03Resolved a:03elukey The new settings see... [16:24:58] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T419099#11688671 (10elukey) 05Open→03Resolved a:03elukey ` elukey@deployment-cache-upload08:~$ sudo run-pu... [16:39:32] 10Gerrit, 06collaboration-services, 07Sustainability (Incident Followup): Update user facing communication to reflect availability of Gerrit during maintenance - https://phabricator.wikimedia.org/T417135#11688791 (10Dzahn) I would say this is declined. per "things have changed" and "not actionable". @MLechvi... [16:39:54] 10Gerrit, 06collaboration-services, 07Sustainability (Incident Followup): Update user facing communication to reflect availability of Gerrit during maintenance - https://phabricator.wikimedia.org/T417135#11688792 (10Dzahn) 05Open→03Declined [17:03:39] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 06DC-Ops, and 2 others: eqiad: request for a decom'ed R440 - Config C - https://phabricator.wikimedia.org/T418544#11688966 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin200... [17:13:24] 10Gerrit: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787#11689002 (10Daimona) >>! In T267787#6658893, @hashar wrote: > But my assumption is wrong. It instead processes an email box. Maybe Gerrit does not send an email when a WIP patch is created and... [17:36:37] elukey: thanks for fixing that CFSSL problem. Much appreciated [17:51:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 06DC-Ops, and 2 others: eqiad: request for a decom'ed R440 - Config C - https://phabricator.wikimedia.org/T418544#11689200 (10Dzahn) Hi @VRiley-WMF Thanks! Now I can reach the DRAC mgmt console. I started the re... [17:58:58] (03CR) 10Jforrester: [C:03+2] Zuul: sort archived repo list [integration/config] - 10https://gerrit.wikimedia.org/r/1249340 (owner: 10Pppery) [17:59:47] (03PS4) 10Jforrester: utils/archive_repo.py: Fix sorting order [integration/config] - 10https://gerrit.wikimedia.org/r/1249342 (owner: 10Pppery) [17:59:49] (03CR) 10Jforrester: [C:03+2] utils/archive_repo.py: Fix sorting order [integration/config] - 10https://gerrit.wikimedia.org/r/1249342 (owner: 10Pppery) [18:01:41] (03CR) 10Jforrester: [C:03+2] utils: Provide a script to find unused Jenkins Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1248901 (owner: 10Jforrester) [18:02:02] (03CR) 10Jforrester: [C:03+2] utils: Provide a script to find unused Docker images [integration/config] - 10https://gerrit.wikimedia.org/r/1248903 (owner: 10Jforrester) [18:02:05] (03Merged) 10jenkins-bot: Zuul: sort archived repo list [integration/config] - 10https://gerrit.wikimedia.org/r/1249340 (owner: 10Pppery) [18:02:21] hi! do we need to coordinate about archiving a github repo (T419226) or is that something the team owning the library can just do on their own? [18:02:22] T419226: Archive wikimedia/oauth2-server - https://phabricator.wikimedia.org/T419226 [18:03:15] (03Merged) 10jenkins-bot: utils/archive_repo.py: Fix sorting order [integration/config] - 10https://gerrit.wikimedia.org/r/1249342 (owner: 10Pppery) [18:04:27] (03Merged) 10jenkins-bot: utils: Provide a script to find unused Jenkins Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1248901 (owner: 10Jforrester) [18:04:33] (03Merged) 10jenkins-bot: utils: Provide a script to find unused Docker images [integration/config] - 10https://gerrit.wikimedia.org/r/1248903 (owner: 10Jforrester) [18:05:33] (03CR) 10Jforrester: "recheck" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1244816 (https://phabricator.wikimedia.org/T414621) (owner: 10Jforrester) [18:18:49] (03PS2) 10Jforrester: plugins/wm-pcc: Switch commands from experimental to new puppet [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1244816 (https://phabricator.wikimedia.org/T414621) [18:18:49] (03PS1) 10Jforrester: build: Upgrade eslint-config-wikimedia from 0.23.0 to 0.32.3 and make pass [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1249368 [18:20:12] (03CR) 10CI reject: [V:04-1] build: Upgrade eslint-config-wikimedia from 0.23.0 to 0.32.3 and make pass [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1249368 (owner: 10Jforrester) [18:23:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 06DC-Ops, and 2 others: eqiad: request for a decom'ed R440 - Config C - https://phabricator.wikimedia.org/T418544#11689371 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 fo... [18:27:43] if a patch to core landed yesterday, will it be included in the release that goes out this week? a 1.46.0-wmf.19 branch does not appear to have been cut yet. [18:30:37] (03PS1) 10Hashar: dockerfiles: [quibble-coverage] keep XML comments [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) [18:32:06] (03CR) 10CI reject: [V:04-1] dockerfiles: [quibble-coverage] keep XML comments [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) (owner: 10Hashar) [18:33:26] (03PS2) 10Hashar: dockerfiles: [quibble-coverage] keep XML comments [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) [18:33:41] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: mwext-phpunit-coverage-patch no longer runs any tests for modified files - https://phabricator.wikimedia.org/T419073#11689408 (10hashar) @Daimona is there a week when you are not a h... [18:35:33] (03CR) 10CI reject: [V:04-1] dockerfiles: [quibble-coverage] keep XML comments [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) (owner: 10Hashar) [18:36:36] (03PS3) 10Hashar: dockerfiles: [quibble-coverage] keep XML comments [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) [18:37:49] (03CR) 10Daimona Eaytoy: dockerfiles: [quibble-coverage] keep XML comments (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) (owner: 10Hashar) [18:39:07] (03CR) 10Hashar: "We should port those scripts to PHP and attached them to mediawiki/core as a composer dependency. It is a pain to maintain here, then it " [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) (owner: 10Hashar) [18:39:37] (03CR) 10Hashar: "It is too late for me to test/verify/deploy this. I'll look at it tomorrow I guess." [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) (owner: 10Hashar) [18:42:44] (03CR) 10Hashar: dockerfiles: [quibble-coverage] keep XML comments (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) (owner: 10Hashar) [18:42:49] (03PS4) 10Hashar: dockerfiles: [quibble-coverage] keep XML comments [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) [18:43:13] Daimona: I can't continue on that XML parsing of fun, it is dinner time and I am tired :) [18:43:40] Yeah, same for me, I'll need to go eat something :D I may take another look later [18:43:48] sounds goood! [18:43:53] Thank you all the same, I'll comment on the patch [18:44:00] if you can find an example phpunit.xml to use as input that would be nice [18:45:47] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: mwext-phpunit-coverage-patch no longer runs any tests for modified files - https://phabricator.wikimedia.org/T419073#11689480 (10hashar) Side note: overall those scripts should proba... [18:46:26] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#11689488 (10Daimona) I made [[htt... [18:47:22] You can use https://gerrit.wikimedia.org/g/mediawiki/core/+/d2863c4743f5f1f4a80b178d5b21524957c6e89e/phpunit.xml.template . The config generation process doesn't alter the comment structure, only the comment [18:47:30] *only the content [19:20:00] Project beta-update-databases-eqiad build #91892: 04FAILURE in 0.53 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/91892/ [19:23:21] (03update) 10dduvall: app.py: Lowercase identity claim value [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/108 [19:23:24] (03update) 10dduvall: app.py: Lowercase identity claim value [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/108 [19:23:30] (03open) 10dduvall: app.py: Lowercase identity claim value [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/108 [19:54:16] (03open) 10fab: vector-search: Traffic pattern similarity search [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/150 [20:24:05] Yippee, build fixed! [20:24:05] Project beta-update-databases-eqiad build #91893: 09FIXED in 4 min 5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/91893/ [20:27:36] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan: Bogus PhanPluginDuplicateArrayKey error in MediaWiki-config - https://phabricator.wikimedia.org/T419476 (10Tgr) 03NEW [20:28:58] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan: Bogus PhanPluginDuplicateArrayKey error in MediaWiki-config - https://phabricator.wikimedia.org/T419476#11689963 (10Tgr) (Also, why is it only being checked during the gate pipeline, not during normal tests?) [20:30:33] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T413810#11689981 (10brennen) [20:31:22] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11689984 (10Tgr) [20:32:14] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11689991 (10Tgr) [20:36:07] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan, 13Patch-For-Review: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11690004 (10Jdforrester-WMF) >>! In T419476#11689963, @Tgr wrote: > (Also, why is it only being checked during the... [20:36:43] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan, 13Patch-For-Review: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11690006 (10Jdforrester-WMF) I cannot replicate your CI failure locally. I've pushed a patch to bump Phan to the cu... [20:47:22] why is the castor-save-workspace-cache job failing in gate-and-submit? eg https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-phpunit-standalone/3618/console [20:48:17] INFO:quibble.commands:>>> Start: Save success cache [20:48:17] INFO:quibble.commands:Saving success cache entry: successcache/9896b21e8f2f915bdbbea66c61a379906616a92be661d18e0c2f3168a1295731 [20:48:17] INFO:quibble.commands:<<< Finish: Save success cache, in 0.000 s [20:48:17] ... [20:48:17] [PostBuildScript] - [INFO] Executing post build scripts. [20:48:17] Waiting for the completion of castor-save-workspace-cache [20:48:17] Build step 'Execute scripts' changed build result to FAILURE [20:48:18] Build step 'Execute scripts' marked build as failure [20:55:47] I've been seeing the same kind of failure a lot lately, been wondering the same [20:56:07] Which is obviously unhelpful, but just saying it isn't an isolated case [21:14:16] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: mwext-phpunit-coverage-patch no longer runs any tests for modified files - https://phabricator.wikimedia.org/T419073#11690093 (10Daimona) >>! In T419073#11689408, @hashar wrote: > Wi... [21:17:34] (03CR) 10Daimona Eaytoy: dockerfiles: [quibble-coverage] keep XML comments (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1249372 (https://phabricator.wikimedia.org/T419073) (owner: 10Hashar) [21:27:04] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan, 13Patch-For-Review: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11690140 (10Umherirrender) Could be a bad zuul merge between master and the patch set (due to the chain of patch se... [21:36:32] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan, 13Patch-For-Review: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11690169 (10Umherirrender) Reproducable when fetching patch set 2 and merging master into the new branch (witho... [21:36:48] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan, 13Patch-For-Review: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11690172 (10Umherirrender) →14Duplicate dup:03T400918 [21:36:52] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 07ci-test-error (WMF-deployed Build Failure), 06Wikidata-Omega (Radar/Epics/Stalled): Zuul does not support Gerrit rebase submit strategy correctly, may merge deterministically broken code (bre... - https://phabricator.wikimedia.org/T400918#11690174 [21:37:45] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440#11690182 (10bd808) `lang=shell-session bd808@deployment-shellbox01:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrievin... [21:51:54] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440#11690219 (10bd808) The fact that things are trying to run `docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox:2024-06-13-133425-video`... [21:53:28] !log Reboot deployment-shellbox01 on the off chance that is makes the new permissions error go away (T419440) [21:53:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:53:31] T419440: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440 [21:54:12] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan, 13Patch-For-Review: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11690226 (10Tgr) Thanks for investigating! So IIUIC the error will go away if I just rebase the patch via the... [21:58:06] 10Continuous-Integration-Config, 10MediaWiki-Configuration, 10phan, 13Patch-For-Review: Bogus PhanPluginDuplicateArrayKey error in mediawiki-config - https://phabricator.wikimedia.org/T419476#11690234 (10Umherirrender) In this case: Yes, the error is gone after rebase to current master (as the chain is... [22:13:56] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440#11690286 (10bd808) The reboot hope failed. I'm not seeing obvious changes in ops/puppet.git in the last 5-7 hours that look like they would have ch... [23:23:22] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440#11690469 (10bd808) I'm not sure why there isn't a ticket for deployment-shellbox-video, but it seems to have the same problem. `lang=shell-session,... [23:30:58] bd808: so regarding beta and CSP - it does appear to be blocking things that it shouldn't. I don't think that's due to TLD issues per-se, my guess is that the list of domains is overridden there and it's been insufficient for a while but we didn't notice because it wasn't enforced. Or.. maybe we're not overriding it at all and we're enforcing an empty list or prod's list [23:31:15] I'm doing something else first, but once done with that I might look at that, maybe it's known/WIP already? [23:36:06] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440#11690488 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/1eb1e98e55e9116c20021a58c6d351b91d43c518%5E%21/#F0 ` diff... [23:37:12] 10Continuous-Integration-Config, 07ci-test-error: PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488 (10Daimona) 03NEW [23:38:02] 10Continuous-Integration-Config, 07ci-test-error: PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488#11690499 (10Daimona) [23:38:48] cscott: ^ for the CI issue you mentioned above [23:38:51] Krinkle: it seems likely to me that the policy needs to vary somehow in Beta Cluster. The current implementation seems to be a single hard coded string in modules/varnish/templates/text-frontend.inc.vcl.erb [23:41:01] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488#11690502 (10A_smart_kitten) [23:46:56] bd808: the vcl is a copy of what we got from curling mediawiki directly. It's mainly there so that it would apply to cache hits immediately. [23:47:13] instead of taking 7-14 days to apply [23:47:33] (30 days if we include varnish/ats renewing 304 responses) [23:47:50] oh I see what you mean [23:47:56] that vcl is active in beta [23:48:02] that would do it yeah [23:48:03] correct [23:48:46] ok, well, there is intent to remove that from vcl "soon", given it shoudl be identical to the one from MW. [23:48:49] when things get jammed into VCL without a feature flag then they are in Beta [23:48:58] depending on how soon, we can get it conditional for realm yeah and not apply it. [23:49:28] to be clear: it doesnt' need to be there and the "real" one from MW works fine and varies by realm already [23:49:34] this is board problem that I would have thought folks had caught on to by now, but I guess not [23:49:39] *broad [23:49:58] well, we were under pressure on Friday I think it's fine to have not considered that right then and there [23:50:10] any other time I would have agreed :) [23:50:51] are the changes like https://github.com/wikimedia/operations-puppet/commit/d6b652e20785dbab20cb29b72f19028c473f05d8 being applied to both Puppet and wmf-config? [23:52:22] Krinkle: yeah Friday was a reasonable time to ignore Beta Cluster [23:55:22] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-shellbox01 in project deployment-prep - https://phabricator.wikimedia.org/T419440#11690557 (10bd808) 05Open→03Resolved a:03bd808 I am going to mark this as resolved because I am just not curious enough to keep it open a...