[01:11:39] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 06ServiceOps new, 10ServiceOps-Mediawiki: helmfile/scap does not reliably bootstrap mediawiki - https://phabricator.wikimedia.org/T397685#11577093 (10Scott_French) Alright, I've updated the task description on {T405703} to reflect two points: * During the... [04:40:31] (03update) 10mstyles: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) [09:33:24] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Gerrit: Gerrit Zuul plugin "Needed by" table links to search of change-id instead of individual commits - https://phabricator.wikimedia.org/T415456#11577729 (10hashar) [09:48:03] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Investigate WMCS Magnum for GitLab runners - https://phabricator.wikimedia.org/T403125#11577791 (10fnegri) Awesome work @dduvall, very excited to see this working! <3 [09:51:56] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Gerrit: Gerrit Zuul plugin "Needed by" table links to search of change-id instead of individual commits - https://phabricator.wikimedia.org/T415456#11577820 (10hashar) I don't know what kind of data is known/reported by the Gerrit Zuul plugin, most prob... [09:57:00] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Gerrit: Gerrit Zuul plugin "Needed by" table links to search of change-id instead of individual commits - https://phabricator.wikimedia.org/T415456#11577846 (10hashar) The change is straightforward but I don't have quite the time to validate/test it bec... [09:59:18] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11577852 (10Samwilson) T416294 is breaking Special:Watchlist in MobileFrontend. A fix has been merged, but should I think be backp... [10:55:03] maintenance-disconnect-full-disks build 777827 integration-agent-docker-1042 (/: 37%, /srv: 98%, /var/lib/docker: 31%): OFFLINE due to disk space [11:00:03] maintenance-disconnect-full-disks build 777828 integration-agent-docker-1042 (/: 37%, /srv: 65%, /var/lib/docker: 30%): RECOVERY disk space OK [11:37:25] (03PS1) 10Hashar: Zuul: [BlueSpiceDistributionConnector] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236268 (https://phabricator.wikimedia.org/T389998) [11:54:22] (03PS1) 10Hashar: Zuul: [BlueSpiceProDistributionConnector] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236272 (https://phabricator.wikimedia.org/T389998) [11:54:56] (03CR) 10Hashar: [C:03+2] Zuul: [BlueSpiceProDistributionConnector] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236272 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [11:56:51] (03Merged) 10jenkins-bot: Zuul: [BlueSpiceProDistributionConnector] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236272 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [12:03:40] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/Petition] Add CLDR dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1236274 [13:32:26] (03PS2) 10Hashar: Zuul: [BlueSpiceDistributionConnector] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236268 (https://phabricator.wikimedia.org/T389998) [13:43:34] 10Gerrit, 06collaboration-services: Add monitoring and alerting for gerrits tcp-proxy - https://phabricator.wikimedia.org/T416189#11578635 (10Jelto) p:05Triage→03Medium I started to collect some relevant dashboards under [Gerrit TCP Proxy](https://grafana-rw.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c18... [13:43:55] 10Gerrit, 06collaboration-services: Add monitoring and alerting for gerrits tcp-proxy - https://phabricator.wikimedia.org/T416189#11578639 (10Jelto) [14:04:33] (03PS1) 10Hashar: Zuul: [BlueSpicePageTemplates] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236299 [14:05:36] (03CR) 10Hashar: [C:03+2] Zuul: [BlueSpicePageTemplates] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236299 (owner: 10Hashar) [14:07:07] (03Merged) 10jenkins-bot: Zuul: [BlueSpicePageTemplates] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236299 (owner: 10Hashar) [15:16:20] 10Phabricator: Phabricator is very slow, with search and views of projects particularly slow - https://phabricator.wikimedia.org/T416359#11579052 (10xcollazo) p:05Triage→03High [15:18:58] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phabricator is very slow, with search and views of projects particularly slow - https://phabricator.wikimedia.org/T416359#11579066 (10brennen) [15:25:54] (03PS3) 10Hashar: Zuul: [BlueSpiceDistributionConnector] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236268 (https://phabricator.wikimedia.org/T389998) [15:26:39] (03CR) 10Hashar: [C:03+2] "Finally it passed 😎" [integration/config] - 10https://gerrit.wikimedia.org/r/1236268 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:27:40] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phabricator is very slow, with search and views of projects particularly slow - https://phabricator.wikimedia.org/T416359#11579099 (10xcollazo) [15:28:16] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phabricator is very slow, with search and views of projects particularly slow - https://phabricator.wikimedia.org/T416359#11579104 (10Jelto) a:03Jelto this is being investigated on a private irc channel, I'm working on a solution. thanks... [15:28:17] (03Merged) 10jenkins-bot: Zuul: [BlueSpiceDistributionConnector] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236268 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:31:38] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phabricator is very slow, with search and views of projects particularly slow - https://phabricator.wikimedia.org/T416359#11579130 (10Jelto) [15:45:53] 10Continuous-Integration-Config, 10Quibble: Check mw-error.log before Save success cache in CI run - https://phabricator.wikimedia.org/T400548#11579231 (10hashar) Timo once added an assertion that the error logs are empty at the end of the build and dump their content if not: ` name=jjb/assert-no-mediawiki-err... [15:46:31] (03PS1) 10Hashar: Zuul: [BlueSpiceExtendedSearch] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236321 [15:46:32] (03CR) 10Hashar: [C:03+2] Zuul: [BlueSpiceExtendedSearch] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236321 (owner: 10Hashar) [15:47:58] (03Merged) 10jenkins-bot: Zuul: [BlueSpiceExtendedSearch] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236321 (owner: 10Hashar) [15:49:35] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phabricator is very slow, with search and views of projects particularly slow - https://phabricator.wikimedia.org/T416359#11579265 (10Jelto) A [rate limiting action](https://requestctl.wikimedia.org/action/cache-text/phabricator_googlebot)... [15:55:47] (03PS1) 10Hashar: Zuul: [BlueSpicePagesVisited] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236325 [15:55:49] (03CR) 10Hashar: [C:03+2] Zuul: [BlueSpicePagesVisited] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236325 (owner: 10Hashar) [15:57:13] (03Merged) 10jenkins-bot: Zuul: [BlueSpicePagesVisited] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236325 (owner: 10Hashar) [15:59:11] (03open) 10dancy: web: Replace v-model:open with explicit bindings in dialogs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1102 [15:59:15] (03update) 10dancy: web: Replace v-model:open with explicit bindings in dialogs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1102 [15:59:15] (03PS5) 10Hashar: Zuul: [BlueSpiceSMWConnector] re-enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/1236285 [15:59:15] (03CR) 10Hashar: [C:03+2] "It passed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1236285 (owner: 10Hashar) [16:00:48] (03Merged) 10jenkins-bot: Zuul: [BlueSpiceSMWConnector] re-enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/1236285 (owner: 10Hashar) [16:01:34] (03merge) 10dancy: web: Replace v-model:open with explicit bindings in dialogs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1102 [16:03:06] 10Continuous-Integration-Config, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11579355 (10hashar) I am down to 24 extensions still requiring recursion, most o... [16:10:59] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phabricator is very slow, with search and views of projects particularly slow - https://phabricator.wikimedia.org/T416359#11579388 (10Jelto) 05Open→03Resolved I'll resolve the task, please reopen if there are any more problems Tha... [16:18:39] Project mediawiki-core-doxygen build #17485: 04FAILURE in 36 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17485/ [16:34:11] Yippee, build fixed! [16:34:11] Project mediawiki-core-doxygen build #17486: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17486/ [16:40:12] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Security-Team, 06SRE, and 2 others: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11579521 (10Dzahn) 05Stalled→03Open a:05DannyS712→03None [16:47:52] (03open) 10dancy: Train.vue: Add a top-level div [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1103 [16:47:57] (03update) 10dancy: Train.vue: Add a top-level div [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1103 [16:55:29] (03open) 10dancy: web: Upgrade to ESLint 9.39.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1104 [16:55:35] (03update) 10dancy: web: Upgrade to ESLint 9.39.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1104 [16:59:34] (03update) 10dancy: Train.vue: Add a top-level div [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1103 [16:59:39] (03update) 10dancy: Train.vue: Add a top-level div [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1103 [17:01:09] (03merge) 10dancy: Train.vue: Add a top-level div [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1103 [17:22:58] (03update) 10dancy: web: Upgrade to ESLint 9.39.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1104 [17:25:29] 10Phabricator (Upstream), 07Upstream: Make it more obvious when attempting to add both a parent and subproject - https://phabricator.wikimedia.org/T416231#11579810 (10Aklapper) p:05Triage→03Low This is currently implemented via `applyProjectConflictRules()` in https://we.phorge.it/source/phorge/browse/mast... [17:26:40] 06Project-Admins, 10MediaWiki-Core-Tests, 06MediaWiki-Engineering: The api-testing library has no specific phab tag and its docs link to an archived one - https://phabricator.wikimedia.org/T415836#11579820 (10Aklapper) [17:28:19] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1236342 [17:28:49] (03open) 10dancy: web: Linter fixes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1105 [17:28:53] (03update) 10dancy: web: Linter fixes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1105 [17:28:57] 10Phabricator (2026-01-13), 07Mobile: Gerrit patch status comments have/cause inconsistent spacing on mobile devices - https://phabricator.wikimedia.org/T414003#11579830 (10Aklapper) 05Open→03Resolved The related merge request was deployed on 2026-01-13; closing as resolved. [17:30:10] (03merge) 10dancy: web: Linter fixes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1105 [17:40:03] maintenance-disconnect-full-disks build 777908 integration-agent-docker-1042 (/: 36%, /srv: 96%, /var/lib/docker: 30%): OFFLINE due to disk space [17:45:03] maintenance-disconnect-full-disks build 777909 integration-agent-docker-1042 (/: 36%, /srv: 91%, /var/lib/docker: 30%): RECOVERY disk space OK [17:55:03] maintenance-disconnect-full-disks build 777911 integration-agent-docker-1042 (/: 36%, /srv: 97%, /var/lib/docker: 30%): OFFLINE due to disk space [18:00:03] maintenance-disconnect-full-disks build 777912 integration-agent-docker-1042 (/: 36%, /srv: 82%, /var/lib/docker: 30%): RECOVERY disk space OK [18:01:36] (03update) 10dancy: web: Upgrade to ESLint 9.39.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1104 [18:01:42] (03update) 10dancy: web: Upgrade to ESLint 9.39.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1104 [18:04:02] (03merge) 10dancy: web: Upgrade to ESLint 9.39.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1104 [19:04:57] 10Phabricator (Upstream), 07CSS, 07Upstream: Custom Policy window overflows when task name is too long - https://phabricator.wikimedia.org/T316308#11580334 (10Aklapper) [19:13:23] 10Beta-Cluster-Infrastructure, 10Page Content Service, 06Content-Transform-Team (Work In Progress), 13Patch-For-Review: PCS/summary returns errors for beta requests - https://phabricator.wikimedia.org/T415877#11580385 (10brennen) Am I correct in thinking this should no longer block with https://gerrit.wiki... [19:49:25] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Security-Team, 06SRE, and 2 others: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11580560 (10A_smart_kitten) >>! In T413634#11489204, @A_smart_kitten wrote: > @dannys712 should your entries on https://www.mediawiki.org/wiki/Git/Re... [19:50:30] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Security-Team, 06SRE, and 2 others: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11580561 (10sbassett) Anything else left to do here? [19:51:47] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Security-Team, 06SRE, and 2 others: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11580566 (10A_smart_kitten) >>! In T413634#11580561, @sbassett wrote: > Anything else left to do here? FWICS there are still the additional Gerrit r... [20:48:13] 10Phabricator (Upstream), 07Upstream: Make it more obvious when attempting to add both a parent and subproject - https://phabricator.wikimedia.org/T416231#11580884 (10Izno) I have no issue with a feature request upstream, but IIRC it's a bit limited to contribute upstream. :) [21:04:24] 10Phabricator, 07CSS: Gerrit comments for patch abandonments without a provided reason are displayed with a green icon (indicating a merged patch) rather than a red icon (indicating an abandoned patch) - https://phabricator.wikimedia.org/T414028#11580957 (10Aklapper) p:05Triage→03Low Does that mean this is... [21:10:22] (03PS1) 10Hashar: Zuul: [MobileFrontend] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236329 (https://phabricator.wikimedia.org/T389998) [21:12:16] (03Merged) 10jenkins-bot: Zuul: [MobileFrontend] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1236329 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [21:40:00] I have finished migrating BlueSpice extensions today. I am left with GrowthExperiments (and the one it breaks: CheckUser, MetricsPlatform/TestKitchen) [21:40:06] + all the Wikibase stuff [21:40:31] so I guess I am not that far from disabling recursion entirely [21:53:19] 10Phabricator, 07CSS: Gerrit comments for patch abandonments without a provided reason are displayed with a green icon (indicating a merged patch) rather than a red icon (indicating an abandoned patch) - https://phabricator.wikimedia.org/T414028#11581112 (10A_smart_kitten) I would personally say 'no' (although... [22:31:27] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11581186 (10hashar) >>! In T413805#11577852, @Samwilson wrote: > T416294 is breaking Special:Watchlist in MobileFrontend. A fix ha... [22:31:39] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11581188 (10hashar) [22:34:01] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11581205 (10hashar) Per @Samwilson , I have made T416294 a blocker. That is primarily so I don't forget about it tomorrow morning... [22:35:08] brennen: thanks for the train blockers triage / poking etc ! :] [22:35:36] I confess I haven't watched the task much today (I took care of kids, finished other CI duties) [22:35:44] I'll check tomorrow morning and see how things are [22:35:50] * hashar yawns & sleep [22:40:25] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11581216 (10Samwilson) Thanks! Next time I'll be bold add such a thing as a subtask. :) [22:42:54] 10Beta-Cluster-Infrastructure, 10Page Content Service, 06Content-Transform-Team (Work In Progress), 13Patch-For-Review: PCS/summary returns errors for beta requests - https://phabricator.wikimedia.org/T415877#11581223 (10BPirkle) Just noticed this task. The error was fixed under {T415771} via https://gerri... [22:44:26] sleep well! [23:38:54] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11581336 (10Dzahn) [23:41:16] (03open) 10thcipriani: JobCard: Format time and date [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1106 [23:41:19] (03update) 10thcipriani: JobCard: Format time and date [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1106 [23:41:32] 10Gerrit, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#11581341 (10Dzahn) So.. meanwhile Gerrit is available behind the CDN (T365259) which was implemented in T411895 and is using the new [ht... [23:41:43] 10Gerrit, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#11581343 (10Dzahn) 05Stalled→03Open [23:44:51] 10Gerrit, 06collaboration-services: Opt-in testing of Gerrit-via-CDN - https://phabricator.wikimedia.org/T414719#11581346 (10Dzahn) checked the box "//One full business day of testing with several volunteers//" on T411895 How about we send a mail to ops-at-large about this now as the next step? [23:47:18] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11581351 (10Dzahn) At this point I think we can say that T394271 **Separate Gerrit https and ssh/git hostnames** was not needed after all and can be closed as declined. T394271#11581341 [23:48:42] (03update) 10thcipriani: JobCard: Format time and date [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1106 [23:57:33] 10Gerrit, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#11581366 (10thcipriani) 05Open→03Declined >>! In T394271#11581341, @Dzahn wrote: > So.. meanwhile Gerrit is available behind the... [23:59:07] 10Gerrit, 06collaboration-services: Gerrit failover process - https://phabricator.wikimedia.org/T412779#11581372 (10Dzahn) Now that Gerrit is (more or less) behind the CDN (T411895), there will be this step to actually change backends in the future. `common/profile/trafficserver/backend.yaml` configures: `...