[00:01:04] (03CR) 10Jforrester: [C:03+1] "LGTM. Did you want Željko to merge?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1247554 (https://phabricator.wikimedia.org/T418461) (owner: 10Hashar) [00:04:28] (03CR) 10Santiago Faci: Zuul: [mediawiki/extensions/MetricsPlatform] Archive (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1247562 (https://phabricator.wikimedia.org/T416865) (owner: 10Santiago Faci) [01:26:19] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T413809#11671569 (10Zabe) [01:29:59] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 06MediaWiki-Platform-Team, 10Projects-Cleanup, 05MW-1.39-notes: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11671571 (10Krinkle) [01:34:37] 10Beta-Cluster-Infrastructure, 06Traffic: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T418884#11671573 (10bd808) 05Open→03Invalid Looks like this was a temporary overload that broke the Prometheus scrape: `lang=shell-session bd808@deployment-cach... [01:36:06] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 06MediaWiki-Platform-Team, 10Projects-Cleanup, 05MW-1.39-notes: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11671576 (10Krinkle) Yeah, until we solve T373933, we need to keep GitHub mirrors of repos that back... [03:15:08] Project beta-code-update-eqiad build #590104: 04FAILURE in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/590104/ [03:20:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:20:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T418970 (10phaultfinder) 03NEW [03:25:15] Yippee, build fixed! [03:25:15] Project beta-code-update-eqiad build #590105: 09FIXED in 2 min 15 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/590105/ [03:25:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:24:28] 10Continuous-Integration-Infrastructure, 07Jenkins: Waiting for the completion of castor-save-workspace-cache sometimes takes almost 4 minutes for core tests - https://phabricator.wikimedia.org/T418974 (10Peter) 03NEW [07:37:19] (03CR) 10Phedenskog: [C:03+1] Write durations report to the log_dir as json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [07:55:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2003:443) - https://phabricator.wikimedia.org/T418970#11671892 (10Jelto) [07:56:40] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2003:443) - https://phabricator.wikimedia.org/T418970#11671897 (10Jelto) 05Open→03Resolved a:03Jelto That happened during a CDN incident (https://portal.victorops.com/ui/wikimedia/incident/7705/details). It resolved after 5 minute... [09:05:02] (03CR) 10Hashar: "At least verify it yes, I have daily meetings with Peter and Zeljko and this issue surfaced while playing with Quibble :-]" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1247554 (https://phabricator.wikimedia.org/T418461) (owner: 10Hashar) [09:20:56] (03CR) 10Zfilipin: [C:03+1] Fix --shell when SHELL is not set [integration/quibble] - 10https://gerrit.wikimedia.org/r/1247554 (https://phabricator.wikimedia.org/T418461) (owner: 10Hashar) [09:21:55] (03CR) 10Hashar: [C:03+2] Fix --shell when SHELL is not set [integration/quibble] - 10https://gerrit.wikimedia.org/r/1247554 (https://phabricator.wikimedia.org/T418461) (owner: 10Hashar) [09:41:40] (03CR) 10CI reject: [V:04-1] Fix --shell when SHELL is not set [integration/quibble] - 10https://gerrit.wikimedia.org/r/1247554 (https://phabricator.wikimedia.org/T418461) (owner: 10Hashar) [09:44:08] hashar: we're block on CI with issues that only affect the 8.4 jobs: https://phabricator.wikimedia.org/T418981 . [09:44:27] should we focus on fixing the issues? Or is there an option to make them non-voting for Wikibase for now? [09:45:43] (Wikibase and I guess WikibaseLexeme, and possibly others) [09:48:30] (03PS4) 10Hashar: jjb, Zuul: wikibase-… => quibble-with-wikibase-extensions… for clarity [integration/config] - 10https://gerrit.wikimedia.org/r/1238317 (owner: 10Jforrester) [09:49:56] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit-spare behind CDN - https://phabricator.wikimedia.org/T418361#11672272 (10ABran-WMF) [09:50:09] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11672273 (10ABran-WMF) [09:52:43] codders: James_F turned on PHP 8.4 earlier this week https://gerrit.wikimedia.org/r/c/integration/config/+/1240989 [09:53:00] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-spare behind CDN - https://phabricator.wikimedia.org/T418361#11672288 (10ABran-WMF) [09:53:47] yeah. we've been struggling to merge things since then :) [09:54:15] but we didn't notice right away because we were blocked on other things [09:54:30] sounds like the strategy would be to try and resolve the issues rather than disabling the jobs again [09:58:07] codders: yeah I commented on https://phabricator.wikimedia.org/T418981#11672300 [09:58:13] (03CR) 10Phedenskog: [C:03+1] "Would it be possible to include if there a success or failure? Thinking if we want to separate the result somehow." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [09:58:47] thanks! [09:59:07] codders: there has been an issue with fgetcsv() which was fixed via https://phabricator.wikimedia.org/T386113 and has a bunch of patches associated with it [09:59:22] that new task is about fputcsv() which I guess is closely related and might need a similar fix [09:59:31] why it was not caught before? I DON'T KNOW [09:59:40] it feels to me it might be an easy fix in Wikibase [10:10:36] I made a patch for propertysuggester [10:11:24] as jakob_WMDE says, there are some interdependencies there. Will have to see if that will merge. And I'm not sure about the other issue ('slottedEntityType') [10:12:50] yeah - so the PropertySuggester patch fails on the scribunto issue. And we can't merge the scribunto issue until the 8.4 job for Wikibase passes... [10:20:39] 10GitLab (Account Approval): Request to unlock GitLab account for Mounil2005 for contributing to VideoCutTool (T331235) - https://phabricator.wikimedia.org/T418987 (10Mounil2005) 03NEW [10:22:12] 10GitLab (Account Approval): Request to unlock GitLab account for Mounil2005 for contributing to VideoCutTool (T331235) - https://phabricator.wikimedia.org/T418987#11672432 (10Mounil2005) [10:23:03] (03open) 10sfaci: TestKitchen renaming (MetricsPlatform => TestKitchen) [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/237 (https://phabricator.wikimedia.org/T416865) [10:25:16] (03update) 10sfaci: TestKitchen renaming (MetricsPlatform => TestKitchen) [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/237 (https://phabricator.wikimedia.org/T416865) [10:30:52] hashar: if we want to produce this image locally, a bookworm-php84-fpm dev image would be handy. Is there an easy way to add that to the registry? [10:31:05] s/produce this image/reproduce this bug/ [10:33:50] codders: I am in a meeting, looks like https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/1247647 is the fix for `fputcsv` under PHP 8.4? [10:34:07] and that one can not be merged because of another unrelated failure? If so I will force merge the fix for fputcsv [10:34:20] yeah, that change looks good to me [10:41:23] codders: I amended it to add a Bug: header [10:41:44] saw that. I added a +2, and the jobs failed in the expected way [10:42:02] codders: what is the other task for that failure? [10:42:12] I'll use as a justification to submit the change / bypass CI [10:42:21] T418774 [10:42:22] T418774: Wikibase CI broken: Call to undefined method WikibaseLibraryTestCase::suite() - https://phabricator.wikimedia.org/T418774 [10:42:43] cool, doing it nowish [10:42:47] thank you! [10:43:13] waiting for the selenium job to complete [10:44:17] 10Continuous-Integration-Infrastructure, 07Jenkins: Waiting for the completion of castor-save-workspace-cache sometimes takes almost 4 minutes for core tests - https://phabricator.wikimedia.org/T418974#11672515 (10hashar) https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Castor [10:47:22] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor: Waiting for the completion of castor-save-workspace-cache sometimes takes almost 4 minutes for core tests - https://phabricator.wikimedia.org/T418974#11672524 (10hashar) [10:50:28] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-spare behind CDN - https://phabricator.wikimedia.org/T418361#11672557 (10ABran-WMF) [10:50:58] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-spare behind CDN - https://phabricator.wikimedia.org/T418361#11672558 (10ABran-WMF) [10:52:11] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11672561 (10ABran-WMF) [10:52:53] codders: merged https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/1247647 [10:54:04] excellent - thank you! We'll take a look at the other blocker, but I think that will probably be in Wikibase itself, so we shouldn't need to force a patch through [10:54:08] codders: and I have `recheck` https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/1247121 [10:54:12] but the 8.4 dev image would help [10:54:12] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-spare behind CDN - https://phabricator.wikimedia.org/T418361#11672568 (10ABran-WMF) 05In progress→03Resolved gerrit-spare now runs behind the CDN [10:54:18] ah yeah hmm [10:55:23] codders: there is a docker-registry.wikimedia.org/quibble-bullseye-php84:latest which is the one used by CI [10:55:32] for the dev image, i guess file a bug for it! :) [10:55:36] thanks! :) [10:58:39] (for mwcli, we need an fpm image, which I guess the CI image is not) [10:59:38] ( /me -> afk ) [11:05:06] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Request to unlock GitLab account for Mounil2005 for contributing to VideoCutTool (T331235) - https://phabricator.wikimedia.org/T418987#11672605 (10Aklapper) 05Open→03Resolved a:03Aklapper Approved. [11:08:07] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-spare behind CDN - https://phabricator.wikimedia.org/T418361#11672630 (10Jelto) Thank you @ABran-WMF for merging the DNS change. I can confirm gerrit-spare host is accessible over the CDN now: ` cur... [11:08:37] codders: the quibble image has a different entry point which spins up Apache + fpm, but is not the default: [11:08:37] docker run --entrypoint=quibble-with-supervisord https://docker-registry.wikimedia.org/quibble-bullseye-php84:latest [11:08:52] that is used for Selenium jobs [11:09:05] and it should certainly be the default entry point in stead of the raw quibble [11:09:07] that is tracked by https://phabricator.wikimedia.org/T397810 [11:10:46] for the php8.4 dev images they are defined in https://gitlab.wikimedia.org/repos/releng/dev-images , and that can surely be requested via a Phabricator task that has #php8.4-support , #releng and whatever tag [11:16:38] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/OpenIDConnect] Add PluggableAuth phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1247972 [11:24:54] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/OpenIDConnect] Add PluggableAuth phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1247972 (owner: 10Fomafix) [11:26:29] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/OpenIDConnect] Add PluggableAuth phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1247972 (owner: 10Fomafix) [12:20:03] Have another propertyselector patch - we missed that there was also fgetcsv in there https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/1247978 [12:20:13] but somehow I can't get CI to run any tests on it, let alone remove the -1 [12:36:38] created https://phabricator.wikimedia.org/T419004 for the 8.4 dev image [12:56:37] (03open) 10arthurtaylor: Add dev images for php-8.4 development [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 (https://phabricator.wikimedia.org/T419004) [12:56:43] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 13Patch-For-Review, 07PHP 8.4 support: Create dev images for php8.4 - https://phabricator.wikimedia.org/T419004#11673000 (10ArthurTaylor) Made a PR here: https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 [13:04:36] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673021 (10ABran-WMF) [13:08:19] 06Release-Engineering-Team, 10dev-images, 10Wikidata, 06Wikidata-Omega, and 2 others: Create dev images for php8.4 - https://phabricator.wikimedia.org/T419004#11673030 (10A_smart_kitten) [13:11:11] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673035 (10ABran-WMF) 05In progress→03Resolved gerrit-replica has been moved behind CDN, everything looks OK [13:19:38] Is CI dead? [13:24:02] zabe: no [13:24:14] can you be more specific? What do you see? [13:25:04] https://integration.wikimedia.org/zuul/ seems to be freezed [13:25:18] and on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1247684 gate-and-submit is not starting while the patch got +2'ed [13:25:26] oh gosh [13:25:44] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673119 (10Jelto) Thank you @ABran-WMF for merging the DNS change. I can confirm gerrit-replica host is accessible over the CDN now: `... [13:27:56] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009 (10hashar) 03NEW [13:28:07] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009#11673145 (10hashar) p:05Triage→03Unbreak! [13:28:08] zabe: https://phabricator.wikimedia.org/T419009 [13:28:16] and Zuul has Queue lengths: 1393 events, 15 results. [13:28:17] hashar: Also other CI jobs like https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/1247978 [13:28:20] which means it cant speak to Gerrit anymore [13:28:25] Ouch. [13:28:27] so the whole system is blocked somehow [13:29:15] Zuul has a queue for events it receives from Gerrit [13:29:30] and another one for reports it is trying to send to Gerrit (aka Verified -1+1) [13:29:36] which shows up on the Zuul status page [13:30:07] so something broken happened roughly one hour and a half ago [13:30:19] that did happen in the past (read years ago) but I don't quite remember what happene [13:30:28] most probably Zuul lost track of the connectiong with Gerrit [13:30:30] I ll check that [13:31:44] Restarting Zuul would lose all the queued jobs, but at least it'd be unblocked? [13:31:56] yeah but I dlike to understand what is happening [13:32:00] * James_F nods. [13:32:26] we changed the DNS of the gerrit-replica which should not have an impact on production CI, but maybe it had. Is zuul talking to the replica? [13:32:26] hashar: see #wikimedia-sre it may be related to a gerrit replica change of some sort [13:32:56] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009#11673159 (10mszwarc) It also seems that CI doesn't pick up new patches, e.g.: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1247957 [13:34:14] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673161 (10ABran-WMF) 05Resolved→03Open [13:36:56] ... [13:37:39] jelto: Yes, much of production CI runs clones off gerrit-replica, I think? [13:37:51] jelto: I don't think that is related, the replica is not used by Zuul and it doesnot connect to it. It would have been nice to let me knwo ahead of time about that operation though [13:37:53] anyway [13:37:53] But not sure if that'd affect Zuul itself, unless that changed. [13:38:00] I am going to try to gracefully stop Zuul [13:40:05] gerrit-replica is getting out of CDN (dns merge in progress) [13:40:27] done [13:41:05] !log Took a Zuul stack dump on contint1002.wikimedia.org using SIGUSR1 # T419009 [13:41:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:41:09] T419009: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009 [13:41:09] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009#11673206 (10hashar) ` 2026-03-04 12:25:10,813 ERROR zuul.GerritEventConnector: Exception moving Gerrit event: Traceback (most recent call last): File "/srv/deployment/zuul/venv... [13:42:10] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673209 (10ABran-WMF) [13:42:10] 06Release-Engineering-Team, 07affects-translatewiki.net, 07ci-test-error: https://integration.wikimedia.org/ci/job/translatewiki-ruby2.7-rake/32/console : FAILURE - https://phabricator.wikimedia.org/T418463#11673211 (10Jdforrester-WMF) The last re-build of the rake-ruby2.7 image (to 0.2.0-s1) was [[https://g... [13:42:20] do we have to coordinate another zuul restart when we switch the replica address back to non-cdn? [13:42:42] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009#11673217 (10ABran-WMF) it could be related to {T418108}, the dns change [[ https://gerrit.wikimedia.org/r/c/operations/dns/+/1247997 | has been reverted ]] [13:43:52] fun it is in queue only mode [13:43:56] but refuses to terminate [13:44:09] I guess because it is trying to connect to Gerrit to report the results it has [13:44:14] Joy. [13:44:19] Hard-reset instead? [13:44:23] na [13:44:43] jelto: Possibly the switch-over caused the indexing in real gerrit to lag, which is what broke CI? [13:45:24] I don't see why James_F, indexing and replication are not impacted by that move behind CDN [13:46:29] well hmm [13:47:05] it refuess to quite cause it can't report [13:47:17] I did disconnect Zuul from Gerrit (by killing the ssh connections) [13:47:25] but it is of course not bothering to reconnect [13:47:28] even for the reports [13:47:43] hashar: so it's pulling data from the primary with ssh? [13:48:19] James_F: I am claiming the queue of events lost, I don't think it has saved them even after a graceful stop triggered by a SIGUSR1 [13:48:55] arnaudb: yes Zuul listens for Gerrit events over ssh on port 29418 using `gerrit stream-events` [13:49:02] (03PS1) 10Jforrester: Docker: [rake-ruby2.7] Add ruby-ffi [integration/config] - 10https://gerrit.wikimedia.org/r/1248003 (https://phabricator.wikimedia.org/T418463) [13:49:03] hashar: Yeah, sadly. [13:49:21] 10Continuous-Integration-Infrastructure, 07Zuul, 06collaboration-services: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009#11673245 (10Jelto) [13:49:23] 06Release-Engineering-Team, 07affects-translatewiki.net, 07ci-test-error, 13Patch-For-Review: https://integration.wikimedia.org/ci/job/translatewiki-ruby2.7-rake/32/console : FAILURE - https://phabricator.wikimedia.org/T418463#11673246 (10Jdforrester-WMF) Looking at https://integration.wikimedia.org/ci/job... [13:49:30] !log Stopping Zuul # T419009 [13:49:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:49:33] T419009: Zuul status page is frozen / CI frozen? - https://phabricator.wikimedia.org/T419009 [13:50:14] not from gerrit-replica? weird timing then? [13:50:44] (we do not expose ssh through the CDN on -replica and -spare) [13:50:55] arnaudb: But does the primary gerrit not have to slow down / get higher load for a bit when the replica connection changes? [13:51:10] (Wild speculation.) [13:51:17] PROBLEM - zuul_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 1 process with regex args bin/zuul-server https://www.mediawiki.org/wiki/Continuous_integration/Zuul [13:51:49] I am still checking it [13:51:50] 10Continuous-Integration-Infrastructure, 07Zuul, 06collaboration-services: Zuul has lost connection to gerrit, freezing all CI in queue-only mode - https://phabricator.wikimedia.org/T419009#11673260 (10Jdforrester-WMF) [13:51:59] you're welcome to speculate :D no change on the replica connection though in that context, and nothing has changed from the replication point of view. Gerrit continues to work as usual, even if it slows down a bit [13:52:09] PROBLEM - zuul_gearman_service on contint1002 is CRITICAL: connect to address 127.0.0.1 and port 4730: Connection refused https://www.mediawiki.org/wiki/Continuous_integration/Zuul [13:52:12] fun thing is that it reconnected [13:52:16] and is still receiving events from Gerrit [13:52:32] But still in queue-only mode? [13:52:47] arnaudb: Ack. Surprising co-incidence then. [13:52:55] but it can't report [13:53:21] and when stopping it, that killed the gearman server :] [13:53:26] it is a zombie [13:53:31] so I am just going to SIGKILL it [13:53:33] :-( [13:53:40] Ah well. [13:54:09] RECOVERY - zuul_gearman_service on contint1002 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 4730 https://www.mediawiki.org/wiki/Continuous_integration/Zuul [13:54:17] RECOVERY - zuul_service_running on contint1002 is OK: PROCS OK: 2 processes with regex args bin/zuul-server https://www.mediawiki.org/wiki/Continuous_integration/Zuul [13:54:25] !log SIGKILL Zuul cause it can't gracefully stop most probably due to being locked attempting to report back to Gerrit # T419009 [13:54:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:54:32] (03CR) 10Jforrester: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1248003 (https://phabricator.wikimedia.org/T418463) (owner: 10Jforrester) [13:54:44] so yeah all events got lost [13:54:53] OK, recheck event succeeded and got processed immediately. [13:54:53] they are saved last minute AFTER the report queue got processed [13:55:03] but since the report queue was somehow not being processed well... deadlock of doom [13:55:06] as to why? [13:55:09] well hmm I don't know [13:55:27] I think that is 2026-03-04 12:25:10,81 ChangeNotFound: Change 1247774,1 not found [13:55:48] which is unexpected and must cause some loophole inside Zuul logic: it get lost / that is not accounted for or whatever [13:55:52] l10n-bot load too much? [13:56:08] maybe yeah, but isn't it running early in the morning? [13:56:21] Yes, but we have humans writing more and more patches every year. [13:56:37] So maybe we should get the various bots (l10n-bot, LibUp) to slow down another 50%? [13:57:22] arnaudb: jelto: I don't think the replica switch did much. It might have been one of multiple triggers that caused the issue, but I don't know what exactly has happened [13:57:49] (03CR) 10Jforrester: [C:03+2] Docker: [rake-ruby2.7] Add ruby-ffi [integration/config] - 10https://gerrit.wikimedia.org/r/1248003 (https://phabricator.wikimedia.org/T418463) (owner: 10Jforrester) [13:58:25] hashar: I reverted just in case, I'll revert back in a while with that in mind, we'll see if it triggers the bug again I guess [13:58:25] I don't even know whether that could be the indexing lagging out [13:58:33] or whether that Change not found could be an issue [13:58:44] arnaudb: I don't think you need to rollback the gerrit-replica move tbehind the cdn [13:58:57] Zuul doesn't know about it [13:59:14] I think it is simply a off by one error inside Zuul [13:59:18] it's already done hashar ^^ [13:59:18] (it has multiples) [13:59:21] or a cosmic ray [13:59:26] it was already rolled back and the replica is not behind the cdn anymore [13:59:29] (03Merged) 10jenkins-bot: Docker: [rake-ruby2.7] Add ruby-ffi [integration/config] - 10https://gerrit.wikimedia.org/r/1248003 (https://phabricator.wikimedia.org/T418463) (owner: 10Jforrester) [13:59:55] !log Docker: [rake-ruby2.7] Add ruby-ffi for T418463 [13:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:00:00] T418463: https://integration.wikimedia.org/ci/job/translatewiki-ruby2.7-rake/32/console : FAILURE - https://phabricator.wikimedia.org/T418463 [14:00:49] 06Release-Engineering-Team, 07affects-translatewiki.net, 07ci-test-error, 13Patch-For-Review: translatewiki-ruby2.7-rake job failing because install job now needs lib-ffi, not pre-installed on base image - https://phabricator.wikimedia.org/T418463#11673303 (10Jdforrester-WMF) [14:00:56] there is not even a bunch of indexing tasks going on https://grafana.wikimedia.org/d/Zh_ncGsWk/queues-upstream?orgId=1&from=now-6h&to=now&timezone=utc&var-instance=gerrit.wikimedia.org:443&var-replica=b&refresh=1m&viewPanel=panel-18 [14:00:57] :b [14:02:01] moving gerrit-replica back behind CDN: https://gerrit.wikimedia.org/r/c/operations/dns/+/1248002 [14:03:26] (cc hashar ↑) [14:04:47] 10Continuous-Integration-Infrastructure, 07Zuul, 06collaboration-services: Zuul has lost connection to gerrit, freezing all CI in queue-only mode - https://phabricator.wikimedia.org/T419009#11673330 (10hashar) p:05Unbreak!→03High https://grafana.wikimedia.org/d/Zh_ncGsWk/queues-upstream?orgId=1&from=now-... [14:04:50] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Request to unlock GitLab account for Mounil2005 for contributing to VideoCutTool (T331235) - https://phabricator.wikimedia.org/T418987#11673333 (10Mounil2005) >>! In T418987#11672605, @Aklapper wrote: > Approved. Thank you so much! [14:08:28] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673348 (10ABran-WMF) the change to put gerrit-replica behind CDN has been reverted to see if it triggered {T419009}, we confirmed with @ha... [14:12:15] (03PS1) 10Jforrester: jjb: Use rake-ruby2.7 image with ruby-ffi [integration/config] - 10https://gerrit.wikimedia.org/r/1248010 (https://phabricator.wikimedia.org/T418463) [14:18:18] hashar, anraudb so we try the gerrit-replica CDN switch again? currently the host is not behind the CDN [14:18:24] arnaudb * [14:20:32] arnaudb: [14:20:35] oh whoops [14:20:40] np ^^ [14:20:45] didn't mean to press enter [14:24:46] 10Continuous-Integration-Infrastructure, 07Zuul, 06collaboration-services: Zuul has lost connection to gerrit, freezing all CI in queue-only mode - https://phabricator.wikimedia.org/T419009#11673456 (10hashar) The last review sent by Zuul to Gerrit was: ` [2026-03-04T12:01:25.976Z] 0ded5238 [SSH gerrit revie... [14:27:03] gerrit-replica is moving again behind CDN [14:29:22] (done, triggered a CI job just after to see if it was impacting or not: https://gerrit.wikimedia.org/r/c/operations/dns/+/1248013) [14:30:35] 10Continuous-Integration-Infrastructure, 07Zuul, 06collaboration-services: Zuul has lost connection to gerrit, freezing all CI in queue-only mode - https://phabricator.wikimedia.org/T419009#11673485 (10hashar) 05Open→03Resolved a:03hashar There is one thread busy reporting back to Gerrit. ` name=Th... [14:32:41] (03CR) 10Hashar: [C:03+2] "Some random failure in GrowthExperiments" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1247554 (https://phabricator.wikimedia.org/T418461) (owner: 10Hashar) [14:33:22] jelto: arnaudb: the best I can tell is that Zuul was stuck trying to read some repo information from Gerrit https interface (over the cdn) [14:33:30] and I am not even sure that was the reason it got stuck [14:33:46] well it was probably not [14:33:50] anyway a restart fixed it [14:33:53] thanks for the confirmation hashar, fyi we moved gerrit-replica agian back behind CDN [14:34:03] I think it broke sometime between 12:00 and 12:20 UTC [14:34:23] we switched the DNS at 13:00 UTC [14:36:55] 10Gerrit, 06collaboration-services: Fix gerrit-restart cookbook - https://phabricator.wikimedia.org/T418264#11673511 (10ABran-WMF) 05Open→03Resolved Things are going as expected: ` DRY-RUN: Executing cookbook sre.gerrit.restart-gerrit with args: ['--host', 'gerrit2002'] DRY-RUN: Found [('conf1009.eqia... [14:37:54] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673527 (10ABran-WMF) [14:38:26] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: gerrit: move gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11673529 (10ABran-WMF) 05Open→03Resolved gerrit-replica is back again behind CDN, was not the trigger for {T419009} [14:51:10] (03Merged) 10jenkins-bot: Fix --shell when SHELL is not set [integration/quibble] - 10https://gerrit.wikimedia.org/r/1247554 (https://phabricator.wikimedia.org/T418461) (owner: 10Hashar) [14:55:14] 10Gerrit, 10Wikibugs: Wikibugs should ignore `check experimental` messages for operations/puppet - https://phabricator.wikimedia.org/T417866#11673607 (10MLechvien-WMF) Removing the SRE tag as this is a feature request for Wikibugs, feel free to add back if we can help [15:07:10] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: remove mod_qos on Gerrit - https://phabricator.wikimedia.org/T417615#11673688 (10ABran-WMF) 05Open→03In progress [15:31:17] (03PS2) 10Jforrester: jjb: Use rake-ruby2.7 image with ruby-ffi [integration/config] - 10https://gerrit.wikimedia.org/r/1248010 (https://phabricator.wikimedia.org/T418463) [15:31:17] (03PS1) 10Jforrester: Docker: [rake-ruby2.7] Add libffi-dev too [integration/config] - 10https://gerrit.wikimedia.org/r/1248030 (https://phabricator.wikimedia.org/T418463) [15:34:10] (03CR) 10Jforrester: [C:03+2] Docker: [rake-ruby2.7] Add libffi-dev too [integration/config] - 10https://gerrit.wikimedia.org/r/1248030 (https://phabricator.wikimedia.org/T418463) (owner: 10Jforrester) [15:35:54] (03Merged) 10jenkins-bot: Docker: [rake-ruby2.7] Add libffi-dev too [integration/config] - 10https://gerrit.wikimedia.org/r/1248030 (https://phabricator.wikimedia.org/T418463) (owner: 10Jforrester) [15:37:54] !log Docker: [rake-ruby2.7] Add libffi-dev too, for T418463 [15:37:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:37:57] T418463: translatewiki-ruby2.7-rake job failing because install job now needs lib-ffi, not pre-installed on base image - https://phabricator.wikimedia.org/T418463 [15:40:46] (03PS3) 10Jforrester: jjb: Use rake-ruby2.7 image with ruby-ffi and libffi-dev [integration/config] - 10https://gerrit.wikimedia.org/r/1248010 (https://phabricator.wikimedia.org/T418463) [15:41:35] (03CR) 10Jforrester: [C:03+2] jjb: Use rake-ruby2.7 image with ruby-ffi and libffi-dev [integration/config] - 10https://gerrit.wikimedia.org/r/1248010 (https://phabricator.wikimedia.org/T418463) (owner: 10Jforrester) [15:43:12] (03Merged) 10jenkins-bot: jjb: Use rake-ruby2.7 image with ruby-ffi and libffi-dev [integration/config] - 10https://gerrit.wikimedia.org/r/1248010 (https://phabricator.wikimedia.org/T418463) (owner: 10Jforrester) [16:14:02] (03open) 10sebastian-berlin-wmse: Add Symbolset to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/148 [16:16:37] (03PS1) 10Jforrester: Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting [integration/config] - 10https://gerrit.wikimedia.org/r/1248037 [16:16:37] (03PS1) 10Jforrester: Zuul: Drop PHP 8.2 enforcement from MediaWiki things [integration/config] - 10https://gerrit.wikimedia.org/r/1248038 [16:18:19] (03CR) 10CI reject: [V:04-1] Zuul: Drop PHP 8.2 enforcement from MediaWiki things [integration/config] - 10https://gerrit.wikimedia.org/r/1248038 (owner: 10Jforrester) [16:21:52] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting [integration/config] - 10https://gerrit.wikimedia.org/r/1248037 (owner: 10Jforrester) [16:23:25] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting [integration/config] - 10https://gerrit.wikimedia.org/r/1248037 (owner: 10Jforrester) [16:23:41] !log Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting [16:23:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:26:17] (03PS2) 10Jforrester: Zuul: Drop PHP 8.2 enforcement from MediaWiki things [integration/config] - 10https://gerrit.wikimedia.org/r/1248038 (https://phabricator.wikimedia.org/T358667) [16:26:48] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services: setup 2 contint machines for zuul or jenkins - https://phabricator.wikimedia.org/T418521#11674088 (10thcipriani) Talked about this with @hashar on Monday and had an IRC convo with @Dzahn adding to this task for pos... [16:35:38] 06Release-Engineering-Team, 10dev-images, 07PHP 8.5 support: create bookworm-php85-fpm and bookworm-php85-jobrunner images - https://phabricator.wikimedia.org/T413817#11674136 (10MSantos) p:05Triage→03High Migrating to PHP 8.5 has become a high priority for the MW Releases Working Group. Please. let us k... [16:40:32] 06Release-Engineering-Team, 07affects-translatewiki.net, 07ci-test-error: translatewiki-ruby2.7-rake job failing because install job now needs lib-ffi, not pre-installed on base image - https://phabricator.wikimedia.org/T418463#11674150 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [16:54:11] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor: Waiting for the completion of castor-save-workspace-cache sometimes takes almost 4 minutes for core tests - https://phabricator.wikimedia.org/T418974#11674177 (10hashar) The caches are namespaces, the build had ` CASTOR_NAMESPACE=mediawiki-core/ma... [16:55:39] 10Continuous-Integration-Infrastructure, 10Castor, 10Pywikibot: CASTOR sync for pywikibot-core-tox can take over 5 minutes in some cases - https://phabricator.wikimedia.org/T403266#11674184 (10hashar) Most probably, multiple builds of the castor-save-workspace job were in a queue and they were waiting for co... [17:17:43] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services: setup 2 contint machines for zuul or jenkins - https://phabricator.wikimedia.org/T418521#11674362 (10Dzahn) Yea, I think splitting domains and the suggested names sound good. [17:29:16] (03CR) 10C. Scott Ananian: "I don't think this is a good idea. The ParserMigration extension enables Parsoid Read Views on articles by default. This implicitly turn" [integration/config] - 10https://gerrit.wikimedia.org/r/1244746 (https://phabricator.wikimedia.org/T415451) (owner: 10Matthias Mullie) [17:32:26] (03CR) 10Hashar: "I was wondering how to implement that because the duration reporting is implemented as a context manager and it does not really have the r" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [17:33:14] (03CR) 10Hashar: [C:04-1] Write durations report to the log_dir as json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [17:33:37] (03CR) 10Jforrester: [C:03+2] "Argh. How are extensions with CI dependencies on PM meant to test it, then? Maybe the default shouldn't be destructive?" [integration/config] - 10https://gerrit.wikimedia.org/r/1244746 (https://phabricator.wikimedia.org/T415451) (owner: 10Matthias Mullie) [17:36:43] (03CR) 10Jforrester: [C:03+2] "I suggest reverting Id576d44d0920f140977dab8089a90b48b3a68397 and instead putting the special config for PatchDemo into PatchDemo." [integration/config] - 10https://gerrit.wikimedia.org/r/1244746 (https://phabricator.wikimedia.org/T415451) (owner: 10Matthias Mullie) [17:40:01] 10Continuous-Integration-Config, 07ci-test-error: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797#11674514 (10Zabe) [17:42:58] 10Continuous-Integration-Config, 07ci-test-error: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797#11674529 (10Zabe) p:05Triage→03High This breaks all backports. [17:49:38] (03PS1) 10Bvibber: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 [17:52:02] (03CR) 10CI reject: [V:04-1] Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 (owner: 10Bvibber) [17:54:23] (03PS2) 10Jforrester: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 (owner: 10Bvibber) [17:54:30] 10GitLab (CI & Job Runners), 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043 (10bvibber) 03NEW [17:55:57] (03PS3) 10Bvibber: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 [17:58:40] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11674649 (10Jdforrester-WMF) [18:02:45] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11674663 (10cscott) [18:03:47] (03PS1) 10Zabe: Make PHP 8.5 voting on core master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) [18:05:17] (03CR) 10CI reject: [V:04-1] Make PHP 8.5 voting on core master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [18:05:33] (03PS2) 10Zabe: Make PHP 8.5 voting on core master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) [18:06:24] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11674677 (10cscott) Also relevant: the tests currently failing in DiscussionTools are failing because there is an initial: `
(03PS3) 10Zabe: Make PHP 8.5 voting on core master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) [18:10:31] James_F: I have made some further change to your patch renaming the wikibase-* jobs https://gerrit.wikimedia.org/r/c/integration/config/+/1238317 [18:10:57] they get named after the extension that is nested inside the Wikibase repository [18:11:32] and after having actually looked at your change, I agree that it brings clarity :] [18:12:48] for the `wikibase-selenium` job which I have added last week, I gotta think about it. It is slightly different (there is a hardcoded set of extensions to gate Wikibase Selenium tests (but not phpunit ones) [18:13:36] and that is a different set of extensions than the quibble-with-WikibaseClient / quibble-with-WikibaseRepository jobs [18:13:55] naming is hard :\ [18:13:58] anyway, dinner time! [18:15:15] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11674706 (10cscott) More context: In the June-November time frame (before MW 1.47 is released) we expect to change the default parser to... [18:15:22] (03CR) 10Bvibber: "The default behavior should definitely not have changed from non-destructive to destructive with no warning..." [integration/config] - 10https://gerrit.wikimedia.org/r/1244746 (https://phabricator.wikimedia.org/T415451) (owner: 10Matthias Mullie) [18:20:17] (03PS4) 10Bvibber: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 (https://phabricator.wikimedia.org/T419043) [18:20:37] (03PS5) 10Bvibber: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 (https://phabricator.wikimedia.org/T419043) [18:21:17] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team, 13Patch-For-Review: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11674746 (10bvibber) [18:21:54] (03CR) 10C. Scott Ananian: "Discussion in T419043. I'd also take issue with 'destructive' -- Parsoid will be the default parser w/in the next ~8 months, so "in theor" [integration/config] - 10https://gerrit.wikimedia.org/r/1244746 (https://phabricator.wikimedia.org/T415451) (owner: 10Matthias Mullie) [18:22:52] (03CR) 10C. Scott Ananian: [C:03+1] Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 (https://phabricator.wikimedia.org/T419043) (owner: 10Bvibber) [18:29:26] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Blast of broken pipe errors from catalyst-api container log stream - https://phabricator.wikimedia.org/T417455#11674779 (10jnuche) [18:41:03] (03PS4) 10Jforrester: Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [18:44:13] 10Continuous-Integration-Config, 07ci-test-error: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797#11674877 (10hashar) Thanks for the ping Zabe, I should have commented earlier today. I need to introduce some new flavor of the jobs but at the same time there are some ot... [18:48:29] (03CR) 10Jforrester: [C:03+2] Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 (https://phabricator.wikimedia.org/T419043) (owner: 10Bvibber) [18:48:32] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team, 13Patch-For-Review: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11674908 (10Jdforrester-WMF) > [x] Ensure that MobileFrontend's tests pass whether ParserMigration is enabled or no... [18:50:31] (03Merged) 10jenkins-bot: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1248065 (https://phabricator.wikimedia.org/T419043) (owner: 10Bvibber) [18:50:46] !log Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency", for T419043 [18:50:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:50:49] T419043: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043 [19:03:40] 10Continuous-Integration-Config, 07ci-test-error: wikibase-selenium broken on release branches - https://phabricator.wikimedia.org/T418797#11674966 (10Umherirrender) The job runs also on - Vector - https://gerrit.wikimedia.org/r/q/status:open+-branch:master+project:mediawiki/skins/Vector - MinervaNeue - https... [19:18:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services: setup 2 contint machines for zuul or jenkins - https://phabricator.wikimedia.org/T418521#11674992 (10Dzahn) We will have to take this list, remove what is not needed, and dump what is left into a new role. I'm pre... [19:42:14] (03PS5) 10Zabe: Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) [19:42:15] (03PS1) 10Zabe: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) [19:43:55] (03update) 10brennen: Add dev images for php-8.4 development [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 (https://phabricator.wikimedia.org/T419004) (owner: 10arthurtaylor) [19:44:26] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [19:46:00] (03merge) 10brennen: Add dev images for php-8.4 development [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 (https://phabricator.wikimedia.org/T419004) (owner: 10arthurtaylor) [19:48:13] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 (T419004) [19:48:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:48:17] T419004: Create dev images for php8.4 - https://phabricator.wikimedia.org/T419004 [19:53:50] 06Release-Engineering-Team, 10dev-images, 10Wikidata, 06Wikidata-Omega, and 2 others: Create dev images for php8.4 - https://phabricator.wikimedia.org/T419004#11675118 (10brennen) Published: ` == Step 0: scanning /srv/dev-images/dockerfiles == Will build the following images: * docker-registry.discovery.w... [20:18:03] (03open) 10bwang: releases: Bump Codex to 2.4.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/122 (https://phabricator.wikimedia.org/T418304) [20:20:04] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11675195 (10bvibber) [20:20:49] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11675196 (10bvibber) >>! In T419043#11674908, @Jdforrester-WMF wrote: >> [x] Ensure that MobileFrontend's tests pass whether ParserMigrat... [20:24:58] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.28.0 released - https://phabricator.wikimedia.org/T415284#11675214 (10dancy) [20:34:57] (03merge) 10dancy: Add Symbolset to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/148 (owner: 10sebastian-berlin-wmse) [21:00:37] (03PS1) 10Hashar: Propagate SuccessCache.Hit() to the caller [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248096 (https://phabricator.wikimedia.org/T417399) [21:06:20] (03CR) 10Jforrester: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [21:06:58] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [21:07:51] (03PS2) 10Zabe: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) [21:08:22] (03CR) 10Zabe: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [21:08:40] (03Merged) 10jenkins-bot: Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248069 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [21:09:22] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [21:09:38] (03PS1) 10Jforrester: Zuul: [mediawiki/vendor] Drop experimental PHP 8.4/8.5 jobs, already voting [integration/config] - 10https://gerrit.wikimedia.org/r/1248098 [21:10:03] !log Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch, for T411814 [21:10:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:10:07] T411814: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814 [21:10:47] (03PS3) 10Zabe: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) [21:11:16] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/vendor] Drop experimental PHP 8.4/8.5 jobs, already voting [integration/config] - 10https://gerrit.wikimedia.org/r/1248098 (owner: 10Jforrester) [21:12:51] (03Merged) 10jenkins-bot: Zuul: [mediawiki/vendor] Drop experimental PHP 8.4/8.5 jobs, already voting [integration/config] - 10https://gerrit.wikimedia.org/r/1248098 (owner: 10Jforrester) [21:13:58] 10Continuous-Integration-Config, 07Epic, 13Patch-For-Review, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11675408 (10Jdforrester-WMF) [21:14:26] (03PS4) 10Zabe: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) [21:14:30] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [21:16:04] (03Merged) 10jenkins-bot: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/1248088 (https://phabricator.wikimedia.org/T411814) (owner: 10Zabe) [21:16:52] !log Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch, for T411814 [21:16:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:16:54] T411814: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814 [21:28:19] 10Continuous-Integration-Config, 10MobileFrontend, 06Reader Growth Team: Remove ParserMigration from CI deps for MobileFrontend - https://phabricator.wikimedia.org/T419043#11675530 (10bvibber) [21:37:21] 10Continuous-Integration-Config, 07Epic, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11675576 (10Jdforrester-WMF) [21:37:32] (03PS7) 10Hashar: Write durations report to the log_dir as json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) [21:37:32] (03CR) 10Hashar: "Done covering a build that fail, one that succeed and one that has hit the success cache." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [21:39:01] (03CR) 10CI reject: [V:04-1] Write durations report to the log_dir as json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [21:40:40] (03CR) 10Hashar: "+ @dduvall@wikimedia.org who wrote the SuccessCache" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1248096 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [21:41:44] (03PS8) 10Hashar: Write durations report to the log_dir as json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1243189 (https://phabricator.wikimedia.org/T417399) [21:46:21] (03PS1) 10Daimona Eaytoy: dockerfiles: [quibble-coverage] Use local PHPUnit config [integration/config] - 10https://gerrit.wikimedia.org/r/1248112 (https://phabricator.wikimedia.org/T345481) [21:47:52] (03CR) 10CI reject: [V:04-1] dockerfiles: [quibble-coverage] Use local PHPUnit config [integration/config] - 10https://gerrit.wikimedia.org/r/1248112 (https://phabricator.wikimedia.org/T345481) (owner: 10Daimona Eaytoy) [21:49:16] (03PS2) 10Daimona Eaytoy: dockerfiles: [quibble-coverage] Use local PHPUnit config [integration/config] - 10https://gerrit.wikimedia.org/r/1248112 (https://phabricator.wikimedia.org/T345481) [21:54:54] (03PS1) 10Daimona Eaytoy: jjb: Generate PHPUnit config before running tests [integration/config] - 10https://gerrit.wikimedia.org/r/1248113 (https://phabricator.wikimedia.org/T345481) [22:06:56] (03PS1) 10Jforrester: Zuul: Add a specific task for every repo not yet running PHP 8.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1248116 [22:07:00] Krinkle: FWICT almost every single non-MW repo left not yet running CI on PHP 8.4 or 8.5 is one of yours in the labs/tools/ hierarchy. :-( Sorry for all the noise from the filed Phabricator tasks. [22:20:13] (03CR) 10Jforrester: [C:03+2] Zuul: Add a specific task for every repo not yet running PHP 8.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1248116 (owner: 10Jforrester) [22:22:37] (03Merged) 10jenkins-bot: Zuul: Add a specific task for every repo not yet running PHP 8.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1248116 (owner: 10Jforrester) [22:24:35] 10Scap (SpiderPig 🕸️): Ask users if they're sure they want to interact with a SpiderPig session's button when they didn't initiate it - https://phabricator.wikimedia.org/T419084 (10Jdforrester-WMF) 03NEW [22:25:05] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 07ci-test-error (WMF-deployed Build Failure): mwext-phpunit-coverage-patch no longer runs any tests for modified files - https://phabricator.wikimedia.org/T419073#11675796 (10Jdforrester-WMF) [22:34:31] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 07ci-test-error (WMF-deployed Build Failure): mwext-phpunit-coverage-patch no longer runs any tests for modified files - https://phabricator.wikimedia.org/T419073#11675858 (10Daimona) Yes, most probably caused by changes in T345481. I have a few patc... [22:41:58] (03merge) 10annet: releases: Bump Codex to 2.4.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/122 (https://phabricator.wikimedia.org/T418304) (owner: 10bwang) [22:47:59] (03update) 10jforrester: Draft: TestKitchen renaming (MetricsPlatform => TestKitchen) [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/237 (https://phabricator.wikimedia.org/T416865) (owner: 10sfaci) [22:58:08] 10Scap (SpiderPig 🕸️): Ask users if they're sure they want to interact with a SpiderPig session's button when they didn't initiate it - https://phabricator.wikimedia.org/T419084#11675953 (10Tgr) Another option would be to indicate to the original user somehow (push notice?) what happened. And/or maybe in the IRC... [22:58:54] 10Continuous-Integration-Config, 07Epic, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11675954 (10Jdforrester-WMF) [23:16:26] (03PS1) 10Jforrester: Zuul: Update extension-quibble comment about tested PHP versions [integration/config] - 10https://gerrit.wikimedia.org/r/1248130 [23:17:20] (03CR) 10Jforrester: [C:03+2] Zuul: Update extension-quibble comment about tested PHP versions [integration/config] - 10https://gerrit.wikimedia.org/r/1248130 (owner: 10Jforrester) [23:18:52] (03Merged) 10jenkins-bot: Zuul: Update extension-quibble comment about tested PHP versions [integration/config] - 10https://gerrit.wikimedia.org/r/1248130 (owner: 10Jforrester) [23:54:08] (03CR) 10Jforrester: "This should also fix T419073 I think?" [integration/config] - 10https://gerrit.wikimedia.org/r/1248112 (https://phabricator.wikimedia.org/T345481) (owner: 10Daimona Eaytoy) [23:54:47] (03PS3) 10Daimona Eaytoy: dockerfiles: [quibble-coverage] Use local PHPUnit config [integration/config] - 10https://gerrit.wikimedia.org/r/1248112 (https://phabricator.wikimedia.org/T345481) [23:55:04] (03CR) 10Jforrester: [C:03+2] dockerfiles: [quibble-coverage] Use local PHPUnit config [integration/config] - 10https://gerrit.wikimedia.org/r/1248112 (https://phabricator.wikimedia.org/T345481) (owner: 10Daimona Eaytoy) [23:55:05] (03CR) 10Daimona Eaytoy: "Yup, it was created after this patch and I forgot to update here." [integration/config] - 10https://gerrit.wikimedia.org/r/1248112 (https://phabricator.wikimedia.org/T345481) (owner: 10Daimona Eaytoy) [23:55:42] (03PS2) 10Daimona Eaytoy: jjb: Generate PHPUnit config before running tests [integration/config] - 10https://gerrit.wikimedia.org/r/1248113 (https://phabricator.wikimedia.org/T345481)