[00:10:06] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: Replace job image variants with webservice image variants - https://phabricator.wikimedia.org/T415322#11547271 (10Raymond_Ndibe) [00:10:28] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: Replace job image variants with webservice image variants - https://phabricator.wikimedia.org/T415322#11547274 (10Raymond_Ndibe) [00:13:45] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: Replace job image variants with webservice image variants - https://phabricator.wikimedia.org/T415322#11547276 (10Raymond_Ndibe) [00:14:17] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: resolve “jobs-framework” and “web” image variants to the same image - https://phabricator.wikimedia.org/T414978#11547277 (10Raymond_Ndibe) 05In progress→03Declined [00:14:40] (03close) 10raymond-ndibe: images::from_url_or_name: match variants of the same image [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/260 (https://phabricator.wikimedia.org/T414978) [00:15:12] (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322) [00:16:40] (03update) 10raymond-ndibe: values.yaml: replace job image variants with webservice image variants [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/18 (https://phabricator.wikimedia.org/T415322) [00:19:36] (03update) 10aghirelli: Draft: feat(linter): add new rules [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/10 [00:21:39] (03update) 10aghirelli: feat(linter): add new rules [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/10 [00:22:49] (03close) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [00:25:14] (03update) 10aghirelli: feat(T410402): request validation [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/10 [00:37:47] (03update) 10bd808: index.php: Show wiki % to 2 decimal places [toolforge-repos/versions] - 10https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8 (https://phabricator.wikimedia.org/T415318) (owner: 10reedy) [00:38:09] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [00:42:26] (03update) 10reedy: index.php: Show wiki % to 2 decimal places [toolforge-repos/versions] - 10https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8 (https://phabricator.wikimedia.org/T415318) [00:42:28] (03update) 10reedy: index.php: Show wiki % to 2 decimal places [toolforge-repos/versions] - 10https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8 (https://phabricator.wikimedia.org/T415318) [00:44:39] (03update) 10aghirelli: feat(T410402): request validation [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/10 [00:45:14] (03approved) 10raymond-ndibe: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [00:45:21] (03merge) 10raymond-ndibe: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [00:48:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.182-20260123004531-c1f9ba44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1114 [00:56:47] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [00:59:39] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:04:44] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [01:05:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:07:53] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [01:09:59] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [01:11:09] (03update) 10raymond-ndibe: components-api: bump to 0.0.182-20260123004531-c1f9ba44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1114 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:11:11] (03approved) 10raymond-ndibe: components-api: bump to 0.0.182-20260123004531-c1f9ba44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1114 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:11:19] (03merge) 10raymond-ndibe: components-api: bump to 0.0.182-20260123004531-c1f9ba44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1114 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:20:56] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [01:26:54] (03merge) 10vriaa: feat: Make editor responsive [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/21 [01:28:31] (03merge) 10vriaa: fix(templates): ensure properties conform to expected schema to prevent runtime/type errors [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/24 [01:32:10] (03merge) 10vriaa: fix: Capitalize CloseIcon file name correctly [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/23 [01:36:33] 06cloud-services-team, 10Toolforge: Toolforge SSH login: connection closed after publickey authentication - https://phabricator.wikimedia.org/T415239#11547550 (10JacobHung) Thanks for checking. My Wikimedia username is "JacobHung" (capital J, as shown on-wiki). When connecting via SSH I use "jacobhung", since... [01:40:08] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [01:40:18] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [01:42:31] 06cloud-services-team, 10Toolforge: Toolforge SSH login: connection closed after publickey authentication - https://phabricator.wikimedia.org/T415239#11547563 (10JacobHung) Thanks for the clarification. It looks like my Wikimedia account "JacobHung" is already linked to an existing Developer/Toolforge account... [01:52:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:34:31] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [02:47:43] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978) [02:49:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:52:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [03:12:28] (03update) 10vriaa: feat: Persist banners in local storage [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/25 [03:13:48] (03merge) 10vriaa: feat: Persist banners in local storage [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/25 [03:14:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:16:38] (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322) [03:52:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:27:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:22:00] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:54:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:14:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [07:14:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:24:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [09:22:15] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:03:13] 10Cloud Services Proposals, 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11548007 (10Gehel) [10:03:20] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11548009 (10Gehel) [10:07:07] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11548110 (10Gehel) [10:07:17] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11548116 (10Gehel) [10:09:50] 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, and 2 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11548158 (10Gehel) [10:23:13] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: Replace job image variants with webservice image variants - https://phabricator.wikimedia.org/T415322#11548292 (10fnegri) p:05Triage→03High [10:46:07] 06cloud-services-team, 10Toolforge: Toolforge SSH login: connection closed after publickey authentication - https://phabricator.wikimedia.org/T415239#11548360 (10fnegri) > Since I cannot submit a new membership request due to the existing linkage, The linkage should not prevent you from submitting a membershi... [12:23:19] 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11548574 (10A_smart_kitten) [12:26:03] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11548582 (10Nurah_Wakili) **Weekly Internship Report** **Week 7: January 19 – January 23** **Work Completed:** * Incorporated mentor feedb... [12:34:35] (03open) 10volans: rotate_clinic_duty: remove unnecessary IRC op/deop [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/1 [12:37:59] (03approved) 10taavi: rotate_clinic_duty: remove unnecessary IRC op/deop [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/1 (owner: 10volans) [13:18:21] (03merge) 10volans: rotate_clinic_duty: remove unnecessary IRC op/deop [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/1 [13:22:15] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:40:59] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360 (10Psymetric) 03NEW [14:27:30] (03open) 10kbach: Draft: Introduce support for passing OAD URL in a parameter [toolforge-repos/api-spec-reader] - 10https://gitlab.wikimedia.org/toolforge-repos/api-spec-reader/-/merge_requests/2 [15:36:01] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549246 (10Aklapper) > a MediaWiki-based research wiki hosted on Miraheze Where to find/see that wiki? What's its URI? How is your wiki related to Wikimedia and it... [16:02:48] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549359 (10bd808) It is generally possible to host a MediaWiki instance as a Toolforge tool. Some examples: * https://mwdemo.toolforge.org/wiki/Main_Page * https://t... [16:16:02] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549410 (10Psymetric) The wiki has not been created yet on Miraheze and is awaiting approval (https://meta.miraheze.org/wiki/Special:RequestWikiQueue/73470). We aim... [16:17:36] 06cloud-services-team, 10Toolforge: Toolforge SSH login: connection closed after publickey authentication - https://phabricator.wikimedia.org/T415239#11549415 (10bd808) There is a user record for the https://ldap.toolforge.org/user/jacobhung Developer account in the https://toolsadmin.wikimedia.org/ database.... [16:19:54] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin, 10Toolforge: updatetools is frequently timing out - https://phabricator.wikimedia.org/T413431#11549423 (10fnegri) [16:31:55] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549441 (10bd808) >>! In T415360#11549410, @Psymetric wrote: > The wiki has not been created yet on Miraheze and is awaiting approval (https://meta.miraheze.org/wiki... [16:44:46] (03update) 10bd808: index.php: Show wiki % to 2 decimal places [toolforge-repos/versions] - 10https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8 (https://phabricator.wikimedia.org/T415318) (owner: 10reedy) [16:45:23] (03merge) 10bd808: index.php: Show wiki % to 2 decimal places [toolforge-repos/versions] - 10https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8 (https://phabricator.wikimedia.org/T415318) (owner: 10reedy) [16:49:41] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549480 (10Psymetric) The connection is technical rather than editorial. The project uses Wikimedia Cloud VPS to develop, test, and document MediaWiki deployment pa... [16:49:58] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin: Import tool-admin jobs to Git repo - https://phabricator.wikimedia.org/T415383 (10fnegri) 03NEW [16:51:11] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin, 10Toolforge: updatetools is frequently timing out - https://phabricator.wikimedia.org/T413431#11549509 (10fnegri) > We should also version the script updatetools.py in a Git repo, if it's not there already (I haven't found it in Codesearch). I split t... [16:52:41] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin: Import tool-admin jobs to Git repo - https://phabricator.wikimedia.org/T415383#11549519 (10fnegri) p:05Triage→03Medium [17:01:54] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549566 (10Aklapper) Sharing the results of your operational testing, your evaluation of use cases, and your validation of MediaWiki patterns from setting up and hos... [17:02:42] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549570 (10Aklapper) a:05Psymetric→03None [17:09:25] (03open) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 (https://phabricator.wikimedia.org/T414978 https://phabricator.wikimedia.org/T415322) [17:09:34] (03update) 10raymond-ndibe: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 (https://phabricator.wikimedia.org/T414978 https://phabricator.wikimedia.org/T415322) [17:10:15] (03update) 10raymond-ndibe: jobs-api: use webservice image variants in one-off job tests [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 (https://phabricator.wikimedia.org/T414978 https://phabricator.wikimedia.org/T415322) [17:10:51] (03update) 10raymond-ndibe: jobs-api: use webservice image variants in one-off job tests [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 (https://phabricator.wikimedia.org/T415322) [17:11:39] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549593 (10Psymetric) Alright, no worries, we'll check out a possible Toolforge implementation. [17:13:07] (03update) 10raymond-ndibe: jobs-api: use webservice image variants in one-off job tests [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 (https://phabricator.wikimedia.org/T415322) [17:20:26] (03update) 10raymond-ndibe: jobs-api: use webservice image variants in one-off job tests [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 (https://phabricator.wikimedia.org/T415322) [17:22:15] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:26:46] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin: Import tool-admin jobs to Git repo - https://phabricator.wikimedia.org/T415383#11549619 (10fnegri) I'm also wondering if we could just delete two out of three cron jobs: - `updatetools.py` is updating two tables in the `toollabs_p` database. According t... [17:28:52] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549623 (10bd808) >>! In T415360#11549593, @Psymetric wrote: > Alright, no worries, we'll check out a possible Toolforge implementation. Toolforge hosting would be... [17:32:16] (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322) [17:33:55] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: OpenAPI linter - performance and dev-ex improvements - https://phabricator.wikimedia.org/T415388 (10AGhirelli-WMF) 03NEW [17:34:42] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin: Import tool-admin jobs to Git repo - https://phabricator.wikimedia.org/T415383#11549657 (10fnegri) I would probably still import everything to git before deleting them, so that we have a copy of the current version. [17:39:20] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: OpenAPI linter - performance and dev-ex improvements - https://phabricator.wikimedia.org/T415388#11549674 (10AGhirelli-WMF) cc. @Mooeypoo [17:51:26] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [18:00:21] 10Tool-translatetagger: Fix "Translatable Wikitext" output formatting to prevent horizontal scrolling - https://phabricator.wikimedia.org/T415390 (10Gopavasanth) 03NEW [18:03:31] 10Tool-translatetagger: Fix "Translatable Wikitext" output formatting to prevent horizontal scrolling - https://phabricator.wikimedia.org/T415390#11549738 (10Gopavasanth) 05Open→03Resolved Commit: https://github.com/indictechcom/TranslateTagger/commit/e8800041f1a7d360495c0ee0d77983c977329dd1 [18:12:01] 10Cloud-VPS (Project-requests): Request creation of Psychopragmatics VPS project - https://phabricator.wikimedia.org/T415360#11549782 (10Aklapper) 05Open→03Declined [18:34:39] (03open) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [18:34:42] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [18:36:49] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [18:38:18] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [18:46:08] 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11549816 (10MLechvien-WMF) I modified the alternate tag of `serviceops new` and I believe this should now be reflected by IRC bot/wikibugs to the -operations channel [18:46:48] 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11549817 (10MLechvien-WMF) 05Open→03Resolved [18:54:18] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [18:54:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:59:39] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [19:14:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:27:22] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [19:28:25] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [19:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:53:41] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [20:06:19] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [20:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:15:54] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [20:19:38] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [20:20:47] (03approved) 10apaskulin: Draft: Introduce support for passing OAD URL in a parameter [toolforge-repos/api-spec-reader] - 10https://gitlab.wikimedia.org/toolforge-repos/api-spec-reader/-/merge_requests/2 (owner: 10kbach) [20:30:36] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [21:33:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [21:48:20] 10Tools, 06Release-Engineering-Team: versions % doesn't add up to 100% - https://phabricator.wikimedia.org/T415318#11550233 (10bd808) 05Open→03Resolved a:03Reedy The display values add up to 99.99% now. We could get incrementally closer to 100% by showing more digits, but hopefully this is good enough. [21:49:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:56:39] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge NFS tracing misses some dumps events - https://phabricator.wikimedia.org/T415199#11550264 (10Volans) After quite few testing the route of LSM_PROBE wasn't feasible because of some missing feature in the kernel that AFAICT can't just be enabled... [22:04:23] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408 (10Nokib_Sarkar) 03NEW [22:04:54] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11550296 (10Nokib_Sarkar) [22:05:00] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11550297 (10Nokib_Sarkar) [22:06:47] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11550300 (10Nokib_Sarkar) [22:07:34] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11550303 (10Nokib_Sarkar) @Sanjaydevs I added some microtasks. [22:10:42] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11550304 (10Nokib_Sarkar) [22:14:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:58:16] 10Wikibugs, 06ServiceOps new: Add serviceops new to the monitored channels on IRC bot/wikibugs - https://phabricator.wikimedia.org/T415352#11550573 (10bd808) The semi-secret magic here was that the `#wikimedia-operations` channel config matches all Phabricator tags and alternate tags that start with `sre-`... [23:13:00] RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [23:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:24:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:44:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown