[00:22:41] 10Tool-paulina, 13Patch-For-Review: search authors for a specific death date year - https://phabricator.wikimedia.org/T409205#11342907 (10Oluwatumininu.m) Hi @Mind_Booster_Noori @Pepe_piton @Nat_WDU I fixed this issue. Kindly have a look to check if I'm on the right track. Here is the link to merge request:... [00:59:04] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244 (10Bamyers99) 03NEW [01:19:29] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343053 (10Zache) It seems that toolsdb is generally down. Here is ticket for montage db connection [https://github.com/hat... [05:05:56] FIRING: SystemdUnitDown: The service unit hdfs_rsync_cirrus_index_dumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:10:56] FIRING: [2x] SystemdUnitDown: The service unit hdfs_rsync_cirrus_index_dumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:33:09] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343298 (10Chlod) Also affects: [fountain](https://fountain.toolforge.org/), [PetScan](https://petscan.wmcloud.org/) [06:56:36] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343376 (10Zache) p:05Triage→03High [06:57:15] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343377 (10Zache) I triaged this high as this is still going on and impacts to multiple tools. [07:00:56] FIRING: SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:05:56] FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:19:23] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343456 (10fgiunchedi) tools-db-4 storage volume is out of space, I'll use this task for tracking [07:26:45] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343478 (10fgiunchedi) disk space free trend for tools-db-4 over the last 30d {F69903693} and over the last couple of day... [07:41:37] 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343501 (10fgiunchedi) Doing a comparison with the replica on tools-db-6, there's ~800G free there: ` root@tools-db-6:/srv... [08:07:31] FIRING: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [08:08:28] FIRING: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-4 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:08:31] FIRING: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-6 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [08:10:56] FIRING: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:24:01] 06cloud-services-team, 10Toolforge: Toolforge outage: toolsdb out of space - https://phabricator.wikimedia.org/T409244#11343581 (10Chlod) [08:24:31] FIRING: ToolsToolsDBReplicationMissing: ToolsDB replication is not running on tools-db-6 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [08:45:56] RESOLVED: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:55:01] RESOLVED: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-6 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [08:57:58] RESOLVED: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-4 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:00:17] (03PS1) 10David Caro: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 [09:01:01] RESOLVED: ToolsToolsDBReplicationMissing: ToolsDB replication is not running on tools-db-6 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [09:01:55] (03PS2) 10David Caro: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 [09:02:58] (03CR) 10David Caro: "Tested in cloudcumin:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:06:04] (03CR) 10CI reject: [V:04-1] toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:10:01] RESOLVED: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [09:13:45] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM modulo CI failures" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:15:56] (03CR) 10Volans: toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:17:16] (03PS3) 10David Caro: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 [09:22:00] (03CR) 10CI reject: [V:04-1] toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:24:42] 10Tool-techcontribs: pie charts with overlapping/unreadable numbers - https://phabricator.wikimedia.org/T384557#11343745 (10Chlod) I've long wanted to use a pie chart but with lines to denote which slice is which, but the library used for the charts do not support this. A different chart is probably the way to g... [09:24:50] 10Tool-techcontribs: pie charts with overlapping/unreadable numbers - https://phabricator.wikimedia.org/T384557#11343746 (10Chlod) p:05Triage→03Low [09:26:01] 10Tool-techcontribs: Add reporting on Quips mentions - https://phabricator.wikimedia.org/T393627#11343748 (10Chlod) p:05Triage→03Medium [09:26:31] 10Tool-techcontribs: in GitHub section, include organizations that the user is a member of - https://phabricator.wikimedia.org/T393623#11343749 (10Chlod) p:05Triage→03Medium [09:26:34] (03CR) 10David Caro: toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:28:20] (03CR) 10Majavah: [C:04-1] toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:28:53] (03PS4) 10David Caro: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 [09:38:18] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322#11343797 (10Chlod) 05Open→03In progress a:03Chlod [09:42:13] (03CR) 10David Caro: toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:46:38] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322#11343860 (10Novem_Linguae) "Admin groups" might not be the most common name for this. I think something like POSIX groups or shell groups might be more common. Wikitech calls it "[[ https://wikitech.wikimedia.org/... [09:47:37] (03CR) 10Volans: "reply inline" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [09:53:13] 06cloud-services-team, 10Toolforge: Toolforge outage: toolsdb out of space - https://phabricator.wikimedia.org/T409244#11343868 (10fnegri) We added some disk space to tools-db-4 and toolsdb is working again. We are still investigating what caused the big increase in size for `ibdata1`. [09:57:47] (03PS5) 10David Caro: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 [10:00:10] 06cloud-services-team, 10Toolforge: Toolforge outage: toolsdb out of space - https://phabricator.wikimedia.org/T409244#11343880 (10Zache) Thank you very much ! [10:01:35] (03CR) 10CI reject: [V:04-1] toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [10:25:17] 10Tool-iw, 13Patch-For-Review: IW ingress config does not work with nginx-ingress-controller:v1.13.3 - https://phabricator.wikimedia.org/T408570#11343925 (10taavi) 05In progress→03Resolved [10:27:58] (03CR) 10David Caro: toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [10:28:38] 06cloud-services-team (FY2025/26-Q1), 10Data-Services: [wikireplicas] Create views for new wiki tokwiki - https://phabricator.wikimedia.org/T404570#11343931 (10fnegri) 05Open→03Stalled This is stalled on {T404566}, that was resolved too soon [10:30:12] 06cloud-services-team (FY2025/26-Q1), 10Data-Services: [wikireplicas] Create views for new wiki thwikimedia - https://phabricator.wikimedia.org/T409201#11343936 (10fnegri) 05In progress→03Resolved Verified working: `lang=shell-session fnegri@tools-bastion-15:~$ sql thwikimedia [...] MariaDB [thwikim... [10:32:30] (03PS6) 10David Caro: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 [10:43:39] (03CR) 10David Caro: toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [10:45:31] (03PS7) 10David Caro: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 [10:46:06] (03CR) 10David Caro: "Man I miss pre-commit... it's kinda painful to run the tests for this repo" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [10:46:30] (03CR) 10David Caro: toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [10:46:36] (03CR) 10David Caro: toolsdb: add readonly property (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [10:54:40] (03CR) 10FNegri: [C:03+1] toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [11:01:03] (03CR) 10David Caro: [C:03+2] toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [11:04:56] (03Merged) 10jenkins-bot: toolsdb: add readonly property [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1202054 (owner: 10David Caro) [11:05:56] FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:42:27] 06cloud-services-team, 10Toolforge: Toolforge outage: toolsdb out of space - https://phabricator.wikimedia.org/T409244#11344198 (10DamianZaremba) This also impacted ClueBot NG, alert for report interface being down was sent at 23:55 UTC (alerting rule is after 2min), then bot not editing at 00:39 UTC (alerting... [11:59:50] 06cloud-services-team, 10Toolforge: [functional tests] Consider adding tool outputs on failure - https://phabricator.wikimedia.org/T409280 (10DamianZaremba) 03NEW [12:39:31] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217950 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a) [12:39:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [12:39:58] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217917 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a) [12:39:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [13:21:36] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11344477 (10fnegri) 05Open→03In progress p:05Triage→03Medium a:03fnegri [13:25:51] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11344495 (10fnegri) [13:26:45] 06Toolforge-standards-committee: Define a process for keeping the committee membership "fresh" - https://phabricator.wikimedia.org/T379844#11344501 (10JJMC89) 05In progress→03Resolved The Wikitech TFSC page is updated per the committee vote. [13:42:21] 06Toolforge-standards-committee: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290 (10JJMC89) 03NEW p:05Triage→03Medium [13:48:54] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11344597 (10fnegri) [13:48:57] 06cloud-services-team, 10Toolforge: Toolforge outage: toolsdb out of space - https://phabricator.wikimedia.org/T409244#11344598 (10fnegri) [13:49:44] 06cloud-services-team, 10Toolforge: Toolforge outage: toolsdb out of space - https://phabricator.wikimedia.org/T409244#11344600 (10fnegri) 05Open→03Resolved a:03fnegri I'm marking this as Resolved, follow-up is {T409287}. [13:57:44] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T409287) [13:57:49] T409287: [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287 [14:02:33] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' (T409287) [14:07:00] 06cloud-services-team, 10Cloud-VPS: Fix MTU on single-NIC Ceph nodes - https://phabricator.wikimedia.org/T409294 (10taavi) 03NEW [14:09:28] FIRING: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-7 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:33:44] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075#11344864 (10taavi) [14:34:46] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11344874 (10fgiunchedi) >>! In T408543#11328875, @cmooney wrote: > # Move cloudvirt/cloudnet to support jumbo frames ensuring VMs can have a 1500 byte MTU Mentioning cloudvirt + ju... [14:48:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-db-7 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:53:34] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud (T409287) [14:53:39] T409287: [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287 [14:55:01] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud (T409287) [15:03:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-db-7 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:06:11] (03approved) 10dcaro: harbor images - return digest in container image [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/232 (https://phabricator.wikimedia.org/T408574) (owner: 10damian) [15:06:11] FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:06:38] (03merge) 10dcaro: harbor images - return digest in container image [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/232 (https://phabricator.wikimedia.org/T408574) (owner: 10damian) [15:07:12] (03update) 10dcaro: Cleanup test_filelog_without_mount_fails [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/241 (https://phabricator.wikimedia.org/T408583) (owner: 10damian) [15:07:13] (03update) 10dcaro: Cleanup test_filelog_without_mount_fails [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/241 (https://phabricator.wikimedia.org/T408583) (owner: 10damian) [15:07:51] (03approved) 10dcaro: Cleanup test_filelog_without_mount_fails [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/241 (https://phabricator.wikimedia.org/T408583) (owner: 10damian) [15:09:57] (03merge) 10dcaro: Cleanup test_filelog_without_mount_fails [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/241 (https://phabricator.wikimedia.org/T408583) (owner: 10damian) [15:10:29] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.437-20251105150651-f3396660 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1058 (https://phabricator.wikimedia.org/T408574) [15:12:45] 06cloud-services-team, 10Cloud-VPS: Fix MTU on single-NIC Ceph nodes - https://phabricator.wikimedia.org/T409294#11345112 (10fgiunchedi) [15:12:48] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.438-20251105151007-b30f0871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1058 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T408583) [15:12:50] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.438-20251105151007-b30f0871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1058 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T408583) [15:17:22] 06cloud-services-team, 10Toolforge, 06Infrastructure-Foundations, 10netops: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309 (10taavi) 03NEW [15:19:00] (03unapproved) 10dcaro: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) (owner: 10damian) [15:19:37] 06cloud-services-team, 10Toolforge, 06Infrastructure-Foundations, 10netops: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#11345168 (10taavi) [15:19:38] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11345169 (10taavi) [15:28:12] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11345217 (10cmooney) Just to note down what we discussed on our meeting about this: # We will need option 2 ** To have connectivit... [15:29:52] 06cloud-services-team, 10Toolforge, 06Infrastructure-Foundations, 10netops: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#11345226 (10cmooney) Thanks for the task @taavi. I'll need to discuss this with @ayounsi on his return, but in principal it makes sense... [15:33:20] (03update) 10dcaro: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [15:59:43] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set http IO to none when creating Diffusion repositories - https://phabricator.wikimedia.org/T407705#11345349 (10Aklapper) This has been merged but does not seem to be deployed yet, looking... [15:59:57] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624#11345353 (10Aklapper) This has been merged but does not seem to be deployed yet... [16:04:48] 06Toolforge-standards-committee: Define a process for keeping the committee membership "fresh" - https://phabricator.wikimedia.org/T379844#11345375 (10bd808) [16:04:51] 06Toolforge-standards-committee: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290#11345376 (10bd808) [16:09:43] 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#11345384 (10Andrew) >>! In T376400#10834875, @taavi wrote: >>>! In T376400#10834856, @Andrew wrote: >> Can you point me to some specific examples? M... [16:28:20] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108852168 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a) [16:28:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:29:35] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c) [16:29:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:38:00] (03open) 10dcaro: images: add digest only when passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/242 [17:49:05] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:53:21] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:57:58] (03update) 10dcaro: images: add digest only when passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/242 [17:58:07] (03approved) 10dcaro: images: add digest only when passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/242 [18:02:43] 06cloud-services-team, 10Cloud-VPS: sso failure in codfw1dev (labtesthorizon.wikimedia.org) - https://phabricator.wikimedia.org/T409328 (10Andrew) 03NEW [18:11:41] (03update) 10dcaro: images: add digest only when passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/242 [18:17:28] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85.11%) at host tools-nfs-3 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [18:29:04] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/buildpacks/apt-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/apt-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T408783) [18:31:04] 06Toolforge-standards-committee, 07User-notice: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290#11346025 (10JJMC89) 05Open→03In progress a:03JJMC89 [18:31:36] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/cmake-buildpack] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/cmake-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [18:31:42] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/cmake-buildpack] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/cmake-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [18:32:03] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/buildpacks/apt-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/apt-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T408783) [18:36:09] (03open) 10raymond-ndibe: [dev] fix CONTRIBUTING.md [repos/cloud/toolforge/buildpacks/go-toml] (v2) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/go-toml/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [18:41:03] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19112506240 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c) [18:41:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:41:19] (03update) 10dcaro: images: add digest only when passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/242 [18:41:21] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [18:44:25] (03merge) 10dcaro: images: add digest only when passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/242 [18:47:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.439-20251105184442-25e20287 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1058 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T408583) [18:47:14] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.439-20251105184442-25e20287 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1058 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T408583) [18:47:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [18:48:00] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/dotnetcore-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/dotnetcore-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [18:57:59] (03update) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291 [19:00:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [19:01:22] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/dotnetcore-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/dotnetcore-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [19:01:28] 06Toolforge-standards-committee, 07User-notice: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290#11346193 (10JJMC89) [19:02:10] (03update) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/dotnetcore-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/dotnetcore-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [19:04:02] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/clojure-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/clojure-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [19:04:55] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/buildpacks/apt-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/apt-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [19:05:59] 06Toolforge-standards-committee, 07User-notice: November 2025 TFSC appointments - https://phabricator.wikimedia.org/T409290#11346204 (10JJMC89) [19:06:11] FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:07:50] (03open) 10raymond-ndibe: [dev] CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/rust-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/rust-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) [19:11:00] (03open) 10raymond-ndibe: [dev] fix/add README.md, CONTRIBUTING.md [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/243 (https://phabricator.wikimedia.org/T408783) [19:11:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [19:12:36] (03open) 10taavi: Update taavi's canonical account name [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/32 [19:12:41] (03update) 10taavi: Update taavi's canonical account name [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/32 [19:14:32] (03open) 10raymond-ndibe: [dev] CONTRIBUTING.md [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/297 (https://phabricator.wikimedia.org/T408783) [19:14:50] (03update) 10raymond-ndibe: [dev] CONTRIBUTING.md [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/297 (https://phabricator.wikimedia.org/T408783) [19:18:04] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1059 (https://phabricator.wikimedia.org/T408783) [19:23:45] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [19:24:02] (03approved) 10dcaro: jobs-api: bump to 0.0.439-20251105184442-25e20287 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1058 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T408583) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:24:10] (03merge) 10dcaro: jobs-api: bump to 0.0.439-20251105184442-25e20287 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1058 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T408583) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:29:11] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set ssh IO to none when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341 (10Aklapper) 03NEW p:05Triage→03Low [19:30:50] (03update) 10taavi: Update taavi's canonical account name [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/32 [19:37:12] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995211 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9) [19:37:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [19:37:26] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995191 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9) [19:37:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [19:46:49] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#11346429 (10Aklapper) [19:48:01] (03PS1) 10Aklapper: Set all IO to none and hidden for internal URIs of Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) [19:49:00] (03CR) 10Aklapper: "**This is untested.** It is also how I imagine Python, a language which I do not know." [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper) [19:50:23] (03CR) 10CI reject: [V:04-1] Set all IO to none and hidden for internal URIs of Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper) [19:57:26] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f) [19:57:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [20:32:28] RESOLVED: NfsAlmostFull: The NFS drive is over 85% capacity (currently 86.04%) at host tools-nfs-3 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [21:30:19] 10Tool-wsindex, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11346900 (10LGoto) p:05Triage→03Medium [21:34:03] 06cloud-services-team, 10Cloud-VPS: `logging` project missing normal DNS zone delegation - https://phabricator.wikimedia.org/T409361 (10bd808) 03NEW [21:38:06] 06cloud-services-team, 10Cloud-VPS: `logging` project missing normal DNS zone delegation - https://phabricator.wikimedia.org/T409361#11346953 (10taavi) [21:47:41] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1059 (https://phabricator.wikimedia.org/T408783) [21:48:31] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/80 (https://phabricator.wikimedia.org/T408783) [21:55:35] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/80 (https://phabricator.wikimedia.org/T408783) [21:56:30] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/68 (https://phabricator.wikimedia.org/T408783) [21:59:23] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/68 (https://phabricator.wikimedia.org/T408783) [22:02:18] 06cloud-services-team, 10Cloud-VPS: `logging` project missing normal DNS zone delegation - https://phabricator.wikimedia.org/T409361#11347067 (10bd808) [22:03:38] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/101 (https://phabricator.wikimedia.org/T408783) [22:07:44] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/84 (https://phabricator.wikimedia.org/T408783) [22:08:02] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/84 (https://phabricator.wikimedia.org/T408783) [22:08:27] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T408783) [22:10:16] 10Cloud-VPS (Quota-requests): Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365 (10bd808) 03NEW [22:10:40] 10Cloud-VPS (Quota-requests): Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11347118 (10bd808) [22:11:54] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T408783) [22:12:19] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783) [22:12:28] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783) [22:12:52] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) [22:13:25] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T408783) [22:14:20] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T408783) [22:14:49] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/33 (https://phabricator.wikimedia.org/T408783) [22:15:14] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/31 (https://phabricator.wikimedia.org/T408783) [22:15:40] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/149 (https://phabricator.wikimedia.org/T408783) [22:17:24] (03approved) 10bd808: Update taavi's canonical account name [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/32 (owner: 10taavi) [22:17:40] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/149 (https://phabricator.wikimedia.org/T408783) [22:18:17] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/35 (https://phabricator.wikimedia.org/T408783) [22:19:49] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/35 (https://phabricator.wikimedia.org/T408783) [22:21:02] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/35 (https://phabricator.wikimedia.org/T408783) [22:21:41] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/78 (https://phabricator.wikimedia.org/T408783) [22:25:15] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/78 (https://phabricator.wikimedia.org/T408783) [22:26:17] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/78 (https://phabricator.wikimedia.org/T408783) [22:28:53] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) [22:29:58] (03merge) 10taavi: Update taavi's canonical account name [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/32 [22:31:54] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) [22:32:17] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) [22:32:59] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/28 (https://phabricator.wikimedia.org/T408783) [22:33:13] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/28 (https://phabricator.wikimedia.org/T408783) [22:33:48] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/40 (https://phabricator.wikimedia.org/T408783) [22:34:16] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/149 (https://phabricator.wikimedia.org/T408783) [22:35:56] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/149 (https://phabricator.wikimedia.org/T408783) [22:36:32] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/99 (https://phabricator.wikimedia.org/T408783) [22:38:46] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/99 (https://phabricator.wikimedia.org/T408783) [22:38:58] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/99 (https://phabricator.wikimedia.org/T408783) [22:39:28] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/67 (https://phabricator.wikimedia.org/T408783) [22:39:51] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/12 (https://phabricator.wikimedia.org/T408783) [22:39:59] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/12 (https://phabricator.wikimedia.org/T408783) [22:41:31] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11347233 (10Nehtechnine) 05Open→03Resolved a:03Nehtechnine [22:42:11] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11347247 (10Nehtechnine) 05In progress→03Resolved a:03Nehtechnine [22:43:21] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/21 (https://phabricator.wikimedia.org/T408783) [22:44:01] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/45 (https://phabricator.wikimedia.org/T408783) [22:44:15] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11347269 (10Nehtechnine) 05Open→03Resolved a:03Nehtechnine [22:46:09] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/45 (https://phabricator.wikimedia.org/T408783) [22:46:37] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/45 (https://phabricator.wikimedia.org/T408783) [22:47:20] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/16 (https://phabricator.wikimedia.org/T408783) [22:47:27] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/16 (https://phabricator.wikimedia.org/T408783) [22:48:19] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/9 (https://phabricator.wikimedia.org/T408783) [22:50:05] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/9 (https://phabricator.wikimedia.org/T408783) [22:50:55] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/24 (https://phabricator.wikimedia.org/T408783) [22:51:48] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/24 (https://phabricator.wikimedia.org/T408783) [22:54:00] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/12 (https://phabricator.wikimedia.org/T408783) [22:55:35] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/12 (https://phabricator.wikimedia.org/T408783) [22:55:37] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/12 (https://phabricator.wikimedia.org/T408783) [22:56:40] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/24 (https://phabricator.wikimedia.org/T408783) [22:58:41] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11347302 (10Pppery) 05Resolved→03Open a:05Nehtechnine→03None [22:59:03] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11347305 (10Pppery) 05Resolved→03Open a:05Nehtechnine→03None [22:59:09] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-gen-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T408783) [22:59:13] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-gen-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T408783) [22:59:15] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11347308 (10Pppery) 05Resolved→03Open a:05Nehtechnine→03None [23:00:13] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/87 (https://phabricator.wikimedia.org/T408783) [23:01:58] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/87 (https://phabricator.wikimedia.org/T408783) [23:02:42] (03open) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/3 (https://phabricator.wikimedia.org/T408783) [23:02:52] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/3 (https://phabricator.wikimedia.org/T408783) [23:04:07] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/3 (https://phabricator.wikimedia.org/T408783) [23:06:11] FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown