[00:03:28] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85%) at host tools-nfs-3 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [02:53:28] RESOLVED: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85.04%) at host tools-nfs-3 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [03:55:07] (03open) 10santanuwiki25: Initial setup for Wiki Loves Butterfly api [toolforge-repos/wlbapi] - 10https://gitlab.wikimedia.org/toolforge-repos/wlbapi/-/merge_requests/11 [03:55:14] (03merge) 10santanuwiki25: Initial setup for Wiki Loves Butterfly api [toolforge-repos/wlbapi] - 10https://gitlab.wikimedia.org/toolforge-repos/wlbapi/-/merge_requests/11 [05:13:07] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11510113 (10Tiven2240) [05:16:14] (03open) 10r4356th: Install mw-api-client from Git [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/1 [05:17:12] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11510114 (10Tiven2240) [05:35:25] (03merge) 10r4356th: Install mw-api-client from Git [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/1 [07:39:56] FIRING: SystemdUnitDown: The service unit backup_vms.service is in failed status on host cloudbackup1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:34:56] FIRING: SystemdUnitDown: The systemd unit backup_vms.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:23:19] (03open) 10santanuwiki25: Initail setup [toolforge-repos/wlbapi] - 10https://gitlab.wikimedia.org/toolforge-repos/wlbapi/-/merge_requests/12 [11:23:21] (03merge) 10santanuwiki25: Initail setup [toolforge-repos/wlbapi] - 10https://gitlab.wikimedia.org/toolforge-repos/wlbapi/-/merge_requests/12 [11:36:08] 10Tool-curator: Add abort function to Mapillary upload batches in Curator - https://phabricator.wikimedia.org/T414208#11510325 (10DaxServer) 05Open→03In progress [12:07:34] FIRING: DiskSpace: Disk space cloudbackup2003:9100:/srv 6.786% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup2003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:21:34] FIRING: DiskSpace: Disk space cloudbackup2003:9100:/srv 3.729% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup2003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:26:38] (03open) 10santanuwiki25: initial code setup [toolforge-repos/wlbapi] - 10https://gitlab.wikimedia.org/toolforge-repos/wlbapi/-/merge_requests/13 [13:26:42] (03merge) 10santanuwiki25: initial code setup [toolforge-repos/wlbapi] - 10https://gitlab.wikimedia.org/toolforge-repos/wlbapi/-/merge_requests/13 [13:35:11] FIRING: SystemdUnitDown: The systemd unit backup_vms.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:54:28] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate patchdemo2.visualeditor.eqiad1.wikimedia.cloud is about to expire in 27d 23h 58m 5s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [16:27:07] (03PS2) 10Pppery: Fix typo on user setting route [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1012648 (https://phabricator.wikimedia.org/T343438) (owner: 10Josefanthony) [16:32:40] (03CR) 10Pppery: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1193969 (https://phabricator.wikimedia.org/T288418) (owner: 10Shivaansh Singh) [16:38:00] (03CR) 10Pppery: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1215160 (https://phabricator.wikimedia.org/T224005) (owner: 10Rehan_khan_78) [16:41:34] RESOLVED: DiskSpace: Disk space cloudbackup2003:9100:/srv 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup2003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:42:34] RESOLVED: DiskSpace: Disk space cloudbackup2003:9100:/srv 4.137% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup2003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:44:56] RESOLVED: SystemdUnitDown: The systemd unit backup_vms.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:59:28] RESOLVED: PuppetCertificateAboutToExpire: Puppet CA certificate patchdemo2.visualeditor.eqiad1.wikimedia.cloud is about to expire in 27d 21h 58m 5s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [18:51:28] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85.01%) at host tools-nfs-3 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [22:22:42] 10VPS-project-Codesearch: Codesearch down/unreachable (2026-01-11) - https://phabricator.wikimedia.org/T414290 (10SomeRandomDeveloper) 03NEW