[00:06:56] FIRING: SystemdUnitDown: The service unit logrotate.service is in failed status on host cloudgw1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:01:56] RESOLVED: SystemdUnitDown: The service unit logrotate.service is in failed status on host cloudgw1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:01:41] PROBLEM - SSH on cloudcephosd1006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:05:47] FIRING: NodeDown: Node cloudcephosd1006 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:06:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [06:36:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [06:39:35] RECOVERY - SSH on cloudcephosd1006 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:40:47] RESOLVED: NodeDown: Node cloudcephosd1006 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:11:24] 06cloud-services-team, 10Toolforge, 07Kubernetes: Unable to load Toolforge job: ERROR: TjfCliError: Unknown error (403 Client Error: Forbidden for url - https://phabricator.wikimedia.org/T399417#11018485 (10Multichill) Problem persists. Any update? [08:40:43] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS: [trove] Disk full for DBapp instance in glamwikidashboard project - https://phabricator.wikimedia.org/T396724#11018497 (10YochayCO) Hi, thank you for your efforts! I really don't know much about the CONCURRENTLY so yeah, I trust you more than I trust GPT ;)... [08:46:35] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS: [trove] Disk full for DBapp instance in glamwikidashboard project - https://phabricator.wikimedia.org/T396724#11018498 (10YochayCO) Regarding the time for the reconfiguration, it is better if we do it after 12:00 UTC because of the daily script that I'm hopefu... [10:44:22] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 06SRE-OnFire, 10Sustainability (Incident Followup): Cloud Ceph misbehaving on Debian Bookworm - https://phabricator.wikimedia.org/T399858#11018534 (10Andrew) cloudcephosd1006 alerted over night; I'm going to reboot it so that we get another 36 or so hours... [13:53:12] FIRING: [2x] ProjectProxyMainProxyCertificateExpiry: Certificate for proxy on proxy-5 is about to expire (6d 23h 29m 52s to expiration) - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyCertificateExpiry [16:16:40] (03PS3) 10Ketulucas: T225806. Setup browser history states for image participation [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1132174 [16:16:40] (03PS2) 10Ketulucas: T225806. Remove all unwanted indentation [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1134334 [16:16:40] (03PS2) 10Ketulucas: Bug:T357238. Updated the Help Page on the ISA Tool. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1134338 [16:16:40] (03PS1) 10Ketulucas: Bug:T336472. I just Add link to user guidelines [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1170729 [16:35:28] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85.6%) at host paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [16:37:19] (03open) 10bd808: Add CORS preflight and response headers [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/11 (https://phabricator.wikimedia.org/T397571) [16:37:59] (03CR) 10Ketulucas: "please help me with this review" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1170729 (owner: 10Ketulucas) [16:38:57] 10Tool-gitlab-content, 13Patch-For-Review: Support CORS in gitlab-content tool - https://phabricator.wikimedia.org/T397571#11018642 (10bd808) 05Open→03In progress a:03bd808 [17:00:22] (03update) 10bd808: Add CORS preflight and response headers [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/11 (https://phabricator.wikimedia.org/T397571) [17:02:16] (03merge) 10bd808: Add CORS preflight and response headers [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/11 (https://phabricator.wikimedia.org/T397571) [17:47:24] (03open) 10bd808: Automatically add 'immutable' Cache-Control header to permalinks [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/12 (https://phabricator.wikimedia.org/T393928) [17:49:24] (03merge) 10bd808: Automatically add 'immutable' Cache-Control header to permalinks [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/12 (https://phabricator.wikimedia.org/T393928) [17:54:25] (03CR) 10Eugene233: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1170729 (owner: 10Ketulucas) [17:58:49] (03CR) 10Eugene233: "Should the guidelines not point to https://commons.wikimedia.org/wiki/Commons:Depiction_guidelines instead?" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1170729 (owner: 10Ketulucas) [18:02:13] 10Tool-gitlab-content, 13Patch-For-Review: Add maxage/smaxage cache header controls to gilab-content proxy - https://phabricator.wikimedia.org/T393928#11018686 (10bd808) 05In progress→03Resolved [18:02:16] 10Tool-gitlab-content: Support CORS in gitlab-content tool - https://phabricator.wikimedia.org/T397571#11018687 (10bd808) 05In progress→03Resolved