[00:06:56] FIRING: SystemdUnitDown: The service unit logrotate.service is in failed status on host cloudgw1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:14:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [00:19:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [01:01:56] RESOLVED: SystemdUnitDown: The service unit logrotate.service is in failed status on host cloudgw1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:39:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [02:49:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-34 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [03:14:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-34 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [04:33:10] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1170323 (owner: 10L10n-bot) [04:33:37] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1170325 (owner: 10L10n-bot) [05:08:36] 10Tool-Global-user-contributions, 10Tool-globalcontribution, 10CheckUser-GlobalContributions, 06Trust and Safety Product Team: Global contributions marked all Project: namespace contributions as Meta: namespace - https://phabricator.wikimedia.org/T399922 (10Lemonaka) 03NEW [05:09:37] 10Tool-Global-user-contributions, 10Tool-globalcontribution, 10CheckUser-GlobalContributions, 06Trust and Safety Product Team: Bug report/;Global contributions marked all Project: namespace contributions as Meta: namespace - https://phabricator.wikimedia.org/T399922#11015747 (10Lemonaka) [05:09:50] 10Tool-Global-user-contributions, 10Tool-globalcontribution, 10CheckUser-GlobalContributions, 06Trust and Safety Product Team: Bug report:Global contributions marked all Project: namespace contributions as Meta: namespace - https://phabricator.wikimedia.org/T399922#11015750 (10Lemonaka) [05:49:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-34 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [07:38:52] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#11015932 (10Sascha) 05Resolved→03Open @dcaro many thanks! I tried, but I still couldn't make it work. 1. Loo... [07:59:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-34 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [08:17:09] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 06SRE-OnFire, 10Sustainability (Incident Followup): Cloud Ceph misbehaving on Debian Bookworm - https://phabricator.wikimedia.org/T399858#11016022 (10fnegri) 05Open→03In progress a:03fnegri Memory usage on cloudcephosd1006 did reset at 18:00 UTC yest... [08:49:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [09:23:02] 10Tool-archive-externa-links, 10Wikidata, 10Wikidata-Gadgets: Création de tableau de bord - https://phabricator.wikimedia.org/T399889#11016165 (10SuzanneWood-WMDE) Could you please add more details to the ticket, such as which kind of dashboard? Thanks [09:39:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [09:43:34] 10Tool-archive-externa-links, 10Wikidata, 10Wikidata-Gadgets: Création de tableau de bord - https://phabricator.wikimedia.org/T399889#11016206 (10paulwiki) Ok [09:48:44] 10Tool-archive-externa-links, 10Wikidata, 10Wikidata-Gadgets: Création de tableau de bord - https://phabricator.wikimedia.org/T399889#11016222 (10paulwiki) [10:02:30] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki zghwiktionary - https://phabricator.wikimedia.org/T399788#11016250 (10fnegri) p:05Triage→03Low [10:03:08] 06cloud-services-team, 10Data-Services, 06Data-Persistence: [wikireplicas] Route alerts to WMCS team - https://phabricator.wikimedia.org/T381589#11016251 (10fnegri) a:05fnegri→03None [10:03:50] 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [dbaas,toolsdb] Add support for management of toolsdb databases within toolforge - https://phabricator.wikimedia.org/T384591#11016252 (10fnegri) a:05fnegri→03None [10:04:23] 06cloud-services-team, 10Toolforge: [toolsdb] Migrate quickstatements db to Trove - https://phabricator.wikimedia.org/T369177#11016253 (10fnegri) a:05fnegri→03None [10:04:30] 06cloud-services-team, 10Toolforge: [toolsdb] Migrate mixnmatch db to Trove - https://phabricator.wikimedia.org/T350862#11016255 (10fnegri) a:05fnegri→03None [10:04:40] 06cloud-services-team, 10Cloud-VPS, 10Cumin, 06Infrastructure-Foundations, 13Patch-For-Review: [cumin] [openstack] Openstack backend fails when project is not set - https://phabricator.wikimedia.org/T346453#11016256 (10fnegri) a:05fnegri→03None [10:04:57] 06cloud-services-team, 10Toolforge, 07Epic: Migrate largest ToolsDB users to Trove - https://phabricator.wikimedia.org/T291782#11016257 (10fnegri) a:05fnegri→03None [10:14:05] 06cloud-services-team, 10wikitech.wikimedia.org, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: wikitech-static: resume daily dumps - https://phabricator.wikimedia.org/T398968#11016266 (10BTullis) OK, I think that this is ready to go. There are three patches that need to be merged in se... [10:26:00] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: Request to increase Object Storage capacity - Wikilink project - https://phabricator.wikimedia.org/T399746#11016297 (10fnegri) I think it's fine to increase the quota to 100K or even 200K objects, but I'd like to understand why the curren... [10:29:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [10:30:18] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: Request to increase Object Storage capacity - Wikilink project - https://phabricator.wikimedia.org/T399746#11016307 (10fnegri) For reference the current quota was set in {T392870}. [10:33:11] 06cloud-services-team, 10wikitech.wikimedia.org, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: wikitech-static: resume daily dumps - https://phabricator.wikimedia.org/T398968#11016326 (10fnegri) Thanks @BTullis! [10:34:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [10:34:29] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 [10:39:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [10:50:48] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 [11:24:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:25:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:30:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:53:11] FIRING: [2x] ProjectProxyMainProxyCertificateExpiry: Certificate for proxy on proxy-5 is about to expire (8d 23h 29m 52s to expiration) - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyCertificateExpiry [14:16:43] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Collect access metrics from cloud-vps web proxy - https://phabricator.wikimedia.org/T371382#11017089 (10Andrew) a:05Andrew→03None [14:27:36] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure (Zuul upgrade): http://169.254.169.254/openstack/latest/user_data semi-regularly unavaliable during Magnum Kubernetes cluster builds - https://phabricator.wikimedia.org/T399596#11017157 (10Andrew) The easiest explanation for this is... [14:31:35] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: Request to increase Object Storage capacity - Wikilink project - https://phabricator.wikimedia.org/T399746#11017171 (10fnegri) p:05Triage→03Medium [14:31:48] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: Request to increase Object Storage capacity - Wikilink project - https://phabricator.wikimedia.org/T399746#11017174 (10fnegri) a:03fnegri [14:51:39] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 06SRE-OnFire, 10Sustainability (Incident Followup): Cloud Ceph misbehaving on Debian Bookworm - https://phabricator.wikimedia.org/T399858#11017233 (10fnegri) a:05fnegri→03Andrew The growth rate is slowing, but it's not flatlining as I hoped... So the s... [15:07:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:12:51] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:37:53] 10Tool-paulina: Translate Paulina into more languages - https://phabricator.wikimedia.org/T392681#11017464 (10Pepe_piton) During the Wikimedia Hackathon 2025 Istanbul, Paulina was translated to 2 new languages: [[ https://hosted.weblate.org/projects/paulina/paulina-app/az/ | Azerbaijani ]] and [[ https://hosted.... [16:39:20] 10Tool-paulina: Translate Paulina into more languages - https://phabricator.wikimedia.org/T392681#11017465 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton [16:48:00] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "auditlogging" project Buster deprecation - https://phabricator.wikimedia.org/T367522#11017483 (10Southparkfan) 05Open→03Resolved a:03Southparkfan The VM has been deleted. [16:49:29] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS: [trove] Disk full for DBapp instance in glamwikidashboard project - https://phabricator.wikimedia.org/T396724#11017487 (10fnegri) Apologies for the late reply, this is still in my backlog but I couldn't make much progress until now. I opened an upstream bug r... [17:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:24:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-72 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:36:01] 10Tool-paulina: New design for work page - https://phabricator.wikimedia.org/T391377#11017577 (10Pepe_piton) 05Open→03Resolved Done! The new design was implemented through [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/786b920b1c6100a272f9edecee3cb779a0a35e8a | this commit ]]. [17:42:52] 10Tool-paulina: New design for author page - https://phabricator.wikimedia.org/T391370#11017600 (10Pepe_piton) 05Open→03Resolved The new design of the author page was implemented through [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/a072d74f480de78091299461406b8086c3978e8e | this commi... [17:51:10] 10Tool-paulina: Allow for users to search an author or work without a search term, just using filters - https://phabricator.wikimedia.org/T393153#11017621 (10Pepe_piton) 05Open→03Resolved Done via [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/663fd5c8bab4998970ef5e39b7b67a508ae10534 |... [18:00:32] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974 (10Pepe_piton) 03NEW [18:01:20] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11017659 (10Pepe_piton) a:03Pepe_piton [18:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:19:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-72 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:03:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-72 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:33:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-72 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:40:05] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-72 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:50:04] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-72 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [21:36:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-72 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses