[04:54:18] RESOLVED: KernelErrors: Server cloudcephosd1020 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephosd1020 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [10:15:23] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: [trove] Disk full for DBapp instance in glamwikidashboard project - https://phabricator.wikimedia.org/T396724#10915701 (10YochayCO) I suspect maybe the problem is in the app and not in the db configuration... I see that we open many 'REFRESH MATERIALIZED... [12:21:47] 06cloud-services-team, 10Cloud-VPS: vps for lemmy - https://phabricator.wikimedia.org/T396948#10915760 (10Peachey88) [15:26:23] 06cloud-services-team, 10Cloud-VPS: vps for lemmy - https://phabricator.wikimedia.org/T396948#10915961 (10Aklapper) 05Open→03Stalled @Gryllida: Hi, please read https://phabricator.wikimedia.org/project/view/2875/ and fill out the task description. [15:27:10] 10Cloud-VPS (Project-requests): Request creation of lemmy VPS project - https://phabricator.wikimedia.org/T396948#10915963 (10Aklapper) [15:34:26] (03CR) 10Aklapper: "Hi, see https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines for automatical linking in the Phabricator task" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1157840 (owner: 10Bovimacoco) [15:34:44] (03PS2) 10Aklapper: Secure: Migrate hardcoded URLs to .env with validation [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1157840 (owner: 10Bovimacoco) [15:34:53] (03PS3) 10Aklapper: Secure: Migrate hardcoded URLs to .env with validation [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1157840 (https://phabricator.wikimedia.org/T390402) (owner: 10Bovimacoco) [15:36:00] (03CR) 10CI reject: [V:04-1] Secure: Migrate hardcoded URLs to .env with validation [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1157840 (https://phabricator.wikimedia.org/T390402) (owner: 10Bovimacoco) [15:52:56] FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:57:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:51:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [16:56:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-47 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [17:01:03] RESOLVED: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-47 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProce [18:41:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:00:22] 10Quarry: UI: Allow downloading output via CLI - https://phabricator.wikimedia.org/T325683#10916051 (10Dusan_Krehel) @taavi Currently, the workflow is as follows: create a request, output (as a HTML preview) and then the option to download. If you don't have a preview in the table, you can't download. If the... [19:06:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:06:54] 10Quarry: Check browser user agent and provide line endings \n for Linux based browsers - https://phabricator.wikimedia.org/T327682#10916055 (10Dusan_Krehel) @taavi How about just adding the ability (ex. check box) to download a document with Unix line endings? [20:38:04] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:47:03] 10Cloud-VPS (Project-requests): Request creation of lemmy VPS project - https://phabricator.wikimedia.org/T396948#10916107 (10Gryllida) [20:47:41] 10Cloud-VPS (Project-requests): Request creation of lemmy VPS project - https://phabricator.wikimedia.org/T396948#10916108 (10Gryllida) @Aklapper , hi, done. [20:49:53] 10Quarry: Check browser user agent and provide line endings \n for Linux based browsers - https://phabricator.wikimedia.org/T327682#10916111 (10Aklapper) @Dusan_Krehel Again, which exact tools/software which you use has problems to handle the current output with `\r\n`? Please describe a real problem to solv... [21:08:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-21 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [21:16:55] 10Cloud-VPS (Project-requests): Request creation of lemmy VPS project - https://phabricator.wikimedia.org/T396948#10916162 (10Aklapper) Thanks. Why should we host another discussion platform, as there are already many of them? This sounds like further audience fragmentation to me?