[00:05:55] FIRING: MaxConntrack: Max conntrack at 84.16% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:14:05] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-47 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [00:31:44] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 [00:34:05] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-47 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [00:40:29] RESOLVED: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85.62%) at host paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [00:47:47] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 [00:55:55] RESOLVED: MaxConntrack: Max conntrack at 83.79% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [01:04:05] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-47 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [01:39:05] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-57 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:01:31] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 4 deleted instances on gitlab-runners-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [07:44:20] 10Toolforge (Toolforge iteration 21): [maintain-harbor,ci] the version number does not get bumped on every release - https://phabricator.wikimedia.org/T396504#10976343 (10Raymond_Ndibe) a:03Raymond_Ndibe [08:53:01] 10Toolforge (Toolforge iteration 21): [maintain-harbor,ci] the version number does not get bumped on every release - https://phabricator.wikimedia.org/T396504#10976365 (10Raymond_Ndibe) Unsure why this is happening since it's all the same CI pipeline. Interesting [08:57:00] (03open) 10raymond-ndibe: [maintain-harbor] remove "environment" from charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/867 (https://phabricator.wikimedia.org/T396504) [08:58:16] (03update) 10raymond-ndibe: [maintain-harbor] remove "environment" from charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/867 (https://phabricator.wikimedia.org/T396504) [09:43:57] (03Restored) 10Eugene233: rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [09:44:11] (03PS3) 10Eugene233: rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [09:44:52] (03CR) 10CI reject: [V:04-1] rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [09:51:41] (03PS4) 10Eugene233: rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [10:04:55] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1163075 (owner: 10Jacob4code) [10:05:09] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166450 (owner: 10Jacob4code) [10:05:24] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1164323 (owner: 10Jacob4code) [10:10:31] (03CR) 10Essa237: [C:03+1] rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [10:11:18] (03CR) 10Eugene233: [C:03+2] rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [10:12:04] (03Merged) 10jenkins-bot: rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [10:12:41] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1163075 (owner: 10Jacob4code) [10:12:47] (03CR) 10CI reject: [V:04-1] Cannot read properties of undefined fixed and minor changes. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1163075 (owner: 10Jacob4code) [10:12:47] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166450 (owner: 10Jacob4code) [10:12:54] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1164323 (owner: 10Jacob4code) [10:12:59] (03CR) 10CI reject: [V:04-1] Create a root package.json file to run both frontend and server [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1164323 (owner: 10Jacob4code) [10:16:41] (03CR) 10Eugene233: Updated README Elaborate steps to run tool on local setup (032 comments) [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166450 (owner: 10Jacob4code) [10:21:49] (03Abandoned) 10Eugene233: Folder restructuring [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1155646 (owner: 10NkwadaNora) [10:37:04] (03Restored) 10NkwadaNora: Folder restructuring [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1155646 (owner: 10NkwadaNora) [11:21:20] (03Abandoned) 10Jacob4code: Create a root package.json file to run both frontend and server [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1164323 (owner: 10Jacob4code) [11:27:20] (03PS2) 10Jacob4code: Updated README Elaborate steps to run tool on local setup [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166450 [11:31:09] (03CR) 10Jacob4code: "Done" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166450 (owner: 10Jacob4code) [11:55:40] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166379 (owner: 10NkwadaNora) [11:55:51] (03CR) 10CI reject: [V:04-1] [fix]: created a pyproject.toml file at the root of the project, this tells tox to skip trying to build a Python distribution and just run your npm lint commands insteadsince the project is javascript, typeScript and not a python project [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166379 (owner: 10NkwadaNora) [15:31:19] (03PS2) 10Jacob4code: Cannot read properties of undefined fixed and minor changes. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1163075 [15:34:42] (03PS3) 10Jacob4code: Cannot read properties of undefined fixed Detail about the collaboration WIP [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1163075 [15:47:10] (03CR) 10Jacob4code: [C:03+1] Cannot read properties of undefined fixed Detail about the collaboration WIP [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1163075 (owner: 10Jacob4code) [15:56:22] (03CR) 10Bovimacoco: "Please do help review this" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1166053 (https://phabricator.wikimedia.org/T390402) (owner: 10Bovimacoco) [17:25:29] (03PS1) 10Eugene233: Adding pagination for campaign for campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1166478 (https://phabricator.wikimedia.org/T398759) [20:02:46] (03CR) 10Eugene233: [C:03+2] Adding pagination for campaign for campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1166478 (https://phabricator.wikimedia.org/T398759) (owner: 10Eugene233) [20:03:05] (03Merged) 10jenkins-bot: Adding pagination for campaign for campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1166478 (https://phabricator.wikimedia.org/T398759) (owner: 10Eugene233) [20:09:09] (03PS1) 10Eugene233: Generate most recent stats file upon request [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1166481 (https://phabricator.wikimedia.org/T398760) [20:09:42] (03CR) 10Eugene233: [C:03+2] Generate most recent stats file upon request [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1166481 (https://phabricator.wikimedia.org/T398760) (owner: 10Eugene233) [20:10:02] (03Merged) 10jenkins-bot: Generate most recent stats file upon request [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1166481 (https://phabricator.wikimedia.org/T398760) (owner: 10Eugene233) [23:59:34] FIRING: DiskSpace: Disk space cloudcontrol2006-dev:9100:/ 5.981% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2006-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace