[00:49:39] !log tools depooling tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 for flavor update [00:49:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:58:25] !log tools repooling tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 after flavor update [00:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:00:59] !log tools depooling tools-sgeexec-0917, tools-sgeexec-0918, tools-sgeexec-0919, tools-sgeexec-0920 for flavor update [01:01:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:08:26] !log tools repooling tools-sgeexec-0917, tools-sgeexec-0918, tools-sgeexec-0919, tools-sgeexec-0920 after flavor update [01:08:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:11:54] !log tools depooling tools-sgeexec-0901, tools-sgeexec-0905, tools-sgeexec-0910, tools-sgeexec-0911, tools-sgeexec-0912 for flavor update [01:11:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:20:43] !log tools repooling tools-sgeexec-0901, tools-sgeexec-0905, tools-sgeexec-0910, tools-sgeexec-0911, tools-sgeexec-0912 after flavor update [01:20:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:42:54] !log tools.bridgebot Restarting bot. Seems to have lost connection with some channels. [04:42:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [04:44:26] * bd808 waves to wm-bb [04:49:08] [telegram] Was it down? Just delivered messages to discord too (sent like 8 hours ago) [08:50:26] !log admin installing iptables from buster-bpo in cloudvirt1036 (T263205 and T262979) [08:50:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:50:29] T263205: Strange NFS client outage on VMs running on cloudvirt1036 - https://phabricator.wikimedia.org/T263205 [08:50:30] T262979: Buster cloudvirts unable to launch new VMs - https://phabricator.wikimedia.org/T262979 [08:59:10] !log admin disable puppet in all buster cloudvirts (cloudvirt[1024,1031-1039].eqiad.wmnet) to merge a patch for T263205 and T262979 [08:59:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:59:13] T263205: Strange NFS client outage on VMs running on cloudvirt1036 - https://phabricator.wikimedia.org/T263205 [08:59:14] T262979: Buster cloudvirts unable to launch new VMs - https://phabricator.wikimedia.org/T262979 [09:50:18] !log admin enabling puppet in cloudvirts and effectively merging patches from T262979 [09:50:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:50:21] T262979: Buster cloudvirts unable to launch new VMs - https://phabricator.wikimedia.org/T262979 [09:56:07] !log admin rebooting cloudvirt1039 (spare) to try to fix some weird libvirtd failure [09:56:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:58:48] 11 [10:16:15] !log admin cloudvirt1039 libvirtd service issues were fixed with a reboot [10:16:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:50:06] !log tools depooling tools-sgeexec-0913, tools-sgeexec-0915, tools-sgeexec-0916 for flavor update [13:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:54:49] !log tools repooling tools-sgeexec-0913, tools-sgeexec-0915, tools-sgeexec-0916 [13:54:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:10:17] !log tools depooling tools-sgewebgrid-lighttpd-0914, tools-sgewebgrid-generic-0902, tools-sgewebgrid-lighttpd-0919, tools-sgewebgrid-lighttpd-0918 [16:10:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:19:18] andrewbogott, same issue with alerts on these as yesterday [16:19:42] but now it doesn't page :) [16:20:00] I will downtime those [16:20:40] no worries, it's useful noise still for me at this point. I'm still tweaking how I'm alerted :-) [16:26:29] * Reedy waits for the wikibugs job on tools-sgeexec-0912 to die [16:26:35] taking a while [16:29:07] ~5 minutes? [16:29:37] !log tools.wikibugs restarted due to huge phab lag to irc [16:29:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [16:38:22] !log tools repooling tools-sgewebgrid-lighttpd-0914, tools-sgewebgrid-generic-0902, tools-sgewebgrid-lighttpd-0919, tools-sgewebgrid-lighttpd-0918 [16:38:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:40:15] !log tools depooling tools-k8s-worker-31, 32, 36, 39, 40 [16:40:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:47:33] !log tools repooling tools-k8s-worker-31, 32, 36, 39, 40 [17:47:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:48:37] !log tools depooling tools-k8s-worker-41, 43, 44, 47, 48, 49, 50, 51 [17:48:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:50:35] Amir1: could you join the global checkuser channel or PM me please? [18:40:21] !help Hello, why is wmopbot tool on Toolforge down? (503 error) [18:40:21] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [18:41:56] Also, I have question for T260647. Is something needed to do on Cloud VPS? [18:41:56] T260647: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647 [18:42:29] Kizule: I will see what happened with wmopbot [18:42:46] Can I create user "kizule" and drop "zoranzoki21"? Will it make some problems? [18:43:02] danilo: OK, thanks [18:54:26] danilo: Now I get "No webservice" [18:54:47] yes, I am working on it [18:55:18] danilo: Great [18:59:08] as for T260647, that'll take a bit of work by WMCS. There are several steps in a rename, and we will have to do some things on Cloud VPS, I'm sure. It's a fairly manual process, unfortunately [18:59:09] T260647: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647 [18:59:34] You won't be able to rename yourself on cloudvps [18:59:41] That's stuff admins need to do [19:00:05] We'll try to keep the ticket updated when there are steps you need to take cc: balloons [19:00:10] bstorm: I mean on creating account "kizule" and deleting "zoranzoki21" on instances. You have to do it? [19:00:39] Ah, that's automatic and is keyed off of LDAP [19:00:55] It won't delete things on instances that are already there. Anything like that will need help from us. [19:01:07] But creating is done automatically [19:01:09] bstorm: okay [19:01:25] thanks for explanations! :) [19:02:00] !log tools repooling tools-k8s-worker-41, 43, 44, 47, 48, 49, 50, 51 [19:02:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:04:42] !log tools depooling tools-k8s-worker-30, 33, 34, 57, 60 [19:04:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:04:52] Kizule: the wmopbot web interface is back [19:05:40] danilo: Thanks, I've requested changing cloak successfully. :) [19:41:17] !log tools repooling tools-k8s-worker-30, 33, 34, 57, 60 [19:41:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL