[09:52:10] !log admin icinga downtime toolschecker, paws, etc for 2h, because cloudvirt reboots [09:52:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:58:31] !log admin cloudvirt1008 is rebooting [09:58:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:59:29] !log tools several worker nodes (5) not available because cloudvirt1008 is rebooting [09:59:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:06:13] !log admin cloudvirt1008 rebooted just fine (very slow though) [10:06:18] !log admin cloudvirt1009 is rebooting [10:06:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:07:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:08:20] !log tools several worker nodes (6) not available because cloudvirt1009 is rebooting [10:08:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:19:02] !log admin cloudvirt1009 rebooted just fine (very slow though) [10:19:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:20:10] !log admin cloudvirt1012 is rebooting [10:20:40] !log tools several worker nodes (7) not available because cloudvirt1012 is rebooting [10:21:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:21:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:32:44] !log admin cloudvirt1012 rebooted just fine (very slow, 35 VMs) [10:32:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:32:48] !log admin cloudvirt1013 is rebooting [10:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:33:57] !log tools several sgewebgrid-lighttpd nodes (9) not available because cloudvirt1013 is rebooting [10:33:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:44:45] !log admin cloudvirt1013 rebooted well [10:44:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:59:47] !log toolsbeta re-create toolsbeta-test-proxy-01 as Debian Buster (T235059) [11:59:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:59:51] T235059: Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster - https://phabricator.wikimedia.org/T235059 [12:33:15] !log tools drain tools-worker-1010 to rebalance load [12:33:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:34:15] arturo: question about the status, will it affect toolforge as well (i forget what cloudvirt are for) [12:34:52] Zppix: cloudvirts are the jupervisor servers, those running the VMs we have in CloudVPS, including Toolforge, yes [12:35:09] !log tools uncordon tools-worker-1029 (was disabled for unknown reasons) [12:35:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:35:21] arturo: okay, sorry for bothering, just need to know if i should expect possible interruptions with my toolforge projects [12:36:05] Zppix: yes. I would say some of your tools or webservices might be rescheduled into other worker nodes if they happen to be running in one of the VMs we need to reboot [12:36:26] arturo: alright thanks, hope everything goes according to plan :) [12:36:52] :-) thanks for your understanding [12:37:46] !log tools drain tools-worker-1038 to rebalance load in the k8s cluster [12:37:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:42:31] np [14:01:07] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @James_F - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:46:31] !log tools drained and cordoned tools-worker-1029 after status reset on reboot [14:46:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:51:02] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @duesen - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:21:16] !log tools drained tools-worker-1020/23/33/35/36/40 to rebalance the cluster [15:31:51] phamhi you will want to relog stashbot came back [15:32:28] Oh good catch.. thanks [15:32:45] !log tools drained tools-worker-1020/23/33/35/36/40 to rebalance the cluster [15:32:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:38:18] Hmm, wikibugs i think needs restarting [15:38:27] it's not outputting gerrit changes [15:38:40] & bug reports [15:40:17] phamhi would you be able to restart wikibugs please? (https://www.mediawiki.org/wiki/Wikibugs) [15:41:03] Hmm, I'm in the gerrit group (labs-tools-wikibugs2) but not a member of the Tool so can't help, sorry. [15:41:04] let me take a look [15:41:39] thanks! [15:47:18] thanks phamhi ! [15:47:40] anytime..thanks for letting us know about the issue [15:47:56] phamhi: probably a good idea to !log the restart [15:49:58] !log tools.wikibugs restarted wikibugs since it had stopped reporting [15:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [15:50:54] thanks bd808..I was looking for the category where to report it to [15:51:36] fair. I guess you haven't memorized all the things yet ;) [15:53:34] what does SAL stand for [15:53:43] Server Access Log [15:54:22] Server Admin Log apparently -- https://wikitech.wikimedia.org/wiki/SAL [15:56:27] ah ok..a bit anti-climatic..I was hoping it was a reference to something like the AI robot HAL from Space Odyssey [15:56:42] lol [15:56:51] SPACE Admin Log [15:57:01] +1 for that new name [15:57:32] "The thing that helps us figure out wtf changed before things broke" [16:06:28] :-) [20:53:26] !log openstack restarting designate-sink in eqiad1 with updated wmfsink handler T235127 [20:53:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [20:53:29] T235127: wmfsink designate handler not running during vm deletes - https://phabricator.wikimedia.org/T235127 [21:33:19] !log openstack cleanup designate dns leaks in eqiad1 T235127 [21:33:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [21:33:22] T235127: wmfsink designate handler not running during vm deletes - https://phabricator.wikimedia.org/T235127 [22:52:18] !log tools removing test instances tools-sssd-sgeexec-test-[12] from SGE [22:52:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:02:27] !log tools.openstack-browser Restarting to see if the webservice ends up on a less loaded exec node [23:02:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL