[11:11:57] !log admin Refreshing all the canary instances (T275354) [11:12:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:12:09] T275354: Puppet failures on many canary machines - https://phabricator.wikimedia.org/T275354 [11:21:24] FYI I'm roll-restarting prometheus on cloudmetrics hosts, no impact expected [11:21:53] ack [11:28:02] !log downtimed all the cloudvirt* on eqiad due to some canary machines not starting up (T275354) [11:28:03] dcaro: Unknown project "downtimed" [11:28:03] T275354: Puppet failures on many canary machines - https://phabricator.wikimedia.org/T275354 [11:55:08] Hi [11:57:41] Hi [11:59:13] !help [11:59:13] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [11:59:35] \o, sup? [15:02:10] !log Re-uploaded the debian buster 10.0 image from rbd to glance, that worked, re-spawning all the broken instances (T275378) [15:02:11] dcaro: Unknown project "Re-uploaded" [15:02:11] T275378: Cloudvirt instances failing to start: Image has no associated data - https://phabricator.wikimedia.org/T275378 [15:02:18] !log admin Re-uploaded the debian buster 10.0 image from rbd to glance, that worked, re-spawning all the broken instances (T275378) [15:02:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:58:45] !log tools cleared error state on several grid queues [16:58:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:14:58] !log admin restarting nova-compute on cloudvirt1016 and cloudvirt1036 in case it helps T275411 [17:15:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:15:10] T275411: At least one VM is live on a host that openstack disagrees with - https://phabricator.wikimedia.org/T275411 [17:56:41] calling it a day [17:56:43] cya! [17:58:42] Have a good evening [18:56:21] !log tools deleted job 1962508 from the grid to clear it up T275301 [18:56:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:56:27] T275301: Job for tools.urbanecmbot stuck in dt state - https://phabricator.wikimedia.org/T275301 [19:03:17] !log tools depooled tools-sgeexec-0918 T275411 [19:03:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:03:22] T275411: At least one VM is live on a host that openstack disagrees with - https://phabricator.wikimedia.org/T275411 [19:05:22] !log tools shutting down tools-sgeexec-0918 (with openstack to see what happens) T275411 [19:05:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:07:52] !log tools shutting down tools-sgeexec-0918 with the VM's command line (not libvirt directly yet) T275411 [19:07:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:09:56] !log tools hard rebooted tools-sgeexec-0918 from openstack T275411 [19:10:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:10:03] T275411: At least one VM is live on a host that openstack disagrees with - https://phabricator.wikimedia.org/T275411 [20:01:14] !log devtools deploy-1002 is broken because mediawiki::sites is not in Hiera (yet) [20:01:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:40:18] !log tools repooled tools-sgeexec-0918.tools.eqiad.wmflabs [20:40:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:58:25] !log devtools fixed puppet run on deploy-1002 by adding empty array of wikimedia-sites to hiera [20:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL