[13:45:55] <andrewbogott>	 !log tools depooled tools-sgeexec-0937 for move to ceph
[13:45:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:55:16] <andrewbogott>	 !log tools repooled tools-sgeexec-0937 after move to ceph
[13:55:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:55:54] <andrewbogott>	 !log tools depooled tools-sgewebgrid-lighttpd-0917 and tools-sgewebgrid-lighttpd-0920
[13:55:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:03:47] <arturo>	 andrewbogott: that paged! :-P
[14:04:12] <andrewbogott>	 I will fix the paging first, then will fix the problem :)
[14:04:23] <arturo>	 page is already ack'd
[14:04:24] <balloons>	 :-)
[14:04:32] <andrewbogott>	 oops too slow, resolved
[14:09:27] <andrewbogott>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/628094
[14:40:49] <arturo>	 thanks andrewbogott !
[14:57:00] <bd808>	 we really should put anti-affinity rules in for the things we are spread checking now that we have that ability :)
[15:00:02] <andrewbogott>	 yeah.  I think brooke did that for at least some things
[15:00:35] <bd808>	 *nod* pretty sure the paws k8s cluster at least
[15:07:46] <arturo>	 I wonder if we can add affinity rules for VMs after they have been created. So far I only saw the option in horizon when creating the VM
[15:20:52] <andrewbogott>	 !log depooling tools-sgeexec-0909, 0908, 0907, 0906, 0904 for flavor remapping
[15:20:53] <stashbot>	 andrewbogott: Unknown project "depooling"
[15:21:58] <andrewbogott>	 !log tools depooling tools-sgeexec-0909, 0908, 0907, 0906, 0904 for flavor remapping
[15:22:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:22:13] <andrewbogott>	 arturo: it might be possible to hack it into the db but I can't predict if that will work properly
[15:22:32] <arturo>	 meh
[15:30:18] <andrewbogott>	 !log tools repooling tools-sgeexec-0909, 0908, 0907, 0906, 0904
[15:30:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:32:46] <andrewbogott>	 !log depooling tools-k8s-worker-70 and tools-k8s-worker-66 for flavor remapping
[15:32:47] <stashbot>	 andrewbogott: Unknown project "depooling"
[15:34:00] <andrewbogott>	 !log tools depooling tools-k8s-worker-70 and tools-k8s-worker-66 for flavor remapping
[15:34:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:38:11] <andrewbogott>	 !log tools repooling tools-k8s-worker-70 and tools-k8s-worker-66 after flavor remapping
[15:38:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:29:04] <andrewbogott>	 !log tools depooling tools-k8s-worker-61,62,64,65,67,68,69 for flavor update
[19:29:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:55:09] <andrewbogott>	 !log tools repooling tools-k8s-worker-61,62,64,65,67,68,69 for flavor update
[19:55:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:58:55] <andrewbogott>	 !log tools depooling tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 for flavor update
[19:58:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:09:01] <andrewbogott>	 !log tools I didn't actually depool tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 because there was some kind of brief outage just now
[20:09:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:11:19] <wm-bot>	 !log tools.anomiebot <root> Started missing grid engine webservice job
[20:11:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.anomiebot/SAL
[20:21:41] <bd808>	 !log tools Restarted webservicemonitor on tools-sgecron-01.tools.eqiad.wmflabs
[20:21:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:34:27] <bd808>	 !log tools Live hacked "--backend=gridengine" into webservicemonitor on tools-sgecron-01 (T263190)
[20:34:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:34:31] <stashbot>	 T263190: webservicemontior failing to restart grid based webservices - https://phabricator.wikimedia.org/T263190
[21:55:06] <bd808>	 !log tools Built and deployed tools-manifest v0.22 (T169695)
[21:55:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:55:10] <stashbot>	 T169695: Run non-interactive commands on Toolforge kubernetes webservices - https://phabricator.wikimedia.org/T169695
[21:55:27] <bd808>	 doh. wrong task
[21:56:19] <bd808>	 !log tools Built and deployed tools-manifest v0.22 (T263190)
[21:56:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:56:22] <stashbot>	 T263190: webservicemontior failing to restart grid based webservices - https://phabricator.wikimedia.org/T263190