[08:36:27] <andrewbogott>	 Cyberpower678: It looks like your cloudvirt host recovered overnight — is your db host looking ok today?
[09:22:07] <arturo>	 !log tools restart maintain-kubeusers service in tools-k8s-master-01 because some tools were missing their namespaces
[09:22:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:32:19] <arturo>	 !log ign2commons delete failed `ign` virtual machine
[10:32:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ign2commons/SAL
[10:56:19] <arturo>	 !log ign2commons created `ign` virtual machine again
[10:56:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ign2commons/SAL
[11:05:25] <arturo>	 !log ign2commons applied the `role::labs::lvm::srv` puppet role to the `ign` virtual machine for using the extra storage
[11:05:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ign2commons/SAL
[12:18:17] <arturo>	 Cyberpower678, andrewbogott cloudvirt1018 is back to high load. We should definitely move that DB virtual machine
[12:18:29] <Cyberpower678>	 :(
[12:18:41] <arturo>	 https://grafana.wikimedia.org/d/aJgffPPmz/wmcs-openstack-eqiad1-hypervisor?orgId=1&var-hypervisor=cloudvirt1018&refresh=30s&from=now-3h&to=now
[12:49:33] <andrewbogott>	 I'll set an alarm to remind me to move it in a couple hours
[13:13:28] <xinbenlv>	 Hi
[13:13:30] <arturo>	 o/
[13:13:34] <xinbenlv>	 nice to meet you
[13:24:51] <arturo>	 xinbenlv: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas
[14:59:44] <andrewbogott>	 Cyberpower678: Is now an ok time for me to start the db server migration?
[15:05:43] * andrewbogott starts the move
[15:05:57] <andrewbogott>	 !log cyberbot moving cyberbot-db-01 to  new cloudvirt
[15:05:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cyberbot/SAL
[15:07:33] <andrewbogott>	 eta, between 2 and 11 hours :/
[15:13:12] <Cyberpower678>	 andrewbogott: I came here to complain but totally missed this ping.  Stupid mute button. :p
[15:13:15] <Cyberpower678>	 ol
[15:13:16] <Cyberpower678>	 LOl
[15:13:46] <andrewbogott>	 Ok if I leave it copying?
[15:13:55] <Cyberpower678>	 What's the current ETA?
[15:15:01] <Cyberpower678>	 andrewbogott: ^
[15:18:29] <andrewbogott>	 Cyberpower678: looks like it errored out, so I'll wait and retry later in the evening.
[15:18:49] <andrewbogott>	 it should be back up in the meantime.
[15:18:51] * andrewbogott going afk
[15:32:52] <jeh>	 !log tools upgraded jobutils debian package to 1.38 T229551
[15:32:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:32:56] <stashbot>	 T229551: Database-reports can't see packages in its virtualenv on the grid - https://phabricator.wikimedia.org/T229551
[15:47:31] <Cyberpower678>	 andrewbogott: did you ever happen to notice absolutely insane IO loads on a weekend ever since you moved my DB VM?
[20:35:30] <andrewbogott>	 !log cyberbot moving cyberbot-db-01 to  new cloudvirt (another attempt)
[20:35:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cyberbot/SAL