[09:01:16] <arturo>	 !log puppet-diffs updating puppetmaster1001 facts to all 3 compilers
[09:01:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs/SAL
[09:40:36] <Wurgl>	 https://tools.wmflabs.org/replag/ <-- 5 hours lag on enwiki? 7 hours on wikidata, others have 2 and 3 hours? What's going on here?
[09:52:45] <arturo>	 Wurgl: that could be related to T233766
[09:52:45] <stashbot>	 T233766: labsdb1011 mariadb crashed - https://phabricator.wikimedia.org/T233766
[09:59:14] <Wurgl>	 oops
[14:01:03] <wm-bot>	  Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @mutante - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[14:51:00] <wm-bot>	  Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @amir1 & @Lucas_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[15:02:16] <andrewbogott>	 !log deployment-prep moving deployment-dumps-puppetmaster02 to cloudvirt1021
[15:02:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[15:15:57] <andrewbogott>	 !log deployment-prep moving deployment-snapshot01 to cloudvirt1021
[15:16:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[15:49:04] <andrewbogott>	 Framawiki: I'm doing some routine cleanup/rebalancing and wondering about your VMs 'frama-test6-sb.sentry' and 'frama-test5.sentry'.  Did you shut them down yourself or did they spontaneously shut off?
[15:49:14] <andrewbogott>	 (They're on a host that's close to OOM, want to make sure the oom-killer didn't do in your VMs)
[15:55:11] <andrewbogott>	 !log integration moving integration-slave-jessie-1002 to cloudvirt1021
[15:55:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL
[16:19:25] <andrewbogott>	 !log integration moving integration-agent-docker-1009 and integration-agent-docker-1010 to cloudvirt1021
[16:19:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL
[17:27:27] <musikanimal>	 hello! are we aware of any maintenance on the replica dbs? starting about 19 hours ago, I'm seeing a lot of errors from my tools about queries timing out
[17:42:24] <musikanimal>	 I got a reply in -databases. T233766 seems to be the cuplrit
[17:42:25] <stashbot>	 T233766: labsdb1011 mariadb crashed - https://phabricator.wikimedia.org/T233766
[17:52:54] <andrewbogott>	 musikanimal: there was an issue with the replicas around then.  I'll look for documentation but bstorm_ may have a quick answer
[17:54:10] <musikanimal>	 yeah T233766 definitely seems related. They said that's probably why there's still replication lag (~9 hours on s1). I haven't seen that subside at all but I shall monitor and report back on that task if nothing improves
[17:54:10] <stashbot>	 T233766: labsdb1011 mariadb crashed - https://phabricator.wikimedia.org/T233766
[17:54:17] <bstorm_>	 It was the labsdb1011 crash
[17:54:23] <bstorm_>	 That was the issue that is
[17:54:23] <mutante>	 ores.wmflabs.org issues already known?
[17:54:35] <halfak>	 Yup.  Looking into it. 
[17:54:36] <halfak>	 Thank you. 
[17:54:40] <mutante>	 thx
[17:54:41] <halfak>	 mutante, ^ 
[17:54:44] <mutante>	 ack
[17:55:31] <andrewbogott>	 musikanimal: in case you missed it, T233766 is probably the cause of your issues
[17:56:04] <musikanimal>	 yes haha, I linked to it above. Thanks
[18:03:17] <halfak>	 !log ores restarted redis services on ores-redis-02
[18:03:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[18:12:55] <halfak>	 !log ores deleting AOF file on ores-redis-02
[18:12:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[18:13:41] <halfak>	 !log ores restarted redis services on ores-redis-02 (again)
[18:13:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[18:16:08] <halfak>	 \o/
[19:08:35] <andrewbogott>	 !log tools moving tools-sgewebgrid-lighttpd-0903 to cloudvirt1021
[19:08:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:50:04] <Zppix>	 !log git create new web auth creds for accraze for Icinga2
[20:50:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[23:01:18] <andrewbogott>	 !log deployment-prep moving deployment-mwmaint01 and deployment-ircd to cloudvirt1021
[23:01:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[23:30:04] <bd808>	 !log toolsbeta Granted Krenair projectadmin
[23:30:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[23:31:26] <bd808>	 !log toolsbeta Updated user list for "roots" sudoer policy
[23:31:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[23:31:49] <bd808>	 Krenair: I think you should have the power now
[23:31:57] <Krenair>	 yep, thanks