[00:15:14] <bstorm_>	 !log tools switch tools-acme-chief config to match the new authdns_servers format upstream
[00:15:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:27:06] <bstorm_>	 !log paws rebooting the paws master since it is in a bad state after the openstack maintenance as well.
[00:27:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[00:28:37] <bstorm_>	 !log tools rebooting the k8s master to clear NFS errors
[00:28:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:44:55] <bstorm_>	 !log tools rebooting tools-static-13
[00:45:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:39:03] <RickB>	 Hello, in the past day or so I've noticed that the map tiles I had been using that are served from https://tiles.wmflabs.org/osm-no-labels/${z}/${x}/${y}.png had started failing with 504 Timeout, is this server permanently offline or will it be coming back up?
[14:40:01] <RickB>	 e.g. "https://tiles.wmflabs.org/osm-no-labels/3/2/5.png"
[14:52:40] <RickB>	 !help
[14:52:40] <wm-bot>	 RickB: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[14:52:42] <jeh>	 Hi RickB, looks like the server behind that was effected by yesterdays maintenance. I'll restart that host (to release and refresh NFS)
[14:53:03] <RickB>	 that would be awesome!
[14:53:54] <jeh>	 !log maps restart maps-tiles1.maps.eqiad.wmflabs to resolve NFS issues after Dec 12th 2019 maintenance 
[14:53:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL
[14:57:29] <jeh>	 RickB: should be all set now
[14:58:16] <RickB>	 yes it is, thanks very much jeh!!
[15:00:21] <jeh>	 you're welcome, happy to help
[15:10:36] <CKoerner_WMF>	 @jeh We fixed our timeout issues with Wikimedia Space. It was something in Discourse that required a rebuild of Discourse. Thanks again for you help. 
[15:12:39] <jeh>	 Glad to hear you got it fixed, I see /blog is running smoothly again too :)
[17:32:10] <bstorm_>	 !log tools rebooting tools-k8s-control-2 to correct mount issue
[17:32:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:47:10] <bstorm_>	 !log tools edited kubeadm-config configMap object to match the new init config
[17:47:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:56:18] <bstorm_>	 !log tools updated tools-k8s-control-1 to the new control plane configuration
[17:56:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:46:27] <bstorm_>	 !log tools updated tools-k8s-control-2 and 3 to the new config as well
[18:46:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:14:35] <ottomata>	 heya, i'm having some problems with 2FA and wikitech & horizon
[20:15:05] <ottomata>	 i need to reset my 2FA tokens and app (phone has been wiped and restored)
[20:15:14] <ottomata>	 am following instructions at https://meta.wikimedia.org/wiki/Help:Two-factor_authentication#Disabling_two-factor_authentication
[20:15:27] <ottomata>	 but disable is not accepting my scratch codes
[20:16:58] <Zppix>	 ottomata:  you may have to have someone with server access disable it then
[20:16:59] <Zppix>	 IIRC
[20:17:10] <ottomata>	 i was just pointed to
[20:17:10] <ottomata>	 https://wikitech.wikimedia.org/wiki/Password_reset#For_users
[20:17:12] <ottomata>	 so am doing that
[20:17:13] <ottomata>	 ty
[20:17:41] <Zppix>	 np ottomata  I had to disable 2fa before since i lost my scratch codes before so I was just working off that knowledge