[00:29:40] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:31:35] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[02:22:48] <Not-8d6a>	 [02IncidentReporting] 07JohnFLewis created branch 03master - 13https://git.io/fh5YJ
[02:22:50] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+17/-0/±0] 13https://git.io/fh5YU
[02:22:51] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis 03aa78ba5 - initial commit
[02:23:57] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+17/-0/±0] 13https://git.io/fh5YT
[02:23:59] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis 0355aeb61 - initial commit
[02:24:06] <travis-ci>	 miraheze/IncidentReporting/master/aa78ba5 - John Lewis The build passed. https://travis-ci.com/miraheze/IncidentReporting/builds/101171645
[02:24:24] <JohnLewis>	 paladox: ^ happy? :P
[02:26:41] <paladox>	 Someone pinged me?
[02:27:44] <paladox>	 JohnLewis: did you ping me? :D
[02:27:52] <JohnLewis>	 yeah
[02:28:00] <JohnLewis>	 https://git.io/fh5YJ
[02:28:38] <paladox>	 Nice!
[02:29:58] * paladox gets JohnLewis to use git repo instead of my db
[02:31:45] <JohnLewis>	 paladox: checking out Tech namespace, do we not have any docs for LizardFS?
[02:32:32] <paladox>	 JohnLewis: nope, I doint think so :(
[02:33:05] <JohnLewis>	 please make some
[02:35:55] <paladox>	 JohnLewis: I’m not sure what you would like in the docs
[02:36:53] <JohnLewis>	 how its set up, what it is, what it does, any key parts of it, anything we need to know, how to do certain things with it that we may need to
[02:37:46] <paladox>	 Ok
[02:39:14] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+17/-0/±0] 13https://git.io/fh5Yq
[02:39:16] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis 03258f31d - initial commit
[02:40:32] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5Ym
[02:40:34] <JohnLewis>	 paladox: ^ is what I asked :P
[02:40:34] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis 0344f640c - push basic IncidentReporting config
[02:41:29] <paladox>	 JohnLewis: asked?
[02:41:34] <JohnLewis>	 *why
[02:41:57] <paladox>	 Ok
[02:42:27] <paladox>	 JohnLewis: mind renewing one of the wild card certs? I need someone else to learn how to do that on the ops team
[02:42:34] <paladox>	 So I’m not the only one :)
[02:42:42] <JohnLewis>	 is it documented?
[02:43:06] <paladox>	 It should be
[02:43:16] <JohnLewis>	 if it then, then problem solved. Otherwise, no
[02:43:46] <paladox>	 Ok, make sure the docs look ok to you then :)
[02:44:43] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[02:47:10] <Not-8d6a>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5Y3
[02:47:12] <Not-8d6a>	 [02miraheze/puppet] 07JohnFLewis 035fecbc0 - add new DB grants
[02:51:51] <JohnLewis>	 !log created tables in incidents database
[02:51:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[03:00:13] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+1/-0/±1] 13https://git.io/fh5Yc
[03:00:15] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 0374e85b7 - add IR
[03:07:01] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5Y8
[03:07:03] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis 03f0ba5b9 - add IR
[03:34:02] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5YX
[03:34:04] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis 03135aff6 - add mDb
[03:34:25] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fh5Y1
[03:34:27] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 03224a962 - update IR
[03:41:00] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5Y9
[03:41:01] <Not-8d6a>	 [02miraheze/mw-config] 07JohnFLewis 038b7d82d - add editincidents to blacklist
[03:46:16] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fh5YN
[03:46:17] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis 03824a44a - fix permissions names
[03:47:03] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fh5YA
[03:47:04] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 039444063 - pull IR
[04:48:00] <khalella>	 how does miraheze implement google analytics? i don't see a HeadScript extension
[05:04:25] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:06:19] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[07:16:59] <Reception123>	 khalella: hi. We don't use Google Analytics, we use Matomo Analytics for our stats
[07:30:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5s3
[07:30:14] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03379e13c - BOT: Updating services config for wikis
[07:33:16] <Not-8d6a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/fh5ss
[07:33:18] <Not-8d6a>	 [02miraheze/ssl] 07Reception123 032431409 - ripto.gq -> wiki.ripto.gq cert
[07:55:09] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5sa
[07:55:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03b745cd4 - BOT: Updating services config for wikis
[08:40:07] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[08:42:07] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[08:47:59] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[08:48:07] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[09:27:59] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[09:28:07] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[09:31:59] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[09:32:07] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[09:35:59] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[09:36:07] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[09:42:07] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[09:44:07] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[09:54:07] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[09:54:52] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[10:08:46] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:10:50] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[11:29:56] <icinga-miraheze>	 PROBLEM - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bh.gyaanipedia.co.in' expires in 15 day(s) (Mon 04 Mar 2019 11:27:05 AM GMT +0000).
[11:30:16] <Not-8d6a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5ns
[11:30:18] <Not-8d6a>	 [02miraheze/ssl] 07MirahezeSSLBot 03e424dc9 - Bot: Update SSL cert for bh.gyaanipedia.co.in
[11:31:56] <icinga-miraheze>	 RECOVERY - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'bh.gyaanipedia.co.in' will expire on Fri 17 May 2019 10:30:09 AM GMT +0000.
[11:37:59] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[11:38:07] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[11:41:59] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[11:42:07] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[12:30:32] <Reception123>	 !log running wikibackups (./test.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php)
[12:30:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:25:10] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5CQ
[14:25:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 032b2dafe - BOT: Updating services config for wikis
[15:43:59] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[15:44:07] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:00:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5lF
[17:00:14] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 0378c5790 - BOT: Updating services config for wikis
[17:53:00] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:54:56] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 07 May 2019 10:15:37 PM GMT +0000.
[20:06:40] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 43365 MB (11% inode=95%);
[23:08:26] <ZppixBot>	 Hello heinezen! If you have any questions feel free to ask and someone should answer soon.
[23:20:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5ub
[23:20:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 0380dc5c9 - BOT: Updating services config for wikis