[00:31:15] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is CRITICAL: DISK CRITICAL - free space: / 10940 MB (5% inode=99%);
[00:35:17] <icinga-miraheze>	 RECOVERY - db5 Disk Space on db5 is OK: DISK OK - free space: / 40602 MB (22% inode=99%);
[00:51:57] <hispano76>	 paladox:  you could try this again to see if it's corrected, there's no rush if you can't  log root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages
[00:52:26] <hispano76>	 https://www.mediawiki.org/wiki/Manual:RecountCategories.php 
[00:52:27] <ZppixBot>	 [ Manual:recountCategories.php - MediaWiki ] - www.mediawiki.org
[00:52:34] <paladox>	 !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages
[00:52:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:52:51] <paladox>	 !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=subcats
[00:53:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:53:03] <paladox>	 !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=files
[00:53:06] <paladox>	 hispano76 done :)
[00:53:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:13:19] <hispano76>	 paladox: Oh, I didn't realize you also used this  root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php refreshLinks.php --wiki=ucroniaswiki required?
[01:13:43] <paladox>	 nope, not required. I just thought i try it per RhinosF1 
[01:13:48] <hispano76>	 curiosity
[01:14:02] <hispano76>	 ah, okay  thanks ´paladox 
[01:14:06] <paladox>	 yw
[02:23:42] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:23:58] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[02:25:38] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 0.502 second response time
[02:25:55] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[06:26:33] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3029 MB (12% inode=94%);
[07:08:18] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[07:09:33] <icinga-miraheze>	 PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[07:10:02] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[07:10:08] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[07:10:42] <icinga-miraheze>	 PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[07:10:46] <icinga-miraheze>	 PROBLEM - cp4 SSH on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:10:51] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[07:11:02] <icinga-miraheze>	 PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:11:03] <icinga-miraheze>	 PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:11:21] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[07:11:38] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[07:11:59] <icinga-miraheze>	 PROBLEM - netazar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:12:14] <icinga-miraheze>	 PROBLEM - Host cp4 is DOWN: PING CRITICAL - Packet loss = 100%
[07:12:41] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:13:02] <RhinosF1>	 paladox, PuppyKun, Reception123, SPF|Cloud: CP4 DOWN
[07:13:30] <MH-Discord>	 <RhinosF1> @Site Reliability Engineers
[07:15:00] <icinga-miraheze>	 RECOVERY - netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Tue 18 Feb 2020 07:52:57 PM GMT +0000.
[07:16:04] <SPF|Cloud>	 RhinosF1: server is suspended
[07:20:53] <RhinosF1>	 SPF|Cloud: :(
[07:21:03] <RhinosF1>	 I’ll guess bandwidth
[07:21:11] <RhinosF1>	 This is why we have a fundraiser
[07:22:43] <RhinosF1>	 SPF|Cloud: they said they wanted us to pay but Paladox got it restarted after a ticket last time
[07:27:34] <SPF|Cloud>	 Probably.
[07:28:13] <SPF|Cloud>	 With a bit of luck we won’t have to suffer anymore soon.
[07:30:12] <RhinosF1>	 SPF|Cloud: hopefully, we’re up to £1622 now + whatever’s gone in directly so over halfway
[07:30:49] <RhinosF1>	 It’ll get us the server and a good few months paid, just need to keep donations up.
[08:18:23] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[08:21:46] <paladox>	 Uh oh
[08:26:19] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:29:24] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[08:37:21] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:43:48] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[09:00:28] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:06:46] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[09:14:29] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:24:00] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[09:30:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYeP
[09:30:12] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 031e02f1c - BOT: Updating services config for wikis
[09:31:49] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:34:55] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[09:42:38] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:55:00] <SPF|Cloud>	 paladox: shall I ask RamNode why the server has been suspended?
[10:30:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYJ2
[10:30:11] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 038003b66 - BOT: Updating services config for wikis
[10:57:12] <paladox>	 SPF|Cloud: I mean you could, but I can already be 99.9% certain that it’s bandwidth
[10:59:55] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[11:07:40] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:10:45] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[11:18:17] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:40:40] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[11:48:09] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:48:15] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=94%);
[12:26:44] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[12:34:26] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:47:41] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[12:55:32] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:08:34] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[13:08:45] <ZppixBot>	 Hello AdrianBennetton! If you have any questions, feel free to ask and someone should answer soon.
[13:09:44] <AdrianBennetton>	 Hi We have Miraheze private wiki (pwiki.arkcls.com).  However it has been down all morning
[13:10:45] <Reception123>	 hi AdrianBennetton 
[13:10:50] <Reception123>	 I will take a look
[13:11:05] <AdrianBennetton>	 Thanks much appreciated
[13:11:49] <Reception123>	 AdrianBennetton: What do you mean by "down"? I am able to access it 
[13:12:51] <AdrianBennetton>	 When we try in our office we get "This site can't be reached".  I will try from my mobile
[13:13:29] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Toshinori "Agent Mace" --logwiki=metawiki
[13:13:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:13:51] <Reception123>	 AdrianBennetton: That is weird, but that seems to be a local problem rather than a problem on our side
[13:13:59] <Reception123>	 Please let me know if it works on mobile for you
[13:14:52] <AdrianBennetton>	 It works from my mobile using 4G network.  I will check that with our office systems admin as to why we can't access wiki from within our office
[13:15:16] <Reception123>	 AdrianBennetton: Okay, please do since it could perhaps be a local block 
[13:15:25] <AdrianBennetton>	 Yes it does look like it is a local problem
[13:15:31] <AdrianBennetton>	 Thanks for checking
[13:17:11] <Reception123>	 RhinosF1: hi. Don't know why you said I needed info from Owen since it looks like all I need to do is add a FUNDING.yaml?
[13:17:18] <Reception123>	 AdrianBennetton: Ok. You're welcome :)
[13:24:34] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:27:04] <PuppyKun>	 Reception123: iirc funding.yml is for the custom links, but you need GH approval or whatever for a proper donations button. maybe not though
[13:27:34] <Reception123>	 PuppyKun: ah ok, I didn't see that being mentioned
[13:27:40] <Reception123>	 But I'll do funding.yaml and see what happens next
[14:06:38] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[14:14:30] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:24:04] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[14:43:25] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:46:29] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[14:54:03] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:57:09] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[15:04:54] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:11:12] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[15:44:39] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:30:33] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[16:38:27] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:46:58] <RhinosF1>	 PuppyKun: GH Approval is needed for direct transfer to bank
[16:51:11] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[16:58:55] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:14:48] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[17:22:24] <RhinosF1>	 Reception123: 16:46:58 <RhinosF1> PuppyKun: GH Approval is needed for direct transfer to bank
[17:31:33] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:32:22] <Reception123>	 RhinosF1: oh ok
[17:32:35] <Reception123>	 RhinosF1: since you proposed it, could you maybe make a PR for FUNDS.yaml? Since I don't really know the syntax
[17:32:53] <RhinosF1>	 Reception123: see the twinkle repo
[17:32:56] <RhinosF1>	 It’s in their
[17:33:12] <RhinosF1>	 Then just turn sponsorships on in settings
[17:33:22] <Reception123>	 RhinosF1: it's already turned on though
[17:35:08] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvYcT
[17:35:10] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 039c8750b - add FUNDING.yaml Per T5114
[17:35:13] <RhinosF1>	 Reception123: then just copy the funding file from the twinkle repo to as many MH ones as possible
[17:35:31] <Reception123>	 oh my mad it's YML not YAML
[17:36:08] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/JvYcI
[17:36:09] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 036148781 - YAML->YML typo
[17:36:18] <Reception123>	 RhinosF1: it already seems to work...
[17:36:26] <Reception123>	 RhinosF1: https://github.com/miraheze/mw-config do you see the button?
[17:36:27] <ZppixBot>	 [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com
[17:40:05] <RhinosF1>	 Reception123: yeah working
[18:00:11] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYcd
[18:00:12] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03eb7d5bc - BOT: Updating services config for wikis
[18:00:59] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[18:08:42] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:48:49] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[18:56:52] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:32:38] <paladox>	 !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3001 --delete
[19:32:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:35:56] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[19:43:40] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:56:39] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[20:04:37] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:24:25] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[20:32:07] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:35:05] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[20:40:54] <Examknow>	 JohnLewis: You around?
[20:43:03] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:49:35] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[20:57:42] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:04:35] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[21:09:07] <JohnLewis>	 Examknow: not really, what's up?
[21:12:19] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:15:22] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[21:18:05] <hispano76>	 Has there been an example of integration of local Wikidata and modules into a Miraheze wiki? 
[21:23:32] <RhinosF1>	 hispano76: not that I know of
[21:26:06] <hispano76>	 It would be interesting to do this (because I would like the format to save bytes..)  :)
[21:27:15] <hispano76>	 By the way, what was the reason why Wikidata can't be used directly?
[21:27:22] <hispano76>	 curiosity*
[21:40:51] <RhinosF1>	 hispano76: broke extension I think
[21:41:17] <hispano76>	 ah, ok. thanks RhinosF1 
[21:42:07] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:48:12] <Zppix>	 paladox:  whats up with the ssl alerts
[21:48:56] <paladox>	 I'm not entirly sure, but the site works.
[21:49:01] <paladox>	 (hence why i didn't remove it)
[21:50:09] <RhinosF1>	 paladox: they started when cp4 failed
[21:50:21] <RhinosF1>	 Sims wiki was on as well
[21:50:22] <paladox>	 host enc.for.uz
[21:50:25] <paladox>	 points to cp4
[21:50:27] <paladox>	 err
[21:50:28] <paladox>	 cp2
[21:50:50] <RhinosF1>	 paladox: hmm, quite a few crashed before cp4 paged down
[21:51:25] <paladox>	 quote a few?
[21:51:49] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[21:52:05] <RhinosF1>	 paladox: whatever is in icinga
[21:52:26] <paladox>	 sims wiki is due to cp4
[21:52:33] <paladox>	 since they have cp4 hardcoded
[21:52:56] <Zppix>	 paladox:  can you compress ungamewiki?
[21:52:58] <RhinosF1>	 guiasdobrasil.com.br
[21:52:59] <paladox>	 guiasdobrasil.com.br  seems to no longer be pointing at us?
[21:53:13] <paladox>	 Zppix you just add ungamewiki to the config and run a maint script
[21:53:42] <Zppix>	 paladox:  im mobile atm
[21:53:48] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvY2t
[21:53:50] <Not-5a0a>	 [02miraheze/ssl] 07paladox 030457aba - Remove guiasdobrasil.com.br No longer pointing at us
[21:53:51] <Not-5a0a>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[21:53:53] <Not-5a0a>	 [02ssl] 07paladox opened pull request 03#263: Remove guiasdobrasil.com.br - 13https://git.io/JvY2q
[21:53:56] <paladox>	 oh
[21:53:57] <RhinosF1>	 paladox: look at logs from here when icinga paged cp4 and you’ll see
[21:54:26] <paladox>	 well seems recovered anyways for the ones your talking about
[21:54:58] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JvY23
[21:54:59] <Not-5a0a>	 [02miraheze/ssl] 07paladox 0387fc9d3 - Delete guiasdobrasil.com.br.crt
[21:55:01] <Not-5a0a>	 [02ssl] 07paladox synchronize pull request 03#263: Remove guiasdobrasil.com.br - 13https://git.io/JvY2q
[21:55:15] <Not-5a0a>	 [02ssl] 07paladox closed pull request 03#263: Remove guiasdobrasil.com.br - 13https://git.io/JvY2q
[21:55:16] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JvY2s
[21:55:17] <Not-5a0a>	 [02miraheze/ssl] 07paladox 033a1ddbc - Remove guiasdobrasil.com.br (#263) * Remove guiasdobrasil.com.br No longer pointing at us * Delete guiasdobrasil.com.br.crt
[21:55:19] <Not-5a0a>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[21:55:20] <Not-5a0a>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[21:55:21] <RhinosF1>	 paladox: well simswiki we know about and that you’ve deleted
[21:55:35] <RhinosF1>	 They went about 10s before cp4
[21:56:08] <RhinosF1>	 netazar.org went 1 min later
[21:56:29] <RhinosF1>	 enc.for.uz 2 mins after that
[21:58:26] <paladox>	 !log MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/guiasdobrasil.com.br "', null ) WHERE wiki_dbname = 'guiaslocaiswiki';
[21:58:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:59:19] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY20
[21:59:20] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 038f029b5 - Compress ungamewiki wiki
[21:59:40] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:00:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2u
[22:00:11] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 035cb4600 - BOT: Updating services config for wikis
[22:00:40] <paladox>	 !log root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=ungamewiki --type=gzip
[22:00:42] <paladox>	 Zppix done ^
[22:00:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:02:12] <paladox>	 i see the issue on misc1...
[22:02:34] <paladox>	 ok, it's using cp4 itermittently???
[22:02:48] <paladox>	 host enc.for.uz
[22:02:48] <paladox>	 enc.for.uz has address 81.4.109.133
[22:02:48] <paladox>	 enc.for.uz has IPv6 address 2604:180:0:33b::2
[22:02:48] <paladox>	 enc.for.uz mail is handled by 10 mx.yandex.net.
[22:02:53] <paladox>	 81.4.109.133 = cp4
[22:03:23] <RhinosF1>	 Ab
[22:03:24] <RhinosF1>	 Ah
[22:03:45] <Zppix>	 thanks paladox 
[22:04:48] * paladox depools cp4
[22:05:13] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2P
[22:05:15] <Not-5a0a>	 [02miraheze/dns] 07paladox 03453a193 - Depool cp4
[22:05:34] <RhinosF1>	 Thanks
[22:06:32] <travis-ci>	 miraheze/dns/master/453a193 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/643120732
[22:06:42] <paladox>	 well that didn't work... (stop it from using cp4)
[22:07:13] <MH-Discord>	 <RhinosF1> 😦
[22:07:30] <paladox>	 wtf
[22:07:42] <paladox>	 it's using a cp4 ipv4 address but cp2 ipv6 address
[22:07:52] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br]
[22:07:56] <RhinosF1>	 Strange
[22:08:00] <RhinosF1>	 Poor icinga-miraheze
[22:08:45] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2D
[22:08:46] <Not-5a0a>	 [02miraheze/ssl] 07paladox 03dea1dca - Remove www.guiasdobrasil.com.br
[22:08:48] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br]
[22:08:53] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br]
[22:09:01] <Zppix>	 lol
[22:09:26] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br]
[22:09:33] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br]
[22:09:44] <RhinosF1>	 Zppix: see discord for a TestWiki question in discrete
[22:10:16] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br]
[22:10:27] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br]
[22:10:48] <paladox>	 oh!
[22:11:20] <Zppix>	 RhinosF1:  im good with whatever void decides :D I'm a consul purely to help out, and i trust his judgement
[22:11:51] <paladox>	 it's itermittently using yandex, then it's using ours.
[22:12:10] <Zppix>	 paladox:  probably need to contact whoever manages the domain then
[22:12:20] <paladox>	 yup
[22:13:10] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[22:13:13] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:13:36] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:14:20] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:14:35] <MH-Discord>	 <RhinosF1> @Zppix: kk
[22:14:48] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:14:54] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:15:50] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:16:09] <paladox>	 https://phabricator.miraheze.org/T5174
[22:16:10] <ZppixBot>	 [ ⚓ T5174 enc.for.uz itermittently using both yandex and miraheze dns ] - phabricator.miraheze.org
[22:16:49] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2b
[22:16:50] <Not-5a0a>	 [02miraheze/dns] 07paladox 03e71e24f - Revert "Depool cp4" This reverts commit 453a1936afed13ba9750d3080dd52a2643dabd02.
[22:17:52] <Zppix>	 paladox:  maybe we should not allow people to hardcode what  cp* they are using?
[22:18:00] <travis-ci>	 miraheze/dns/master/e71e24f - Paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/643125670
[22:18:09] <paladox>	 well some have to unfournitly.
[22:18:29] <paladox>	 like simswiki (as they want access to the wiki on the main domain rather then a subdomain)
[22:18:49] <Zppix>	 paladox:  what do you mean?
[22:19:20] <paladox>	 Simswiki can only use an ip, they carn't use mw-lib. This is due to the use of the main domain.
[22:19:20] <Zppix>	 they cant do it to point to mw-lb's ip?
[22:19:30] <paladox>	 mw-lib does not have an ip
[22:19:48] <Zppix>	 a domain has to have a valid ip?
[22:20:26] <paladox>	 Zppix https://github.com/miraheze/dns/blame/master/zones/miraheze.org#L115
[22:20:27] <ZppixBot>	 [ dns/zones/miraheze.org at master · miraheze/dns · GitHub ] - github.com
[22:22:28] <Zppix>	 oh i see paladox
[22:22:35] <Zppix>	 maybe we could assign mw-lb an ip paladox ?
[22:22:51] <paladox>	 but an ip is pointless?
[22:23:03] <paladox>	 that means if that server goes down, things will go down which use it
[22:23:13] <Zppix>	 true
[23:14:23] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[23:27:41] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:30:44] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[23:36:31] <paladox>	 !log upgrade grafana - misc1
[23:36:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:41:28] <paladox>	 !log upgrade php - misc1
[23:41:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:42:31] <paladox>	 !log upgrase puppet-agent & salt on misc1
[23:42:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:42:45] <Zppix>	 paladox:  upgrase?
[23:43:01] <paladox>	 yes, there's updates available in apt for the packages
[23:43:15] <Zppix>	 paladox:  but you didnt update it you upgrased it :P
[23:43:22] <paladox>	 oh
[23:43:35] <paladox>	 fixed
[23:43:41] <Zppix>	 he
[23:44:35] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:45:31] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYw3
[23:45:33] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c4b846a - icingaweb2: upgrade to php 7.3
[23:47:23] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:50:53] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000.
[23:51:43] <icinga-miraheze>	 PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway
[23:51:46] <icinga-miraheze>	 PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.005 second response time
[23:52:07] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYwu
[23:52:08] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03f6be00d - roundcubemail: Upgrade php to 7.3
[23:53:32] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:53:33] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYwg
[23:53:34] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0354fe1da - roundcubemail: Update to 1.4.2
[23:54:46] <icinga-miraheze>	 RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 334 bytes in 0.009 second response time