[02:39:35] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1441 MB (5% inode=94%);
[02:41:38] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1455 MB (6% inode=94%);
[02:52:24] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:54:22] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.390 second response time
[02:56:05] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1445 MB (5% inode=94%);
[06:27:08] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2399 MB (9% inode=94%);
[07:10:59] <SantaRhino>	 paladox, Zppix: please ping when around
[07:13:23] <paladox>	 SantaRhino: ?
[07:17:38] <SantaRhino>	 paladox: you’re up early but you missed a restart of redis in the incident report
[07:17:45] <SantaRhino>	 Reception restarted it twice
[07:27:39] <paladox>	 Ok, i see you wrote it on task. So not sure why I needed to be pinged here too?
[07:27:54] <paladox>	 Also what restart?
[07:28:13] <SantaRhino>	 paladox: because someone needs to fix it
[07:28:24] <SantaRhino>	 and on task I didn't say what
[07:30:10] <paladox>	 Well I’m not sure what restart your talking about
[07:30:26] <paladox>	 As I wasent around to deal with the incident.
[07:30:36] <SantaRhino>	 .mh meta Tech:SAL
[07:30:36] <ZppixBot>	 https://meta.miraheze.org/wiki/Tech:SAL
[07:30:50] <SantaRhino>	 paladox: ^
[07:30:57] <SantaRhino>	 it'll tell you
[07:31:10] <paladox>	 That’s not helpful as I see many restarts
[07:31:13] <SantaRhino>	 07:46 and 08:26
[07:33:52] <paladox>	 Ok
[07:33:53] <paladox>	 I’m currently mobile 
[07:36:55] <SantaRhino>	 paladox: k, Also, probably would have taken a lot longer if it hadn't of been for me paging everyone on Discord/IRC and then helping reception debug over PM and us agreeing things can't get worse with a restart and memory errors will likely fix with a restart although reception might be best to put that.
[07:41:08] <paladox>	 Reception123: want to take care of ^? ;)
[07:42:18] <SantaRhino>	 paladox: Reception123 isn't around while later and said ask you
[07:44:26] <paladox>	 Oh
[07:44:27] <paladox>	 I’m mobile and will be till the afternoon
[07:44:54] <SantaRhino>	 ok
[07:45:17] <Reception123>	 Same for me, so I guess I'll just do it later since it's the same thing then 
[07:48:10] <paladox>	 The times seem to be wrong but I think that’s due to me being a bad reviewer :=
[07:48:14] <paladox>	 *:P
[07:51:41] <paladox>	 (Didn’t catch that error)
[07:52:20] <SantaRhino>	 paladox: It's not just you 3 ops staff and an mw-admin have reviewed or edited that report. It just is annoying none of you can read a SAL that's there to be read.
[07:54:36] <paladox>	 Uh?
[07:56:09] <SantaRhino>	 paladox: Reviewed: Paladox, Southparkfan, John. Not Reviewed: Reception123, NDKilla. + Zppix edited it. That's 3 ops and an mw-admin that can't review properly and read SAL
[07:59:50] <paladox>	 Ok
[08:19:25] <paladox>	 SantaRhino: Doint forget we are only human
[08:20:52] <SantaRhino>	 paladox: I know, but it should be a basic checklist. Is everything on the SAL during the incident logged? Does what's logged match SAL?
[08:21:18] <SantaRhino>	  to me that means that review = ops have working mouses
[08:23:26] <paladox>	 Basic checklist?
[08:23:27] <paladox>	 We do make mistakes
[08:24:23] <paladox>	 And well it was correct
[08:24:29] <SantaRhino>	 paladox: yes, it stood out like a saw thumb. If you're reviewing, review it fully not glance.
[08:24:43] <paladox>	 Log wise, reception restarted redis twice
[08:24:56] <SantaRhino>	 paladox: I told you that
[08:25:05] <paladox>	 Uh
[08:25:36] <paladox>	 I mean on the incident report, there’s one mention of reception restarting redis
[08:25:50] <SantaRhino>	 paladox: that's what I've raised
[08:25:54] <SantaRhino>	 there should be two
[08:26:19] <paladox>	 Yes I know
[08:26:55] <SantaRhino>	 paladox: and you've said yourself that times may be wrong as you're a bad reviewer.
[08:27:08] <paladox>	 Yes that was a joke
[08:27:19] <paladox>	 You know it was a joke, I didn’t look at the times
[08:27:54] <SantaRhino>	 paladox: incident reports need to be accurate, they're showing the public what went wrong.
[08:28:07] <paladox>	 Ok
[08:28:07] <SantaRhino>	 not funny when I'm complaining about accuracy
[08:29:54] <paladox>	 Uh, no your being rude. Rather then constructively criticising us which would allow us to improve as we’ve gained constructive feedback, your offending us. Saying “we carnt read the logs”.
[08:31:20] <SantaRhino>	 paladox: make a checklist of things to look at before publishing reports. SAL should be a key one. (Surprisingly, I used it for reports I wrote)
[08:32:14] <paladox>	 Ok, maybe someone could create that on wiki for reviewers?
[08:32:43] <SantaRhino>	 paladox: Let me eat breakfast and I'll email a draft to staff@
[08:32:56] <paladox>	 SantaRhino: if you want to create an on-wiki guide for reviewers, it would be appreciated (what you just said above)
[08:33:36] <SantaRhino>	 paladox: I will
[08:33:42] <paladox>	 Thanks!
[08:33:49] <SantaRhino>	 but I'll get someone to check it first!
[08:35:23] <paladox>	 Ok
[09:16:22] <SantaRhino>	 paladox: email sent so staff can review
[11:47:26] <paladox>	 SantaRhino: thanks!
[14:15:57] <Not-5a0a>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5iG
[14:15:59] <Not-5a0a>	 [02miraheze/CreateWiki] 07translatewiki 03872ca41 - Localisation updates from https://translatewiki.net.
[14:15:59] <ZppixBot>	 [ Main page - translatewiki.net ] - translatewiki.net.
[14:16:00] <Not-5a0a>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/Je5iZ
[14:16:02] <Not-5a0a>	 [02miraheze/ManageWiki] 07translatewiki 03b95d9e8 - Localisation updates from https://translatewiki.net.
[14:16:02] <ZppixBot>	 [ Main page - translatewiki.net ] - translatewiki.net.
[14:54:53] <SantaRhino>	 paladox: 1.34!!!
[15:01:39] <Reception123>	 It's finally out?
[15:02:20] <SantaRhino>	 Reception123: yep, git tag is coming shortly
[15:19:32] * hispano76 greeting
[15:19:51] <SantaRhino>	 Hi
[15:49:20] <paladox>	 Yup
[15:49:43] <SantaRhino>	 paladox: you doing it when git tag comes?
[15:51:02] <SantaRhino>	 paladox: https://github.com/wikimedia/mediawiki/releases/tag/1.34.0!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[15:54:55] <paladox>	 No, may do it later
[15:55:34] <SantaRhino>	 ok
[15:56:37] <Reception123>	 when a steward is around please let them know they can lock https://meta.miraheze.org/wiki/Special:CentralAuth/Meta_Wiki
[15:56:39] <ZppixBot>	 [ Global account information for Meta Wiki - Miraheze Meta ] - meta.miraheze.org
[15:56:47] <Reception123>	 as there hasn't been an answer in a reasonable amount of time
[16:14:14] <SantaRhino>	 JohnLewis: IncidentReporting is buggy, things aren't in the right order when times were changed
[16:15:34] <Reception123>	 JohnLewis: hi. Could you please lock https://meta.miraheze.org/wiki/Special:CentralAuth/Meta_Wiki for violation of the username policy?
[16:15:35] <ZppixBot>	 [ Global account information for Meta Wiki - Miraheze Meta ] - meta.miraheze.org
[16:15:40] <Reception123>	 They've been warned and not reacted in quite a while 
[16:17:18] <JohnLewis>	 SantasRhinos: noted, file a bug - but logs should flow in order from start to finish and not be disjointed
[16:18:01] <SantaRhino>	 JohnLewis: that's the bug, when the time is changed. They don't go to the right order
[16:19:02] <JohnLewis>	 But my point is it should be down in the right order originally 
[16:19:30] <JohnLewis>	 Because the constructor form is done based on ID not time 
[16:20:49] <SantaRhino>	 JohnLewis: but people make mistakes and forget to add logs
[16:21:00] <SantaRhino>	 (see my ranty staff@ email)
[16:21:07] <SantaRhino>	 https://phabricator.miraheze.org/T5001
[16:21:08] <ZppixBot>	 [ ⚓ T5001 IncidentReporting doesn't reorder logs when time is changed. ] - phabricator.miraheze.org
[16:21:24] <JohnLewis>	 And they should fix it as such - accuracy from the off is imperative for a public facing reporting mechanism 
[16:22:00] <SantaRhino>	 JohnLewis: I've already gone on a large rant at paladox over that this morning
[16:22:21] <JohnLewis>	 Though re the report, who wrote it and who dealt with the incident?
[16:22:48] <SantaRhino>	 JohnLewis: not sure who wrote it, me and reception, then spf, then paladox then you
[16:23:04] <SantaRhino>	 I helped debug the log and work out what the immediate fix could be
[16:23:36] <SantaRhino>	 you did review the report though
[16:23:59] <JohnLewis>	 Reception should have started the report and checked the accuracy of the part he dealt with before it went live
[16:24:15] <SantaRhino>	 JohnLewis: reception did neither
[16:24:21] <SantaRhino>	 afaik
[16:24:30] <JohnLewis>	 It’s a principles of ownership, plus all debugging and incident communication should be public, not private 
[16:24:40] <JohnLewis>	 which would potentially account to you being given no credit
[16:24:44] <Reception123>	 I didn't write the rpeort 
[16:24:46] <Reception123>	 *report
[16:25:04] <Reception123>	 neither did I edit nor review until right now
[16:25:07] <JohnLewis>	 That’s the issue, Reception should have if he dealt with the incident 
[16:25:13] <JohnLewis>	 or all communication should have been public 
[16:25:23] <SantaRhino>	 JohnLewis: yeah which is why paladox didn't get shouted at over that. only the poor listing of logs
[16:25:32] <SantaRhino>	 I get you and fully agree
[16:26:06] <JohnLewis>	 I do get logs is indeed an issue there and they should have been thoroughly checked by Paladox when making the timeline 
[16:26:54] <SantaRhino>	 JohnLewis: I went mental over the fact 3 of you reviewed it and didn't spot it. I know.
[16:26:55] <JohnLewis>	 Recetion123: the report shouldn’t have been live without you reviewing it if you were involved.
[16:27:16] <JohnLewis>	 Reviewed means we reviewed the contents, not accuracy 
[16:27:30] <JohnLewis>	 accuracy isn’t our job, it’s the job of the report writer(a)
[16:27:31] <Reception123>	 Yes, what I did not get is why it was made public
[16:27:46] <SantaRhino>	 ^^^^
[16:27:48] <Reception123>	 The rule was always that it isn't made public until we are sure of accuracy 
[16:28:06] <SantaRhino>	 defo
[16:28:12] <Reception123>	 It was made public without me even having read it
[16:30:08] <JohnLewis>	 Indeed
[16:30:42] <JohnLewis>	 Zppix made it public, so remind him :)
[16:30:56] <SantaRhino>	 Zppix: :(
[16:31:10] <SantaRhino>	 don't do that again
[16:31:20] <Reception123>	 Zppix: yes, please see the policy on staffwiki
[16:32:24] <Reception123>	 JohnLewis: https://meta.miraheze.org/wiki/Requests_for_global_rights#Se.27s_Request_for_CTV mind a speedy close? the user has enough blocks to qualify for a lock and they are requesting 'CTV', a right that doesn't exist 
[16:32:26] <ZppixBot>	 [ Requests for global rights - Miraheze Meta ] - meta.miraheze.org
[16:32:36] <Reception123>	 they're the last person to get any sort of global rights
[16:39:48] <ShakespeareFan00>	 pinging JohnLewis?
[16:40:25] <JohnLewis>	 Hey
[16:41:03] <ShakespeareFan00>	 Been reading some stuff on your meta..
[16:41:18] <ShakespeareFan00>	 I can't discuss it an open channel, but a PM or query is fine
[16:41:22] <JohnLewis>	 Mhm?
[16:41:32] <JohnLewis>	 Sure
[16:42:19] <Reception123>	 JohnLewis: please also see my pings if possible :)
[16:44:05] <JohnLewis>	 Reception123: did the account, will look at the CTV request :P
[16:46:06] <Reception123>	 ok
[16:46:11] <Reception123>	 thanks
[16:46:31] <Reception123>	 maybe a new right was created and I'm not aware, I'll go apply for it :P
[16:54:12] <JohnLewis>	 Reception123: yeah, it's the team which makes vandalism - Causes of Terrible Vandalism (CTV)
[16:54:36] <Reception123>	 ah, in that case I fit perfectly into that new role 
[16:54:46] <SantaRhino>	 :)
[16:54:50] <Reception123>	 I'll resign CVT since it's a conflict of interest and enrol in this new role 
[17:25:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5Mh
[17:25:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 033188d88 - BOT: Updating services config for wikis
[18:27:11] <Not-5a0a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Je5yR
[18:27:12] <Not-5a0a>	 [02miraheze/mediawiki] 07paladox 03a2ea871 - Update VisualEditor
[18:28:29] <Not-5a0a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/Je5yz
[18:28:30] <Not-5a0a>	 [02miraheze/mediawiki] 07paladox 03a7896d3 - Update VisualEditor
[18:39:24] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[18:44:29] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 35405 MB (9% inode=95%);
[18:56:16] <hispano76>	 hello
[18:57:31] <hispano76>	  do you understand this translation?  "GeoCumbs must be configured to be used in name spaces other than the main space"  is for an request at Phabricator Wikimedia 
[19:00:22] <Zppix>	 hispano76:  that makes sense yes
[19:01:28] <Zppix>	 hispano76:  even if it doesn't im sure they will be like me and understand what you meant, WMF developer community has people that are capable of understanding people that don't speak english or are ESL
[19:07:07] <hispano76>	 ok, thanks Zppix 
[19:07:54] <Zppix>	 hispano76:  no problem, anytime!
[19:13:04] <Zppix>	 !log sudo -u www-data php sql.php --wiki=metawiki --query="update cw_requests set cw_status = 'approved' where cw_id = 10227;"
[19:13:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:33:08] <Not-5a0a>	 [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_33 [+0/-0/±4] 13https://git.io/Je59v
[19:33:10] <Not-5a0a>	 [02miraheze/mediawiki] 07Pix1234 036bbf33b - update various ext from upstream
[19:39:17] <Zppix>	 !log rebuild lc on mw123
[19:39:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:40:22] <paladox>	 !log rebuild lc on lizardfs6
[19:40:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:47:47] <Not-5a0a>	 [02puppet] 07Pix1234 opened pull request 03#1163: Add puppet last run to MOTD - 13https://git.io/Je59B
[19:50:20] <Not-5a0a>	 [02puppet] 07Pix1234 synchronize pull request 03#1163: Add puppet last run to MOTD - 13https://git.io/Je59B
[19:51:40] <Not-5a0a>	 [02puppet] 07paladox reviewed pull request 03#1163 commit - 13https://git.io/Je59E
[19:52:00] <Not-5a0a>	 [02puppet] 07paladox reviewed pull request 03#1163 commit - 13https://git.io/Je59u
[19:53:35] <Not-5a0a>	 [02puppet] 07Pix1234 synchronize pull request 03#1163: Add puppet last run to MOTD - 13https://git.io/Je59B
[19:54:24] <Not-5a0a>	 [02puppet] 07paladox reviewed pull request 03#1163 commit - 13https://git.io/Je59a
[19:55:33] <Not-5a0a>	 [02puppet] 07Pix1234 synchronize pull request 03#1163: Add puppet last run to MOTD - 13https://git.io/Je59B
[19:56:04] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1163: Add puppet last run to MOTD - 13https://git.io/Je59B
[19:56:21] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1163: Add puppet last run to MOTD - 13https://git.io/Je59B
[19:56:23] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Je59r
[19:56:24] <Not-5a0a>	 [02miraheze/puppet] 07Pix1234 03b7c5e0e - Add puppet last run to MOTD (#1163) * Add puppet last run to MOTD * Create 97-last-puppet-run * fix file locations * yamll -> yaml * Update puppet.pp
[20:36:28] <paladox>	 !log rebuild lc on mw2
[20:36:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:39:18] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[22:06:13] <icinga-miraheze>	 RECOVERY - glp.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'glp.ink' will expire on Mon 20 Jan 2020 04:34:07 PM GMT +0000.
[22:11:00] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb
[22:16:03] <icinga-miraheze>	 PROBLEM - www.guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:16:04] <icinga-miraheze>	 PROBLEM - interno.guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:16:40] <icinga-miraheze>	 PROBLEM - theatlas.pw - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:17:08] <icinga-miraheze>	 PROBLEM - disabled.life - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:17:32] <icinga-miraheze>	 PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:18:21] <icinga-miraheze>	 PROBLEM - nonlinearly.com - CloudFlare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:18:57] <SantaRhino>	 paladox: ^
[22:19:30] <icinga-miraheze>	 PROBLEM - vmcodex.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:20:31] <icinga-miraheze>	 PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:21:35] <icinga-miraheze>	 PROBLEM - wiki.udia.ca - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:22:11] <paladox>	 looking
[22:22:22] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:22:33] <icinga-miraheze>	 PROBLEM - www.eerstelijnszones.be - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:23:45] <icinga-miraheze>	 PROBLEM - meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:24:16] <icinga-miraheze>	 PROBLEM - yellowiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:24:28] <paladox>	 curl reviwiki.info -I works for me
[22:24:32] <paladox>	 so that is strange
[22:26:13] <SantaRhino>	 paladox: could something be impacting icinga’s connection?
[22:26:19] <SantaRhino>	 Site is up fine
[22:26:34] <paladox>	 I wouldn't think so, i can curl it fine
[22:26:38] <paladox>	 icinga uses a script
[22:26:42] <paladox>	 for check_http
[22:26:49] * SantaRhino needs to sleep
[22:27:19] * SantaRhino wishes paladox luck fixing it
[22:27:30] <icinga-miraheze>	 PROBLEM - www.schulwiki.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:27:40] <paladox>	 works now
[22:27:47] <paladox>	 (icinga)
[22:27:50] <icinga-miraheze>	 PROBLEM - vsfan.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:28:10] <SantaRhino>	 paladox: k weird, have an eye out
[22:28:22] <icinga-miraheze>	 RECOVERY - meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meregos.com' will expire on Fri 24 Jan 2020 10:15:38 AM GMT +0000.
[22:28:52] <SantaRhino>	 https://icinga.miraheze.org/monitoring/list/services?service_problem=1&sort=service_severity&dir=desc has 53 issues but could be still recovering if so paladox
[22:28:53] <ZppixBot>	 [ Icinga Web 2 Login ] - icinga.miraheze.org
[22:29:00] <SantaRhino>	 51
[22:29:03] <paladox>	 i am aware :)
[22:29:20] <SantaRhino>	 Night
[22:30:31] <icinga-miraheze>	 RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Wed 26 Feb 2020 06:22:14 PM GMT +0000.
[22:30:38] <icinga-miraheze>	 RECOVERY - theatlas.pw - LetsEncrypt on sslhost is OK: OK - Certificate 'theatlas.pw' will expire on Wed 29 Jan 2020 12:56:59 PM GMT +0000.
[22:32:13] <icinga-miraheze>	 RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Thu 30 Jan 2020 09:54:37 AM GMT +0000.
[22:33:00] <paladox>	 seems to be recovering now
[22:33:08] <paladox>	 !log restarted networking on misc1
[22:33:10] <icinga-miraheze>	 PROBLEM - wiki.contraao.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:33:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:33:56] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[22:34:16] <icinga-miraheze>	 PROBLEM - netazar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:35:01] <paladox>	 !log reboot misc1
[22:36:33] <icinga-miraheze>	 RECOVERY - wiki.contraao.com - LetsEncrypt on sslhost is OK: OK - Certificate 'contraao.com' will expire on Thu 23 Jan 2020 11:56:16 PM GMT +0000.
[22:36:43] <icinga-miraheze>	 PROBLEM - mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:36:48] <paladox>	 !log [22:35:01]  <+paladox>	!log reboot misc1
[22:37:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:37:49] <icinga-miraheze>	 RECOVERY - www.schulwiki.de - LetsEncrypt on sslhost is OK: OK - Certificate 'www.schulwiki.de' will expire on Tue 14 Jan 2020 06:42:50 PM GMT +0000.
[22:40:43] <icinga-miraheze>	 RECOVERY - vsfan.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vsfan.net' will expire on Sun 26 Jan 2020 01:28:19 PM GMT +0000.
[22:43:22] <icinga-miraheze>	 RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Fri 10 Jan 2020 11:06:58 AM GMT +0000.
[22:45:27] <icinga-miraheze>	 RECOVERY - netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Tue 18 Feb 2020 07:52:57 PM GMT +0000.
[22:46:14] <icinga-miraheze>	 PROBLEM - wikipuk.cl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:47:27] <icinga-miraheze>	 PROBLEM - vmcodex.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:47:40] <icinga-miraheze>	 PROBLEM - www.programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:49:53] <icinga-miraheze>	 RECOVERY - www.eerstelijnszones.be - LetsEncrypt on sslhost is OK: OK - Certificate 'eerstelijnszones.be' will expire on Fri 17 Jan 2020 11:50:22 AM GMT +0000.
[22:50:15] <icinga-miraheze>	 PROBLEM - panadev.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:51:26] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:51:33] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:52:15] <icinga-miraheze>	 PROBLEM - www.sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:53:51] <icinga-miraheze>	 RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Thu 30 Jan 2020 09:54:37 AM GMT +0000.
[22:54:03] <icinga-miraheze>	 RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sat 29 Feb 2020 04:34:39 PM GMT +0000.
[22:56:09] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:58:16] <icinga-miraheze>	 RECOVERY - www.sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sun 12 Jan 2020 07:44:31 PM GMT +0000.
[22:58:50] <icinga-miraheze>	 RECOVERY - wikipuk.cl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipuk.cl' will expire on Thu 23 Jan 2020 09:57:16 AM GMT +0000.
[23:00:21] <icinga-miraheze>	 PROBLEM - en.gyaanipedia.co.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:01:50] <icinga-miraheze>	 PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:02:34] <icinga-miraheze>	 PROBLEM - or.gyaanipedia.co.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:02:34] <icinga-miraheze>	 RECOVERY - panadev.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'panadev.ir' will expire on Wed 11 Mar 2020 11:47:06 AM GMT +0000.
[23:03:03] <icinga-miraheze>	 RECOVERY - disabled.life - LetsEncrypt on sslhost is OK: OK - Certificate 'disabled.life' will expire on Thu 23 Jan 2020 10:01:30 AM GMT +0000.
[23:03:06] <icinga-miraheze>	 RECOVERY - yellowiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'yellowiki.xyz' will expire on Mon 02 Mar 2020 05:03:40 AM GMT +0000.
[23:03:11] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[23:05:19] <icinga-miraheze>	 RECOVERY - en.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000.
[23:06:15] <icinga-miraheze>	 RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Fri 17 Jan 2020 11:59:53 AM GMT +0000.
[23:07:01] <icinga-miraheze>	 RECOVERY - or.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Mon 13 Jan 2020 02:17:34 PM GMT +0000.
[23:12:37] <icinga-miraheze>	 RECOVERY - udc-japan.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'udc-japan.xyz' will expire on Sun 09 Feb 2020 12:03:48 AM GMT +0000.
[23:13:15] <icinga-miraheze>	 PROBLEM - drag.lgbt - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:13:29] <icinga-miraheze>	 PROBLEM - froggy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:13:53] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Sat 11 Jan 2020 06:16:06 PM GMT +0000.
[23:18:20] <icinga-miraheze>	 RECOVERY - drag.lgbt - LetsEncrypt on sslhost is OK: OK - Certificate 'drag.lgbt' will expire on Wed 29 Jan 2020 12:32:13 PM GMT +0000.
[23:18:34] <icinga-miraheze>	 RECOVERY - froggy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'froggy.info' will expire on Thu 23 Jan 2020 10:14:02 AM GMT +0000.
[23:19:00] <paladox>	 !log switch off TUN/TAP on misc1
[23:19:10] <icinga-miraheze>	 PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:20:57] <icinga-miraheze>	 PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:23:34] <icinga-miraheze>	 PROBLEM - mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:25:27] <icinga-miraheze>	 RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 29 Jan 2020 12:32:42 PM GMT +0000.
[23:26:30] <icinga-miraheze>	 PROBLEM - vsfan.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:27:06] <icinga-miraheze>	 RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Fri 17 Jan 2020 11:59:53 AM GMT +0000.
[23:27:22] <icinga-miraheze>	 PROBLEM - garrettcountyguide.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:28:14] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:29:26] <icinga-miraheze>	 RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Fri 10 Jan 2020 11:06:58 AM GMT +0000.
[23:32:13] <icinga-miraheze>	 RECOVERY - vsfan.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vsfan.net' will expire on Sun 26 Jan 2020 01:28:19 PM GMT +0000.
[23:33:12] <icinga-miraheze>	 RECOVERY - garrettcountyguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'garrettcountyguide.com' will expire on Fri 24 Jan 2020 10:13:59 AM GMT +0000.
[23:33:38] <icinga-miraheze>	 RECOVERY - udc-japan.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'udc-japan.xyz' will expire on Sun 09 Feb 2020 12:03:48 AM GMT +0000.
[23:36:46] <icinga-miraheze>	 PROBLEM - panadev.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:38:20] <icinga-miraheze>	 PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:40:20] <icinga-miraheze>	 PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:40:32] <icinga-miraheze>	 PROBLEM - wikipuk.cl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:42:08] <icinga-miraheze>	 PROBLEM - madgenderscience.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:42:30] <icinga-miraheze>	 RECOVERY - panadev.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'panadev.ir' will expire on Wed 11 Mar 2020 11:47:06 AM GMT +0000.
[23:45:03] <icinga-miraheze>	 RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Fri 10 Jan 2020 01:06:00 PM GMT +0000.
[23:45:15] <icinga-miraheze>	 RECOVERY - wikipuk.cl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipuk.cl' will expire on Thu 23 Jan 2020 09:57:16 AM GMT +0000.
[23:46:31] <icinga-miraheze>	 PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:48:33] <icinga-miraheze>	 RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Thu 23 Jan 2020 10:01:16 AM GMT +0000.
[23:53:28] <icinga-miraheze>	 PROBLEM - disabled.life - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:54:22] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:54:24] <icinga-miraheze>	 PROBLEM - programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:54:37] <icinga-miraheze>	 PROBLEM - yellowiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:54:51] <icinga-miraheze>	 PROBLEM - vmcodex.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:57:02] <paladox>	 Network issues on misc1 ^
[23:57:02] <icinga-miraheze>	 RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Fri 24 Jan 2020 10:06:59 AM GMT +0000.
[23:57:05] <icinga-miraheze>	 RECOVERY - madgenderscience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madgenderscience.wiki' will expire on Thu 23 Jan 2020 10:14:59 AM GMT +0000.
[23:57:14] <paladox>	 Ticket opened with them (with a mtr report showing the issue)
[23:57:25] <paladox>	 (them being ramnode)
[23:58:09] <icinga-miraheze>	 RECOVERY - disabled.life - LetsEncrypt on sslhost is OK: OK - Certificate 'disabled.life' will expire on Thu 23 Jan 2020 10:01:30 AM GMT +0000.
[23:59:33] <icinga-miraheze>	 RECOVERY - wiki.udia.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'sni242919.cloudflaressl.com' will expire on Tue 26 May 2020 11:59:59 PM GMT +0000.
[23:59:37] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.
[23:59:37] <icinga-miraheze>	 RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sat 29 Feb 2020 04:34:39 PM GMT +0000.
[23:59:45] <icinga-miraheze>	 RECOVERY - yellowiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'yellowiki.xyz' will expire on Mon 02 Mar 2020 05:03:40 AM GMT +0000.
[23:59:50] <icinga-miraheze>	 RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Thu 30 Jan 2020 09:54:37 AM GMT +0000.