[00:01:09] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:02:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYku [00:02:10] [02miraheze/ssl] 07MirahezeSSLBot 03c76b15f - Bot: Update SSL cert for www.christipedia.nl [00:03:42] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 9895 MB (25% inode=96%); [00:05:11] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:05:22] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYk9 [00:05:24] [02miraheze/ssl] 07paladox 0339eb19d - Update www.portalsofphereon.com.crt [00:08:59] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYIf [00:09:01] [02miraheze/ssl] 07paladox 03656c2f3 - Update www.pornwiki.org.crt [00:09:12] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:09:22] Universal_Omega; yea but why do you need it? [00:10:20] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYIL [00:10:22] [02miraheze/ssl] 07paladox 034c061b2 - Update www.baharna.org.crt [00:15:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYIn [00:15:11] [02miraheze/ssl] 07paladox 0355255c0 - Update www.iceria.org.crt [00:17:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYIC [00:17:06] [02miraheze/ssl] 07paladox 0383a9ea8 - Update www.erikapedia.com.crt [00:17:49] Universal_Omega: if it’s for the meta namespace stuff in MWN, I don’t think MWN additional settings is appropriate - it’s not a per namespace setting. It should either be via changing the name or in MWS [00:18:40] Additional settings are designed to be used for settings that change per namespace not as a hack to making a one namespace only setting available [00:19:36] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYtq [00:19:38] [02miraheze/ssl] 07paladox 0361820bc - Update www.petrawiki.org.crt [00:21:27] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYmT [00:21:29] [02miraheze/ssl] 07paladox 03ecf77e1 - Update www.bluevies.org.crt [00:23:47] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYYd [00:23:48] [02miraheze/ssl] 07paladox 03cb8f915 - Update www.lukaibrisimovic.org.crt [00:24:21] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 336.51 ms [00:25:43] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYOT [00:25:45] [02miraheze/ssl] 07paladox 0348f0d4e - Update www.christipedia.nl.crt [00:26:04] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Fri 26 Mar 2021 00:22:18 GMT +0000). [00:26:24] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 250.02 ms [00:26:26] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.climatechange.ai' expires in 15 day(s) (Fri 26 Mar 2021 00:22:54 GMT +0000). [00:26:29] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.graalmilitary.com' expires in 15 day(s) (Fri 26 Mar 2021 00:24:28 GMT +0000). [00:26:44] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Fri 26 Mar 2021 00:23:10 GMT +0000). [00:26:45] PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Fri 26 Mar 2021 00:22:44 GMT +0000). [00:26:52] PROBLEM - wiki.campaign-labour.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.campaign-labour.org' expires in 15 day(s) (Fri 26 Mar 2021 00:22:31 GMT +0000). [00:26:56] PROBLEM - wiki.fourta.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fourta.org' expires in 15 day(s) (Fri 26 Mar 2021 00:24:02 GMT +0000). [00:27:09] PROBLEM - wiki.iufs.jp - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.iufs.jp' expires in 15 day(s) (Fri 26 Mar 2021 00:25:09 GMT +0000). [00:27:17] PROBLEM - wiki.azimuth.navy - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.azimuth.navy' expires in 15 day(s) (Fri 26 Mar 2021 00:22:25 GMT +0000). [00:27:42] PROBLEM - wiki.dom64.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dom64.xyz' expires in 15 day(s) (Fri 26 Mar 2021 00:23:38 GMT +0000). [00:27:44] PROBLEM - wiki.abyogsms.org.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.abyogsms.org.in' expires in 15 day(s) (Fri 26 Mar 2021 00:21:47 GMT +0000). [00:27:59] PROBLEM - wiki.kali-team.cn - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kali-team.cn' expires in 15 day(s) (Fri 26 Mar 2021 00:25:57 GMT +0000). [00:28:05] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Fri 26 Mar 2021 00:22:49 GMT +0000). [00:28:07] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.conworlds.org' expires in 15 day(s) (Fri 26 Mar 2021 00:23:02 GMT +0000). [00:28:15] PROBLEM - wiki.arnab.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.arnab.at' expires in 15 day(s) (Fri 26 Mar 2021 00:21:56 GMT +0000). [00:28:28] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cyberfurs.org' expires in 15 day(s) (Fri 26 Mar 2021 00:23:16 GMT +0000). [00:28:38] PROBLEM - wiki.etn.link - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.etn.link' expires in 15 day(s) (Fri 26 Mar 2021 00:23:56 GMT +0000). [00:28:38] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.danvs.net' expires in 15 day(s) (Fri 26 Mar 2021 00:23:24 GMT +0000). [00:28:38] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.erisly.com' expires in 15 day(s) (Fri 26 Mar 2021 00:23:45 GMT +0000). [00:28:41] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.anglish.info' expires in 15 day(s) (Fri 26 Mar 2021 00:21:51 GMT +0000). [00:29:01] PROBLEM - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jacksonheights.nyc' expires in 15 day(s) (Fri 26 Mar 2021 00:25:15 GMT +0000). [00:29:16] PROBLEM - wiki.hrznstudio.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.hrznstudio.com' expires in 15 day(s) (Fri 26 Mar 2021 00:24:43 GMT +0000). [00:29:20] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Fri 26 Mar 2021 00:24:19 GMT +0000). [00:29:40] PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mikrodev.com' expires in 15 day(s) (Fri 26 Mar 2021 00:26:55 GMT +0000). [00:29:49] Universal_Omega, if you're around there's 174 jobs on `metawiki` to determine the make up of the job types outstanding [00:29:52] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Fri 26 Mar 2021 00:24:33 GMT +0000). [00:30:14] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minkyu.kim' expires in 15 day(s) (Fri 26 Mar 2021 00:27:10 GMT +0000). [00:30:27] PROBLEM - wiki.fraterniterebellion.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fraterniterebellion.com' expires in 15 day(s) (Fri 26 Mar 2021 00:24:08 GMT +0000). [00:30:30] PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meeusen.net' expires in 15 day(s) (Fri 26 Mar 2021 00:26:48 GMT +0000). [00:30:34] Nevermind, jobs cleared [00:30:45] PROBLEM - wiki.candela.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.candela.digital' expires in 15 day(s) (Fri 26 Mar 2021 00:22:38 GMT +0000). [00:30:56] If jobs clear after a minute, it’s hardly a backlog really [00:31:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYOB [00:31:02] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nowchess.org' expires in 15 day(s) (Fri 26 Mar 2021 00:27:48 GMT +0000). [00:31:03] [02miraheze/ssl] 07MirahezeSSLBot 03cce82a7 - Bot: Update SSL cert for wiki.autocountsoft.com [00:31:09] PROBLEM - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lefrenchmelee.fr' expires in 15 day(s) (Fri 26 Mar 2021 00:26:10 GMT +0000). [00:31:11] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mobilityengineer.com' expires in 15 day(s) (Fri 26 Mar 2021 00:27:29 GMT +0000). [00:31:11] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Fri 26 Mar 2021 00:27:40 GMT +0000). [00:31:16] PROBLEM - wiki.funkey-project.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.funkey-project.com' expires in 15 day(s) (Fri 26 Mar 2021 00:24:13 GMT +0000). [00:31:17] PROBLEM - wiki.invinciblewellbeing.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.invinciblewellbeing.com' expires in 15 day(s) (Fri 26 Mar 2021 00:24:57 GMT +0000). [00:31:23] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kourouklides.com' expires in 15 day(s) (Fri 26 Mar 2021 00:26:05 GMT +0000). [00:31:42] PROBLEM - www.fraterniterebellion.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fraterniterebellion.com' expires in 15 day(s) (Fri 26 Mar 2021 00:24:08 GMT +0000). [00:32:03] PROBLEM - wiki.mnzp.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mnzp.xyz' expires in 15 day(s) (Fri 26 Mar 2021 00:27:22 GMT +0000). [00:32:11] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.joust.ro' expires in 15 day(s) (Fri 26 Mar 2021 00:25:48 GMT +0000). [00:32:14] JohnLewis, yeah, these aren't good examples, as they're being cleared fairly quickly. I'll keep an eye for when there's a backlog that doesn't clear right away [00:32:21] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mastodon.kr' expires in 15 day(s) (Fri 26 Mar 2021 00:26:36 GMT +0000). [00:32:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYOE [00:32:24] [02miraheze/ssl] 07MirahezeSSLBot 031fb1f29 - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de [00:32:28] PROBLEM - wiki.helioss.co - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.helioss.co' expires in 15 day(s) (Fri 26 Mar 2021 00:24:38 GMT +0000). [00:32:58] PROBLEM - wiki.jafaron.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jafaron.net' expires in 15 day(s) (Fri 26 Mar 2021 00:25:42 GMT +0000). [00:33:04] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Fri 26 Mar 2021 00:26:18 GMT +0000). [00:33:55] PROBLEM - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pablojprieto.com.es' expires in 15 day(s) (Fri 26 Mar 2021 00:29:21 GMT +0000). [00:33:57] PROBLEM - wiki.responsibly.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.responsibly.ai' expires in 15 day(s) (Fri 26 Mar 2021 00:30:56 GMT +0000). [00:34:06] PROBLEM - wiki.mussa.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mussa.id' expires in 15 day(s) (Fri 26 Mar 2021 00:27:34 GMT +0000). [00:34:12] PROBLEM - wiki.raghuveer.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.raghuveer.net' expires in 15 day(s) (Fri 26 Mar 2021 00:30:18 GMT +0000). [00:34:14] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seamly.net' expires in 15 day(s) (Fri 26 Mar 2021 00:31:45 GMT +0000). [00:34:19] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minecraftathome.com' expires in 15 day(s) (Fri 26 Mar 2021 00:26:59 GMT +0000). [00:34:38] PROBLEM - wiki.ostans.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ostans.com' expires in 15 day(s) (Fri 26 Mar 2021 00:28:19 GMT +0000). [00:34:56] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineland.eu' expires in 15 day(s) (Fri 26 Mar 2021 00:27:05 GMT +0000). [00:35:12] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.make717.org' expires in 15 day(s) (Fri 26 Mar 2021 00:26:23 GMT +0000). [00:35:12] PROBLEM - wiki.patriam.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.patriam.cc' expires in 15 day(s) (Fri 26 Mar 2021 00:30:08 GMT +0000). [00:35:14] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [00:35:15] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mcpirevival.tk' expires in 15 day(s) (Fri 26 Mar 2021 00:26:40 GMT +0000). [00:35:31] PROBLEM - wiki.openhatch.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.openhatch.org' expires in 15 day(s) (Fri 26 Mar 2021 00:28:01 GMT +0000). [00:35:45] PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.triplescripts.org' expires in 15 day(s) (Fri 26 Mar 2021 00:33:11 GMT +0000). [00:35:53] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 15 day(s) (Fri 26 Mar 2021 00:33:03 GMT +0000). [00:35:57] PROBLEM - wiki.starship.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.starship.digital' expires in 15 day(s) (Fri 26 Mar 2021 00:32:23 GMT +0000). [00:36:11] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.qadrishattari.xyz' expires in 15 day(s) (Fri 26 Mar 2021 00:30:13 GMT +0000). [00:36:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYOM [00:36:27] [02miraheze/ssl] 07MirahezeSSLBot 0315b2a26 - Bot: Update SSL cert for wiki.hrznstudio.com [00:36:35] PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.scvo.org' expires in 15 day(s) (Fri 26 Mar 2021 00:31:29 GMT +0000). [00:36:54] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.shaazzz.ir' expires in 15 day(s) (Fri 26 Mar 2021 00:32:05 GMT +0000). [00:37:22] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.teessidehackspace.org.uk' expires in 15 day(s) (Fri 26 Mar 2021 00:32:41 GMT +0000). [00:37:24] PROBLEM - wiki.seredrau.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seredrau.xyz' expires in 15 day(s) (Fri 26 Mar 2021 00:31:56 GMT +0000). [00:37:35] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 15 day(s) (Fri 26 Mar 2021 00:32:18 GMT +0000). [00:37:38] PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.vinesh.eu.org' expires in 15 day(s) (Fri 26 Mar 2021 00:34:20 GMT +0000). [00:37:48] PROBLEM - wiki.usagihime.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.usagihime.ml' expires in 15 day(s) (Fri 26 Mar 2021 00:33:46 GMT +0000). [00:37:50] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Fri 26 Mar 2021 00:34:12 GMT +0000). [00:37:55] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rebirthofthenight.com' expires in 15 day(s) (Fri 26 Mar 2021 00:30:39 GMT +0000). [00:37:55] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Fri 26 Mar 2021 00:34:34 GMT +0000). [00:37:56] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Fri 26 Mar 2021 00:33:21 GMT +0000). [00:37:58] PROBLEM - wiki.thehall.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thehall.xyz' expires in 15 day(s) (Fri 26 Mar 2021 00:32:53 GMT +0000). [00:38:04] PROBLEM - wikien.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikien.wildterra2.com' expires in 15 day(s) (Fri 26 Mar 2021 00:35:57 GMT +0000). [00:38:10] PROBLEM - wiki.teamrelectric.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.teamrelectric.ca' expires in 15 day(s) (Fri 26 Mar 2021 00:32:29 GMT +0000). [00:38:10] PROBLEM - wiki.redeemer.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.redeemer.live' expires in 15 day(s) (Fri 26 Mar 2021 00:30:48 GMT +0000). [00:38:15] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Fri 26 Mar 2021 00:31:21 GMT +0000). [00:38:28] PROBLEM - wiki.xysspon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.xysspon.com' expires in 15 day(s) (Fri 26 Mar 2021 00:35:11 GMT +0000). [00:38:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqY3n [00:38:46] [02miraheze/ssl] 07MirahezeSSLBot 030430e9a - Bot: Update SSL cert for wiki.mnzp.xyz [00:39:19] PROBLEM - wiki.ripto.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ripto.gq' expires in 15 day(s) (Fri 26 Mar 2021 00:31:12 GMT +0000). [00:39:22] PROBLEM - wiki.sdbx.dev - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sdbx.dev' expires in 15 day(s) (Fri 26 Mar 2021 00:31:38 GMT +0000). [00:39:24] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Fri 26 Mar 2021 00:31:04 GMT +0000). [00:39:41] JohnLewis: Multiple configs set to it in LS.php already, because it works in MWS, MWN, etc, not in LS, so they use wgSitename in LS itself. [00:39:44] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worlduniversityandschool.org' expires in 15 day(s) (Fri 26 Mar 2021 00:34:49 GMT +0000). [00:39:46] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiru.wildterra2.com' expires in 15 day(s) (Fri 26 Mar 2021 00:36:34 GMT +0000). [00:40:10] PROBLEM - www.wikilukas.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikilukas.tk' expires in 15 day(s) (Fri 26 Mar 2021 00:36:12 GMT +0000). [00:40:37] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Fri 26 Mar 2021 00:35:47 GMT +0000). [00:40:43] PROBLEM - wiki.wysteria.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wysteria.ga' expires in 15 day(s) (Fri 26 Mar 2021 00:34:57 GMT +0000). [00:40:45] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.valkyrienskies.org' expires in 15 day(s) (Fri 26 Mar 2021 00:33:56 GMT +0000). [00:40:50] PROBLEM - wiki.yunachannel.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yunachannel.com' expires in 15 day(s) (Fri 26 Mar 2021 00:35:42 GMT +0000). [00:41:04] PROBLEM - wiki.velaan.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.velaan.org' expires in 15 day(s) (Fri 26 Mar 2021 00:34:04 GMT +0000). [00:41:33] PROBLEM - wiki.twilightsignal.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.twilightsignal.com' expires in 15 day(s) (Fri 26 Mar 2021 00:33:30 GMT +0000). [00:42:04] PROBLEM - wiki.xfbs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.xfbs.net' expires in 15 day(s) (Fri 26 Mar 2021 00:35:05 GMT +0000). [00:42:28] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 15 day(s) (Fri 26 Mar 2021 00:34:29 GMT +0000). [00:42:33] Universal_Omega: mkay, though can we use MWN additional settings for its original purpose and get rid of meta namespace settings from it in favour of changing the NS name or MWS? [00:42:34] PROBLEM - wikibeats.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikibeats.net' expires in 15 day(s) (Fri 26 Mar 2021 00:35:52 GMT +0000). [00:42:54] PROBLEM - wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiescola.com.br' expires in 15 day(s) (Fri 26 Mar 2021 00:36:03 GMT +0000). [00:43:09] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worldsofweary.com' expires in 15 day(s) (Fri 26 Mar 2021 00:34:42 GMT +0000). [00:43:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYZ8 [00:43:11] [02miraheze/ssl] 07MirahezeSSLBot 034bb27e4 - Bot: Update SSL cert for wiki.vinesh.eu.org [00:43:59] PROBLEM - www.wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiescola.com.br' expires in 15 day(s) (Fri 26 Mar 2021 00:36:03 GMT +0000). [00:44:31] PROBLEM - wikilukas.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikilukas.tk' expires in 15 day(s) (Fri 26 Mar 2021 00:36:12 GMT +0000). [00:44:42] PROBLEM - wikipariksha.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipariksha.com' expires in 15 day(s) (Fri 26 Mar 2021 00:36:17 GMT +0000). [00:45:20] JohnLewis: The reason I did those in MWN, is because only MWN has the support needed to actually read them, MWS does not work with them, however if you'd really like, I can remove them period, though I do like the fact that I finally found a way to support them, but I'm ok with removing them and I guess running through another test with support in MWS if you wish? [00:46:24] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqYc8 [00:46:24] No, it’s fine I guess... it’s just an entirely inappropriate use of how the features were designed to be used but if you want to do it that way you can [00:46:26] [02miraheze/CreateWiki] 07Universal-Omega 03f17a87a - add sitename variable for LocalSettings.php to access [00:46:27] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [00:46:29] [02CreateWiki] 07Universal-Omega opened pull request 03#199: add sitename variable for LocalSettings.php to access - 13https://git.io/JqYc4 [00:47:11] JohnLewis: Ok, I will look into another option then as I see your point, and also ^ is that ok? Or just do that at top of LS instead like mentioned before? [00:47:23] To be honest, I’m considering stepping back from maintaining the extensions anyway. People seem to want to disregard my advice and continually give my resistance and with my other commitments I don’t regard the extensions an effective use of my time anymore [00:47:26] miraheze/CreateWiki - Universal-Omega the build passed. [00:48:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYcV [00:49:00] [02miraheze/ssl] 07MirahezeSSLBot 03ce385b1 - Bot: Update SSL cert for wiki.campaign-labour.org [00:49:56] With that CW change, do it which ever way works best for you - dealing with extensions and MediaWiki code isn’t a priority for me anymore [00:55:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYCU [00:55:04] [02miraheze/ssl] 07MirahezeSSLBot 03172194e - Bot: Update SSL cert for wiki.worldsofweary.com [00:55:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqYCI [00:55:59] [02miraheze/puppet] 07paladox 03073acb4 - Add some logging to mirahezerenewssl [00:56:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqYCI [00:56:02] [02miraheze/puppet] 07paladox 03073acb4 - Add some logging to mirahezerenewssl [00:56:03] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:56:05] [02puppet] 07paladox opened pull request 03#1696: Add some logging to mirahezerenewssl - 13https://git.io/JqYCL [00:56:08] JohnLewis: I understand, but I am truly sorry to hear that. [00:56:35] [02CreateWiki] 07Universal-Omega closed pull request 03#199: add sitename variable for LocalSettings.php to access - 13https://git.io/JqYc4 [00:56:36] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYCq [00:56:38] [02miraheze/CreateWiki] 07Universal-Omega 036b584ab - add sitename variable for LocalSettings.php to access (#199) [00:56:39] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [00:56:41] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:57:33] miraheze/CreateWiki - Universal-Omega the build passed. [00:58:17] [02CreateWiki] 07Universal-Omega closed pull request 03#198: Add 'name' to bio and purpose fields for consistency - 13https://git.io/JqLUu [00:58:19] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:58:21] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [00:59:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqYC6 [00:59:18] [02miraheze/ssl] 07MirahezeSSLBot 036134af9 - Bot: Update SSL cert for wiki.meeusen.net [01:01:51] PROBLEM - www.allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Fri 26 Mar 2021 00:59:24 GMT +0000). [01:02:21] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'antiguabarbudacalypso.com' expires in 15 day(s) (Fri 26 Mar 2021 00:59:37 GMT +0000). [01:03:19] PROBLEM - allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Fri 26 Mar 2021 00:59:24 GMT +0000). [01:04:08] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.stellurgists.wiki' expires in 15 day(s) (Fri 26 Mar 2021 00:59:54 GMT +0000). [01:04:15] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bebaskanpengetahuan.id' expires in 15 day(s) (Fri 26 Mar 2021 01:01:18 GMT +0000). [01:05:03] PROBLEM - battlecats.f5.si - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'battlecats.f5.si' expires in 15 day(s) (Fri 26 Mar 2021 01:00:06 GMT +0000). [01:05:06] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:02:57 GMT +0000). [01:05:29] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [01:06:14] [02miraheze/ssl] 07paladox created branch 03master 13https://git.io/JqYRv [01:06:15] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ao90.pinho.org' expires in 15 day(s) (Fri 26 Mar 2021 00:59:46 GMT +0000). [01:06:30] PROBLEM - civclassic.civwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'civclassic.civwiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:02:19 GMT +0000). [01:06:51] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aman.awiki.org' expires in 15 day(s) (Fri 26 Mar 2021 00:59:33 GMT +0000). [01:07:08] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:07:08] PROBLEM - en.nocyclo.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.nocyclo.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:04:55 GMT +0000). [01:07:22] PROBLEM - debilepedie.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'debilepedie.gq' expires in 15 day(s) (Fri 26 Mar 2021 01:04:20 GMT +0000). [01:07:35] PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cyberlaw.ccdcoe.org' expires in 15 day(s) (Fri 26 Mar 2021 01:02:50 GMT +0000). [01:07:43] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'commons.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:02:25 GMT +0000). [01:07:59] PROBLEM - dreamsit.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dreamsit.com.br' expires in 15 day(s) (Fri 26 Mar 2021 01:04:25 GMT +0000). [01:08:04] PROBLEM - cubashpedia.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cubashpedia.cf' expires in 15 day(s) (Fri 26 Mar 2021 01:02:42 GMT +0000). [01:08:15] PROBLEM - bharatwiki.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bharatwiki.online' expires in 15 day(s) (Fri 26 Mar 2021 01:01:23 GMT +0000). [01:08:39] PROBLEM - de.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'de.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:04:01 GMT +0000). [01:08:47] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 15 day(s) (Fri 26 Mar 2021 01:05:47 GMT +0000). [01:08:51] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [01:09:08] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [01:09:19] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fr.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:06:54 GMT +0000). [01:09:36] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:05:57 GMT +0000). [01:10:05] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:02:57 GMT +0000). [01:10:31] PROBLEM - duepedia.uk.to - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'duepedia.uk.to' expires in 15 day(s) (Fri 26 Mar 2021 01:04:30 GMT +0000). [01:10:35] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:03:35 GMT +0000). [01:10:37] PROBLEM - halloween.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'halloween.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:07:58 GMT +0000). [01:10:46] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Fri 26 Mar 2021 01:07:45 GMT +0000). [01:11:13] PROBLEM - electowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electowiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:04:35 GMT +0000). [01:11:26] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'icclopedia.org' expires in 15 day(s) (Fri 26 Mar 2021 01:09:09 GMT +0000). [01:11:55] PROBLEM - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dc-multiverse.dcwikis.com' expires in 15 day(s) (Fri 26 Mar 2021 01:03:45 GMT +0000). [01:11:57] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [01:11:58] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:11:58] PROBLEM - jobrunner3 MirahezeRenewSsl on jobrunner3 is CRITICAL: connect to address 51.195.236.220 and port 5000: Connection refused [01:12:07] PROBLEM - froggy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'froggy.info' expires in 15 day(s) (Fri 26 Mar 2021 01:07:18 GMT +0000). [01:12:39] PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.sdtef.org' expires in 15 day(s) (Fri 26 Mar 2021 01:08:37 GMT +0000). [01:12:42] PROBLEM - iesd.mobagenie.my.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'iesd.mobagenie.my.id' expires in 15 day(s) (Fri 26 Mar 2021 01:09:38 GMT +0000). [01:12:48] PROBLEM - en.phgalaxy.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.phgalaxy.xyz' expires in 15 day(s) (Fri 26 Mar 2021 01:05:20 GMT +0000). [01:12:52] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:04:48 GMT +0000). [01:13:09] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'index.pdcommunity.ir' expires in 15 day(s) (Fri 26 Mar 2021 01:09:46 GMT +0000). [01:13:14] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hi.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:08:03 GMT +0000). [01:13:22] PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.esporo.net' expires in 15 day(s) (Fri 26 Mar 2021 01:07:53 GMT +0000). [01:13:22] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'franchise.franchising.org.ua' expires in 15 day(s) (Fri 26 Mar 2021 01:07:00 GMT +0000). [01:13:53] PROBLEM - es.nocyclo.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.nocyclo.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:06:05 GMT +0000). [01:14:11] PROBLEM - innersphere.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'innersphere.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:10:21 GMT +0000). [01:14:21] PROBLEM - history.estill.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.estill.org' expires in 15 day(s) (Fri 26 Mar 2021 01:08:08 GMT +0000). [01:14:34] PROBLEM - hololive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hololive.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:08:49 GMT +0000). [01:14:44] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediaofastrobiology.org' expires in 15 day(s) (Fri 26 Mar 2021 01:05:52 GMT +0000). [01:15:07] PROBLEM - kkutu.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kkutu.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:10:54 GMT +0000). [01:15:10] PROBLEM - factorio-mods.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'factorio-mods.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:06:41 GMT +0000). [01:15:16] PROBLEM - idleon.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'idleon.info' expires in 15 day(s) (Fri 26 Mar 2021 01:09:27 GMT +0000). [01:15:22] PROBLEM - holonet.pw - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'holonet.pw' expires in 15 day(s) (Fri 26 Mar 2021 01:08:55 GMT +0000). [01:15:45] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fc.songcontests.eu' expires in 15 day(s) (Fri 26 Mar 2021 01:06:49 GMT +0000). [01:15:46] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lg.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:11:16 GMT +0000). [01:16:10] PROBLEM - kk.uncyc.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kk.uncyc.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:10:44 GMT +0000). [01:16:43] PROBLEM - indiancannabis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'indiancannabis.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:09:55 GMT +0000). [01:17:22] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Fri 26 Mar 2021 01:11:02 GMT +0000). [01:17:23] PROBLEM - lama.madooa.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lama.madooa.org' expires in 15 day(s) (Fri 26 Mar 2021 01:11:10 GMT +0000). [01:17:25] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ipv6bolivia.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:10:26 GMT +0000). [01:17:51] PROBLEM - housing.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'housing.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:09:03 GMT +0000). [01:17:58] PROBLEM - meregos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meregos.com' expires in 15 day(s) (Fri 26 Mar 2021 01:15:39 GMT +0000). [01:18:26] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'infectowiki.com' expires in 15 day(s) (Fri 26 Mar 2021 01:10:02 GMT +0000). [01:18:38] PROBLEM - jobrunner3 Puppet on jobrunner3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 1 failures [01:19:05] PROBLEM - miraheze.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:16:42 GMT +0000). [01:19:09] PROBLEM - madzebrascience.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'madzebrascience.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:14:58 GMT +0000). [01:19:27] PROBLEM - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'beaconspace.unrestrictedlorefare.com' expires in 15 day(s) (Fri 26 Mar 2021 01:12:39 GMT +0000). [01:19:27] PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Fri 26 Mar 2021 01:15:58 GMT +0000). [01:19:45] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Fri 26 Mar 2021 01:15:07 GMT +0000). [01:19:55] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Fri 26 Mar 2021 01:15:58 GMT +0000). [01:20:12] PROBLEM - chouverse.cludisciples.cyou - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'chouverse.cludisciples.cyou' expires in 15 day(s) (Fri 26 Mar 2021 01:13:19 GMT +0000). [01:20:15] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 15 day(s) (Fri 26 Mar 2021 01:18:07 GMT +0000). [01:20:25] PROBLEM - madgenderscience.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'madgenderscience.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:14:50 GMT +0000). [01:20:34] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 15 day(s) (Fri 26 Mar 2021 01:18:22 GMT +0000). [01:20:35] PROBLEM - nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:17:36 GMT +0000). [01:20:39] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ga' expires in 15 day(s) (Fri 26 Mar 2021 01:16:07 GMT +0000). [01:20:42] PROBLEM - meta.nocyclo.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meta.nocyclo.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:15:52 GMT +0000). [01:20:52] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Fri 26 Mar 2021 01:15:07 GMT +0000). [01:20:53] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:21:01] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 15 day(s) (Fri 26 Mar 2021 01:18:22 GMT +0000). [01:21:11] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mediawiki.ecfin.eu' expires in 15 day(s) (Fri 26 Mar 2021 01:15:12 GMT +0000). [01:21:25] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meta.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:15:44 GMT +0000). [01:21:26] PROBLEM - munwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'munwiki.info' expires in 15 day(s) (Fri 26 Mar 2021 01:17:21 GMT +0000). [01:21:35] PROBLEM - n64brew.dev - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'n64brew.dev' expires in 15 day(s) (Fri 26 Mar 2021 01:17:26 GMT +0000). [01:21:44] PROBLEM - oecumene.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'oecumene.org' expires in 15 day(s) (Fri 26 Mar 2021 01:18:12 GMT +0000). [01:21:56] PROBLEM - miraheze.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ml' expires in 15 day(s) (Fri 26 Mar 2021 01:16:33 GMT +0000). [01:22:05] PROBLEM - planetnomads.speedbomb.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'planetnomads.speedbomb.de' expires in 15 day(s) (Fri 26 Mar 2021 01:19:01 GMT +0000). [01:22:13] PROBLEM - monarchia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'monarchia.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:16:57 GMT +0000). [01:22:19] PROBLEM - programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Fri 26 Mar 2021 01:19:42 GMT +0000). [01:22:34] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Fri 26 Mar 2021 01:20:00 GMT +0000). [01:22:46] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mrlove.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:17:16 GMT +0000). [01:22:49] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqY0Q [01:22:51] [02miraheze/ManageWiki] 07Universal-Omega 03b2aea04 - Use createwiki-infuse for ManageWikiNamespacesAdditional [01:22:52] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:23:12] PROBLEM - www.programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Fri 26 Mar 2021 01:19:42 GMT +0000). [01:23:13] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:17:57 GMT +0000). [01:23:39] PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:17:36 GMT +0000). [01:23:50] miraheze/ManageWiki - Universal-Omega the build passed. [01:23:54] PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pa.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:18:27 GMT +0000). [01:24:29] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.gq' expires in 15 day(s) (Fri 26 Mar 2021 01:16:16 GMT +0000). [01:24:47] PROBLEM - sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Fri 26 Mar 2021 01:21:02 GMT +0000). [01:24:54] PROBLEM - stablestate.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'stablestate.org' expires in 15 day(s) (Fri 26 Mar 2021 01:21:47 GMT +0000). [01:24:54] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:21:37 GMT +0000). [01:24:59] PROBLEM - studentwiki.ddns.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'studentwiki.ddns.net' expires in 15 day(s) (Fri 26 Mar 2021 01:22:00 GMT +0000). [01:25:03] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pubwiki.lab.co.pl' expires in 15 day(s) (Fri 26 Mar 2021 01:20:09 GMT +0000). [01:25:11] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mr.gyaanipedia.com' expires in 15 day(s) (Fri 26 Mar 2021 01:17:11 GMT +0000). [01:25:27] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Fri 26 Mar 2021 01:19:10 GMT +0000). [01:25:31] PROBLEM - www.sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Fri 26 Mar 2021 01:21:02 GMT +0000). [01:25:36] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tensegritywiki.com' expires in 15 day(s) (Fri 26 Mar 2021 01:22:13 GMT +0000). [01:25:50] PROBLEM - pastport.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pastport.org' expires in 15 day(s) (Fri 26 Mar 2021 01:18:41 GMT +0000). [01:25:55] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:18:47 GMT +0000). [01:26:01] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'monarchists.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:17:05 GMT +0000). [01:26:07] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: Full, 541500 files, 40.45GB, 2021-02-07 01:25:00 (4.4 weeks ago) [01:26:10] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thelonsdalebattalion.co.uk' expires in 15 day(s) (Fri 26 Mar 2021 01:22:34 GMT +0000). [01:26:10] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:18:47 GMT +0000). [01:26:12] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Fri 26 Mar 2021 01:23:33 GMT +0000). [01:26:16] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tl.awiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:22:39 GMT +0000). [01:26:52] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spcodex.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:21:17 GMT +0000). [01:26:55] PROBLEM - es.publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Fri 26 Mar 2021 01:20:00 GMT +0000). [01:27:25] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vise.dayid.org' expires in 15 day(s) (Fri 26 Mar 2021 01:23:27 GMT +0000). [01:27:30] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Fri 26 Mar 2021 01:20:00 GMT +0000). [01:27:36] PROBLEM - techwiki.techboyg5blog.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'techwiki.techboyg5blog.com' expires in 15 day(s) (Fri 26 Mar 2021 01:22:05 GMT +0000). [01:27:39] PROBLEM - vortekspedia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vortekspedia.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:23:49 GMT +0000). [01:27:43] PROBLEM - tep.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tep.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:22:24 GMT +0000). [01:27:48] PROBLEM - rupowerrangersfanon.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rupowerrangersfanon.tk' expires in 15 day(s) (Fri 26 Mar 2021 01:20:51 GMT +0000). [01:28:10] PROBLEM - privacy-wiki.0x.no - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'privacy-wiki.0x.no' expires in 15 day(s) (Fri 26 Mar 2021 01:19:35 GMT +0000). [01:28:11] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'runzeppelin.ru' expires in 15 day(s) (Fri 26 Mar 2021 01:20:43 GMT +0000). [01:28:15] PROBLEM - virus23.uk.to - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'virus23.uk.to' expires in 15 day(s) (Fri 26 Mar 2021 01:23:19 GMT +0000). [01:28:24] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vedopedia.witches-empire.com' expires in 15 day(s) (Fri 26 Mar 2021 01:23:01 GMT +0000). [01:28:25] PROBLEM - tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tradwiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:22:53 GMT +0000). [01:28:54] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Fri 26 Mar 2021 01:23:41 GMT +0000). [01:28:59] PROBLEM - warmage.necrocity.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'warmage.necrocity.co.uk' expires in 15 day(s) (Fri 26 Mar 2021 01:23:57 GMT +0000). [01:29:02] PROBLEM - savagepedia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'savagepedia.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:20:56 GMT +0000). [01:29:16] PROBLEM - spiral.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spiral.wiki' expires in 15 day(s) (Fri 26 Mar 2021 01:21:42 GMT +0000). [01:29:28] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.arkcls.com' expires in 15 day(s) (Fri 26 Mar 2021 01:20:36 GMT +0000). [01:30:04] PROBLEM - viileapedia.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'viileapedia.ga' expires in 15 day(s) (Fri 26 Mar 2021 01:23:11 GMT +0000). [01:30:22] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'storytime.jdstroy.cf' expires in 15 day(s) (Fri 26 Mar 2021 01:21:55 GMT +0000). [01:30:25] PROBLEM - towarzystwo.prawa.kredytowego.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'towarzystwo.prawa.kredytowego.pl' expires in 15 day(s) (Fri 26 Mar 2021 01:22:47 GMT +0000). [01:30:54] PROBLEM - www.tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tradwiki.org' expires in 15 day(s) (Fri 26 Mar 2021 01:22:53 GMT +0000). [01:33:39] SPF|Cloud hi [01:48:51] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:52:51] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [02:18:03] FYI puppet is disabled in jobrunner 3 [02:25:41] PROBLEM - miraheze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.com' expires in 15 day(s) (Fri 26 Mar 2021 02:18:45 GMT +0000). [02:26:16] paladox, that shouldn't have any effect on the global rename job I just initiated, right? [02:27:09] no [02:27:10] it wont [02:28:00] okay, thanks, Zppix. I didn't think it would, but thought I better ask just to be sure [02:38:37] PROBLEM - meta.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 26 Mar 2021 02:34:11 GMT +0000). [02:38:58] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: CRITICAL: Full, 304278 files, 45.75GB, 2021-02-07 02:38:00 (4.4 weeks ago) [02:39:03] PROBLEM - bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Fri 26 Mar 2021 02:32:28 GMT +0000). [02:39:45] PROBLEM - tools1.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Fri 26 Mar 2021 02:32:28 GMT +0000). [02:40:00] PROBLEM - db1.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Fri 26 Mar 2021 02:32:28 GMT +0000). [02:40:03] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Fri 26 Mar 2021 02:32:28 GMT +0000). [02:40:14] PROBLEM - orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 26 Mar 2021 02:34:11 GMT +0000). [02:40:37] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 26 Mar 2021 02:34:11 GMT +0000). [02:41:00] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 26 Mar 2021 02:34:11 GMT +0000). [02:41:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqY9N [02:41:15] [02miraheze/mw-config] 07Universal-Omega 03e54f159 - wgCreateWikiBlacklistedSubdomains: change some \d{1,2} to \d* [02:41:17] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:41:18] [02mw-config] 07Universal-Omega opened pull request 03#3767: wgCreateWikiBlacklistedSubdomains: change some \d{1,2} to \d* - 13https://git.io/JqY9A [02:41:25] PROBLEM - bots1.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Fri 26 Mar 2021 02:32:28 GMT +0000). [02:42:20] miraheze/mw-config - Universal-Omega the build passed. [02:59:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqYN0 [02:59:38] [02miraheze/mw-config] 07Universal-Omega 031d42883 - Update LocalSettings.php [02:59:39] [02mw-config] 07Universal-Omega synchronize pull request 03#3767: wgCreateWikiBlacklistedSubdomains: change some \d{1,2} to \d* - 13https://git.io/JqY9A [02:59:57] [02mw-config] 07Universal-Omega edited pull request 03#3767: wgCreateWikiBlacklistedSubdomains: change some \d{1,2} to \d{0,2} - 13https://git.io/JqY9A [03:00:32] miraheze/mw-config - Universal-Omega the build passed. [03:53:06] [02mw-config] 07Universal-Omega closed pull request 03#3767: wgCreateWikiBlacklistedSubdomains: change some \d{1,2} to \d{0,2} - 13https://git.io/JqY9A [03:53:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqO3n [03:53:09] [02miraheze/mw-config] 07Universal-Omega 03f6fc86a - wgCreateWikiBlacklistedSubdomains: change some \d{1,2} to \d{0,2} (#3767) [03:53:11] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [03:53:12] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [03:54:02] miraheze/mw-config - Universal-Omega the build passed. [03:54:36] [02mw-config] 07Universal-Omega closed pull request 03#3675: Fix config name - 13https://git.io/Jt350 [03:54:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqO3l [03:54:39] [02miraheze/mw-config] 07Universal-Omega 0322fdf13 - Fix config name (#3675) [03:54:41] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [03:54:42] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [03:55:35] miraheze/mw-config - Universal-Omega the build passed. [03:57:24] !log UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgDisqusShortname', 'egDisqusShortname'); [03:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:04:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JqOsg [04:04:42] [02miraheze/mw-config] 07Universal-Omega 0379f3e43 - Update LocalSettings.php [04:04:44] [02mw-config] 07Universal-Omega synchronize pull request 03#3765: Add ahinfoboxeswiki configs (T6939) - 13https://git.io/JqmsJ [04:05:27] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [04:05:38] [02mw-config] 07Universal-Omega closed pull request 03#3765: Add ahinfoboxeswiki configs (T6939) - 13https://git.io/JqmsJ [04:05:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqOGw [04:05:41] [02miraheze/mw-config] 07Universal-Omega 03401fade - Add ahinfoboxeswiki configs (T6939) (#3765) [04:05:42] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [04:05:44] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [04:05:53] miraheze/mw-config - Universal-Omega the build passed. [04:06:43] miraheze/mw-config - Universal-Omega the build passed. [04:12:03] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [04:21:47] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CRITICAL: Full, 207255 files, 80.26GB, 2021-02-07 04:21:00 (4.4 weeks ago) [04:58:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±2] 13https://git.io/JqOrR [04:58:10] [02miraheze/mediawiki] 07Universal-Omega 036698f64 - Update CreateWiki and ManageWiki [05:01:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqOrd [05:01:27] [02miraheze/mw-config] 07Universal-Omega 03609db57 - use $wi->sitename not $wgSitename [05:01:29] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [05:01:31] [02mw-config] 07Universal-Omega opened pull request 03#3768: use $wi->sitename not $wgSitename - 13https://git.io/JqOrF [05:02:37] miraheze/mw-config - Universal-Omega the build passed. [05:03:03] [02mw-config] 07Universal-Omega closed pull request 03#3768: use $wi->sitename not $wgSitename - 13https://git.io/JqOrF [05:03:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqOof [05:03:06] [02miraheze/mw-config] 07Universal-Omega 03c62019a - use $wi->sitename not $wgSitename (#3768) [05:03:08] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [05:03:09] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [05:04:02] miraheze/mw-config - Universal-Omega the build passed. [06:22:03] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jq3Tf [06:22:05] [02miraheze/MirahezeMagic] 07Universal-Omega 03dd6892d - Update MirahezeIRCRCFeedFormatter.php [06:22:06] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#215: Bring up-to-date with IRCColourfulRCFeedFormatter - 13https://git.io/Jtbj7 [06:23:06] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:23:41] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jq3TZ [06:23:42] [02miraheze/MirahezeMagic] 07Universal-Omega 0309ee88f - Update MirahezeIRCRCFeedFormatter.php [06:23:44] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#215: Bring up-to-date with IRCColourfulRCFeedFormatter - 13https://git.io/Jtbj7 [06:24:46] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:24:52] [02MirahezeMagic] 07Universal-Omega closed pull request 03#215: Bring up-to-date with IRCColourfulRCFeedFormatter - 13https://git.io/Jtbj7 [06:24:54] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 034 commits to 03master [+0/-0/±4] 13https://git.io/Jq3TE [06:24:55] [02miraheze/MirahezeMagic] 07Universal-Omega 0362eee17 - Merge pull request #215 from miraheze/Universal-Omega-patch-2 [06:24:57] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [06:24:58] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [06:25:56] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:29:50] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#214 commit - 13https://git.io/Jq3Ty [06:40:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jq3m4 [06:40:10] [02miraheze/services] 07MirahezeSSLBot 03fbbc753 - BOT: Updating services config for wikis [06:45:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jq3Od [06:45:11] [02miraheze/services] 07MirahezeSSLBot 0360dc830 - BOT: Updating services config for wikis [08:19:16] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.68, 3.18, 1.89 [08:21:15] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.00, 2.69, 1.86 [11:39:39] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:29] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineland.eu' expires in 15 day(s) (Fri 26 Mar 2021 00:27:05 GMT +0000). [13:21:43] [02mw-config] 07R4356th synchronize pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [13:22:45] miraheze/mw-config - R4356th the build passed. [13:28:32] [02mw-config] 07R4356th commented on pull request 03#3761: Better canned responses - 13https://git.io/JqG1T [15:24:38] PROBLEM - reviwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'reviwiki.info' expires in 15 day(s) (Fri 26 Mar 2021 15:22:32 GMT +0000). [15:27:00] [02mw-config] 07dmehus commented on pull request 03#3761: Better canned responses - 13https://git.io/JqZuC [15:33:38] JohnLewis, have you heard back from Outlook.com at all? Miraheze users (including myself) are still reporting an inability to communicate with other users via Special:EmailUser when the user has an Outlook.com/Hotmail.com e-mail address [15:34:49] dmehus: I haven't I'll follow it back up with them today [15:36:14] JohnLewis, ack and thanks :) [16:00:16] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JqZgN [16:00:17] [02miraheze/ssl] 07Reception123 038bfb526 - meregos.com -> wiki.meregos.com [16:09:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqZ24 [16:09:06] [02miraheze/mw-config] 07Universal-Omega 03f8545f3 - Delete ISSUE_REPLY_TEMPLATE.md [16:09:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqZ2B [16:09:15] [02miraheze/mw-config] 07Universal-Omega 03d91c6d4 - Delete ISSUE_TEMPLATE.md [16:09:49] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqZ2u [16:09:51] [02miraheze/MirahezeMagic] 07Universal-Omega 03ac8e52c - Delete ISSUE_TEMPLATE.md [16:10:16] miraheze/mw-config - Universal-Omega the build passed. [16:10:22] miraheze/mw-config - Universal-Omega the build passed. [16:10:56] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:13:36] [02dns] 07Universal-Omega opened pull request 03#198: Delete ISSUE_REPLY_TEMPLATE.md and ISSUE_TEMPLATE.md - 13https://git.io/JqZ2i [16:14:34] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqZ2P [16:14:36] [02miraheze/landing] 07Universal-Omega 03e8db44b - Delete ISSUE_REPLY_TEMPLATE.md [16:14:42] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqZ2M [16:14:43] [02miraheze/landing] 07Universal-Omega 0311f2bac - Delete ISSUE_TEMPLATE.md [16:15:41] miraheze/landing - Universal-Omega the build passed. [16:15:58] [02dns] 07Reception123 closed pull request 03#198: Delete ISSUE_REPLY_TEMPLATE.md and ISSUE_TEMPLATE.md - 13https://git.io/JqZ2i [16:16:00] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-2/±0] 13https://git.io/JqZ29 [16:16:01] [02miraheze/dns] 07Universal-Omega 033ef7ef0 - Delete ISSUE_REPLY_TEMPLATE.md and ISSUE_TEMPLATE.md (#198) [16:16:36] miraheze/landing - Universal-Omega the build passed. [16:18:25] [02puppet] 07Universal-Omega opened pull request 03#1697: Delete ISSUE_REPLY_TEMPLATE.md and ISSUE_TEMPLATE.md - 13https://git.io/JqZ2j [16:19:13] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-2/±0] 13https://git.io/JqZaJ [16:19:14] [02miraheze/puppet] 07Universal-Omega 03a13b82b - Delete ISSUE_REPLY_TEMPLATE.md and ISSUE_TEMPLATE.md (#1697) [16:19:16] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-2/±0] 13https://git.io/JqZaJ [16:19:17] [02miraheze/puppet] 07Universal-Omega 03a13b82b - Delete ISSUE_REPLY_TEMPLATE.md and ISSUE_TEMPLATE.md (#1697) [16:19:19] [02puppet] 07Reception123 closed pull request 03#1697: Delete ISSUE_REPLY_TEMPLATE.md and ISSUE_TEMPLATE.md - 13https://git.io/JqZ2j [16:19:55] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03aiimplement [+0/-0/±1] 13https://git.io/JqZak [16:19:56] [02miraheze/CreateWiki] 07JohnFLewis 03b78552c - Check against Threshold Variable + support job only [16:19:58] [02CreateWiki] 07JohnFLewis created branch 03aiimplement - 13https://git.io/vpJTL [16:21:03] !log install python3-yaml [16:21:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:21:07] on jobrunner3 [16:21:28] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03aiimplement [+0/-0/±1] 13https://git.io/JqZat [16:21:30] [02miraheze/CreateWiki] 07JohnFLewis 03467e36f - add config [16:25:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZaB [16:25:11] [02miraheze/services] 07MirahezeSSLBot 032c3af8c - BOT: Updating services config for wikis [16:25:34] paladox, I just realized your Meta user page says the same thing as your global user page on Loginwiki except a bit out of date. Would you like me to delete your Meta user page for you? [16:25:47] sure [16:25:50] ok [16:25:59] thanks [16:27:08] Reception123 around? [16:27:44] np [16:29:19] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZao [16:29:21] [02miraheze/ssl] 07paladox 03bb88faa - Disable events for wildcard.bots.miraheze.wiki [16:29:40] paladox: yes :) [16:29:57] Reception123 i'm guessing wildcard.bots.miraheze.wiki was generated by rhinos on their server? [16:30:12] I'm not sure, but I would imagine that yeah [16:31:44] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03aiimplement [+0/-0/±1] 13https://git.io/JqZaX [16:31:46] [02miraheze/CreateWiki] 07JohnFLewis 037019456 - add AI code to AI job with ability to auto-approve [16:32:54] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03aiimplement [+0/-0/±1] 13https://git.io/JqZa9 [16:32:56] [02miraheze/CreateWiki] 07JohnFLewis 03be5181e - fix [16:33:27] [02CreateWiki] 07JohnFLewis opened pull request 03#200: AI Implementation - 13https://git.io/JqZaH [16:34:30] miraheze/CreateWiki - JohnFLewis the build has errored. [16:35:11] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03aiimplement [+0/-0/±1] 13https://git.io/JqZab [16:35:13] [02miraheze/CreateWiki] 07JohnFLewis 031aad9da - fix [16:35:14] [02CreateWiki] 07JohnFLewis synchronize pull request 03#200: AI Implementation - 13https://git.io/JqZaH [16:36:20] miraheze/CreateWiki - JohnFLewis the build has errored. [16:36:27] [02CreateWiki] 07JohnFLewis synchronize pull request 03#200: AI Implementation - 13https://git.io/JqZaH [16:36:29] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03aiimplement [+0/-0/±1] 13https://git.io/JqZVT [16:36:30] [02miraheze/CreateWiki] 07JohnFLewis 032a3d555 - third fix, c&p is bad [16:37:30] miraheze/CreateWiki - JohnFLewis the build passed. [16:42:33] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03aiimplement [+0/-0/±1] 13https://git.io/JqZVC [16:42:34] [02miraheze/CreateWiki] 07JohnFLewis 037e8aa42 - add double if for safeguarding [16:42:36] [02CreateWiki] 07JohnFLewis synchronize pull request 03#200: AI Implementation - 13https://git.io/JqZaH [16:43:10] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZVW [16:43:11] [02miraheze/ssl] 07paladox 0318d930d - Disable events for a few of bots.miraheze.wiki sub domains [16:43:40] miraheze/CreateWiki - JohnFLewis the build passed. [16:47:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqZVa [16:47:35] [02miraheze/mw-config] 07Universal-Omega 030bb558b - Add documentation to CONTRIBUTING.md for updating MWN/MWS/MWE [16:47:36] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [16:47:38] [02mw-config] 07Universal-Omega opened pull request 03#3769: Add documentation to CONTRIBUTING.md for updating MWN/MWS/MWE - 13https://git.io/JqZVV [16:48:51] miraheze/mw-config - Universal-Omega the build passed. [16:48:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±261] 13https://git.io/JqZVr [16:48:57] [02miraheze/ssl] 07MirahezeSSLBot 03550d47d - Bot: Update SSL cert [16:49:43] oh [16:50:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±261] 13https://git.io/JqZVo [16:50:11] [02miraheze/ssl] 07paladox 030c4f732 - Revert "Bot: Update SSL cert" [16:51:37] [02mw-config] 07Universal-Omega closed pull request 03#3769: Add documentation to CONTRIBUTING.md for updating MWN/MWS/MWE - 13https://git.io/JqZVV [16:51:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZVX [16:51:40] [02miraheze/mw-config] 07Universal-Omega 0356864be - Add documentation to CONTRIBUTING.md for updating MWN/MWS/MWE (#3769) [16:51:41] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [16:51:43] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:52:41] miraheze/mw-config - Universal-Omega the build passed. [16:54:19] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:54:55] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:55:03] assuming ^ is expected [16:55:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZVp [16:55:52] [02miraheze/mw-config] 07Universal-Omega 0334fe92a - Add one more documentation point for extensions [16:56:51] miraheze/mw-config - Universal-Omega the build passed. [17:03:26] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZw3 [17:03:28] [02miraheze/dns] 07Reception123 039df3f38 - add two subdomains to pornwiki.org [17:09:54] After trying to delete and replace an OPK file on my wiki (https://wiki.funkey-project.com/wiki) this morning, I was having issues uploading OPK files. When trying to get the error message again a few minutes ago, however, the uploads seem to be working again, so perhaps it's an issue with trying to delete the non-standard OPK files? [17:09:56] [ The FunKey Wiki ] - wiki.funkey-project.com [17:13:02] @Coolie, not sure [17:13:49] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±4] 13https://git.io/JqZwh [17:13:50] [02miraheze/ssl] 07Reception123 03985bc6a - *.pornwiki.org changes [17:13:52] [02miraheze/ssl] 07Reception123 033229ee3 - *.pornwiki.org changes [17:14:45] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZrv [17:14:46] [02miraheze/ssl] 07Reception123 0340bc744 - fix [17:22:37] PROBLEM - mon2 Check correctness of the icinga configuration on mon2 is CRITICAL: Icinga configuration contains errors [17:23:27] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[icinga2] [17:30:59] When attempting to delete an OPK file, I get this message: "A database query error has occurred. This may indicate a bug in the software. [77c56c552f35aaec555fa775] 2021-03-10 17:30:29: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"" [17:36:37] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[files.pornwiki.org],Service[nginx] [17:39:00] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[files.pornwiki.org] [17:45:10] @Coolie, oh, that's helpful. Thanks for posting that. [17:45:26] @Universal_Omega, can you trace that error ^ when you get a chance? [17:46:33] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[files.pornwiki.org] [17:46:38] hmm [17:47:34] I know Reception123 was adding the DNS entries for that wiki's domain; not sure if it's related or what the issue is [17:47:53] definitely is related [17:47:59] (not to the DNS entries, but to SSL) [17:48:03] but I wonder what I did wrong this time [17:48:04] ah [17:48:05] [02mw-config] 07Reception123 commented on pull request 03#3761: Better canned responses - 13https://git.io/JqZoi [17:48:09] hrm [17:48:19] nginx: [emerg] BIO_new_file("/etc/ssl/localcerts/files.pornwiki.org.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/ssl/localcerts/files.pornwiki.org.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file) [17:48:20] Reception123 [17:48:42] oh [17:48:45] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/JqZoy [17:48:47] [02miraheze/ssl] 07paladox 0351b19e0 - Fix [17:48:48] paladox: forgot the .crt yeah [17:48:51] is that the private cert file? [17:48:55] thanks for fixing that :) [17:49:05] I've made that mistake a few times :( [17:49:26] dmehus: nope, it's the public cert, forgot to add .crt to the file name [17:49:34] oh easy one to fix, though. almost like a typo fix, which paladox likes fixing [17:49:40] Reception123, ah [17:49:40] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:50:21] yeah, he's great at finding those [17:50:56] I think Reception123 needs a Master of the Missing `.crt` Barnstar on his Meta talk page :P [17:51:13] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:51:16] heh, it's probably happened 3 or 4 times :P [17:51:23] heh [17:51:44] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZoh [17:51:45] [02miraheze/ssl] 07paladox 0383459f5 - Disable events for wildcard.bots.miraheze.wiki [17:52:37] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:53:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±261] 13https://git.io/JqZKY [17:53:47] [02miraheze/ssl] 07MirahezeSSLBot 037e0aa5d - Bot: Update SSL certs [17:53:51] Reception123, what does `disable_event` do? I've meant to ask that actually [17:55:32] dmehus: it's an Icinga thing [17:55:33] https://github.com/miraheze/puppet/blob/2815534395f49982fc818843e840dca4f38de000/modules/monitoring/templates/ssl.conf.erb [17:55:33] [ puppet/ssl.conf.erb at 2815534395f49982fc818843e840dca4f38de000 · miraheze/puppet · GitHub ] - github.com [17:56:10] Reception123, ah, that makes sense. Thanks! :) [17:56:21] Reception123: have you seen OVH Strausburg last night? [17:56:35] yes [17:56:44] what happened, RhinosF1? [17:56:56] dmehus: https://mobile.twitter.com/olesovhcom/status/1369665936379281409 [17:56:58] dmehus: there was a fire :( [17:57:01] A fire [17:57:04] RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Tue 25 May 2021 00:49:39 GMT +0000. [17:57:04] RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Tue 25 May 2021 00:44:22 GMT +0000. [17:57:05] RECOVERY - factorio-mods.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'factorio-mods.tk' will expire on Tue 25 May 2021 00:41:29 GMT +0000. [17:57:05] RECOVERY - munwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'munwiki.info' will expire on Tue 25 May 2021 00:48:39 GMT +0000. [17:57:07] RECOVERY - wikilukas.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wikilukas.tk' will expire on Tue 25 May 2021 19:21:17 GMT +0000. [17:57:08] RECOVERY - wiki.mnzp.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mnzp.xyz' will expire on Mon 07 Jun 2021 23:38:39 GMT +0000. [17:57:08] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Tue 25 May 2021 00:56:04 GMT +0000. [17:57:09] RECOVERY - planetnomads.speedbomb.de - LetsEncrypt on sslhost is OK: OK - Certificate 'planetnomads.speedbomb.de' will expire on Tue 25 May 2021 00:51:13 GMT +0000. [17:57:12] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Tue 25 May 2021 00:53:37 GMT +0000. [17:57:12] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Mon 07 Jun 2021 23:32:18 GMT +0000. [17:57:13] RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Tue 25 May 2021 01:16:43 GMT +0000. [17:57:13] RECOVERY - meta.nocyclo.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.nocyclo.tk' will expire on Tue 25 May 2021 00:47:22 GMT +0000. [17:57:14] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Tue 25 May 2021 01:17:24 GMT +0000. [17:57:15] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Tue 25 May 2021 19:15:50 GMT +0000. [17:57:16] RECOVERY - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'zw.fontainebleau-avon.fr' will expire on Tue 25 May 2021 19:26:25 GMT +0000. [17:57:16] RECOVERY - wikibeats.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikibeats.net' will expire on Tue 25 May 2021 19:20:54 GMT +0000. [17:57:17] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Tue 25 May 2021 01:08:32 GMT +0000. [17:57:17] RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Mon 07 Jun 2021 22:53:32 GMT +0000. [17:57:22] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Tue 25 May 2021 01:09:52 GMT +0000. [17:57:24] RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Mon 07 Jun 2021 22:53:01 GMT +0000. [17:57:27] RECOVERY - www.wikimicrofinanza.it - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikimicrofinanza.it' will expire on Mon 07 Jun 2021 22:51:41 GMT +0000. [17:57:28] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Tue 25 May 2021 00:55:01 GMT +0000. [17:57:28] RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Mon 07 Jun 2021 22:51:13 GMT +0000. [17:57:29] wow now icinga-miraheze wakes up just as we were discussing disable_event [17:57:29] RECOVERY - wiki.jafaron.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jafaron.net' will expire on Tue 25 May 2021 01:07:53 GMT +0000. [17:57:29] RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Tue 25 May 2021 00:46:00 GMT +0000. [17:57:32] RECOVERY - nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Tue 25 May 2021 00:49:01 GMT +0000. [17:57:32] now that's a funny coincidence [17:57:35] RECOVERY - wiki.danvs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Tue 25 May 2021 00:59:46 GMT +0000. [17:57:35] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Tue 25 May 2021 00:59:24 GMT +0000. [17:57:36] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 25 May 2021 00:46:51 GMT +0000. [17:57:36] RECOVERY - madgenderscience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madgenderscience.wiki' will expire on Tue 25 May 2021 00:46:34 GMT +0000. [17:57:36] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Tue 25 May 2021 00:36:31 GMT +0000. [17:57:37] RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Mon 07 Jun 2021 23:09:54 GMT +0000. [17:57:39] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Tue 25 May 2021 00:51:24 GMT +0000. [17:57:39] Reception123: we really should consider disaster recovery plans [17:57:40] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Tue 25 May 2021 00:52:44 GMT +0000. [17:57:43] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Tue 25 May 2021 01:11:19 GMT +0000. [17:57:43] RECOVERY - wikien.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Tue 25 May 2021 19:21:01 GMT +0000. [17:57:44] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Tue 25 May 2021 00:49:27 GMT +0000. [17:57:44] Reception123 that's me [17:57:44] RECOVERY - wiki.ssij.se - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssij.se' will expire on Tue 25 May 2021 01:14:53 GMT +0000. [17:57:45] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Tue 25 May 2021 00:57:27 GMT +0000. [17:57:46] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Tue 25 May 2021 01:10:28 GMT +0000. [17:57:48] ah ok [17:57:50] RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 25 May 2021 00:36:07 GMT +0000. [17:57:55] RECOVERY - wiki.helioss.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.helioss.co' will expire on Tue 25 May 2021 01:05:42 GMT +0000. [17:57:55] RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Mon 07 Jun 2021 23:04:40 GMT +0000. [17:57:58] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Tue 25 May 2021 00:38:10 GMT +0000. [17:57:59] RECOVERY - wiki.arnab.at - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.arnab.at' will expire on Tue 25 May 2021 00:57:55 GMT +0000. [17:58:05] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Tue 25 May 2021 19:21:37 GMT +0000. [17:58:06] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Tue 25 May 2021 00:59:39 GMT +0000. [17:58:08] RECOVERY - wiki.mussa.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mussa.id' will expire on Tue 25 May 2021 01:11:44 GMT +0000. [17:58:09] RECOVERY - halloween.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'halloween.gyaanipedia.com' will expire on Tue 25 May 2021 00:42:56 GMT +0000. [17:58:09] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Tue 25 May 2021 00:52:00 GMT +0000. [17:58:11] RECOVERY - es.nocyclo.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'es.nocyclo.tk' will expire on Tue 25 May 2021 00:41:07 GMT +0000. [17:58:13] RECOVERY - www.opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Tue 25 May 2021 00:49:47 GMT +0000. [17:58:13] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Tue 25 May 2021 00:53:59 GMT +0000. [17:58:16] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Tue 25 May 2021 01:13:32 GMT +0000. [17:58:19] RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pornwiki.org' will expire on Mon 07 Jun 2021 23:08:24 GMT +0000. [17:58:20] RECOVERY - de.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'de.gyaanipedia.com' will expire on Tue 25 May 2021 00:38:32 GMT +0000. [17:58:22] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Tue 25 May 2021 01:14:35 GMT +0000. [17:58:22] RECOVERY - wiki.responsibly.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.responsibly.ai' will expire on Tue 25 May 2021 01:13:26 GMT +0000. [17:58:23] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Tue 08 Jun 2021 01:02:24 GMT +0000. [17:58:23] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Tue 25 May 2021 00:43:20 GMT +0000. [17:58:25] RECOVERY - www.johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Mon 07 Jun 2021 22:53:32 GMT +0000. [17:58:25] RECOVERY - privacy-wiki.0x.no - LetsEncrypt on sslhost is OK: OK - Certificate 'privacy-wiki.0x.no' will expire on Tue 25 May 2021 00:51:52 GMT +0000. [17:58:27] RECOVERY - lama.madooa.org - LetsEncrypt on sslhost is OK: OK - Certificate 'lama.madooa.org' will expire on Tue 25 May 2021 00:46:20 GMT +0000. [17:58:29] RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worlduniversityandschool.org' will expire on Tue 25 May 2021 01:18:37 GMT +0000. [17:58:29] RECOVERY - petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.petrawiki.org' will expire on Mon 07 Jun 2021 23:19:10 GMT +0000. [17:58:31] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Tue 25 May 2021 01:04:40 GMT +0000. [17:58:31] RECOVERY - wiki.invinciblewellbeing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.invinciblewellbeing.com' will expire on Tue 25 May 2021 01:06:40 GMT +0000. [17:58:31] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Tue 25 May 2021 19:21:24 GMT +0000. [17:58:35] RECOVERY - wiki.seredrau.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seredrau.xyz' will expire on Tue 25 May 2021 01:14:20 GMT +0000. [17:58:37] RECOVERY - techwiki.techboyg5blog.com - LetsEncrypt on sslhost is OK: OK - Certificate 'techwiki.techboyg5blog.com' will expire on Tue 25 May 2021 00:54:20 GMT +0000. [17:58:42] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Mon 07 Jun 2021 22:52:26 GMT +0000. [17:58:46] RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Tue 25 May 2021 00:43:35 GMT +0000. [17:58:47] RECOVERY - wiki.iufs.jp - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.iufs.jp' will expire on Tue 25 May 2021 01:07:29 GMT +0000. [17:58:48] RECOVERY - wiki.abyogsms.org.in - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.abyogsms.org.in' will expire on Tue 25 May 2021 00:57:15 GMT +0000. [17:58:52] RECOVERY - en.nocyclo.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'en.nocyclo.tk' will expire on Tue 25 May 2021 00:39:47 GMT +0000. [17:58:56] RECOVERY - vortekspedia.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'vortekspedia.tk' will expire on Tue 25 May 2021 00:56:46 GMT +0000. [17:58:56] RECOVERY - froggy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'froggy.info' will expire on Tue 25 May 2021 00:42:17 GMT +0000. [17:58:58] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:58:59] RECOVERY - debilepedie.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'debilepedie.gq' will expire on Tue 25 May 2021 00:38:40 GMT +0000. [17:59:00] RECOVERY - ndg.nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Tue 25 May 2021 00:49:01 GMT +0000. [17:59:00] RECOVERY - warmage.necrocity.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'warmage.necrocity.co.uk' will expire on Tue 25 May 2021 00:56:58 GMT +0000. [17:59:02] RECOVERY - en.phgalaxy.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'en.phgalaxy.xyz' will expire on Tue 25 May 2021 00:40:07 GMT +0000. [17:59:09] RECOVERY - www.lukaibrisimovic.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lukaibrisimovic.org' will expire on Mon 07 Jun 2021 23:23:21 GMT +0000. [17:59:12] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Tue 25 May 2021 01:10:58 GMT +0000. [17:59:12] RECOVERY - tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tradwiki.org' will expire on Tue 25 May 2021 00:55:47 GMT +0000. [17:59:13] RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Tue 25 May 2021 00:38:18 GMT +0000. [17:59:15] RECOVERY - monarchia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchia.wiki' will expire on Tue 25 May 2021 00:48:09 GMT +0000. [17:59:15] RECOVERY - miraheze.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.tk' will expire on Tue 25 May 2021 00:48:02 GMT +0000. [17:59:19] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Tue 25 May 2021 00:53:44 GMT +0000. [17:59:21] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Tue 25 May 2021 00:43:58 GMT +0000. [17:59:22] RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Tue 25 May 2021 00:48:46 GMT +0000. [17:59:23] RECOVERY - wiki.etn.link - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.etn.link' will expire on Tue 25 May 2021 19:15:57 GMT +0000. [17:59:26] RECOVERY - kk.uncyc.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'kk.uncyc.tk' will expire on Tue 25 May 2021 00:45:52 GMT +0000. [17:59:28] RhinosF1, Reception123, oh interesting [17:59:30] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Tue 25 May 2021 00:56:24 GMT +0000. [17:59:31] RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Mon 07 Jun 2021 23:43:01 GMT +0000. [17:59:32] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Tue 25 May 2021 19:20:41 GMT +0000. [17:59:33] RECOVERY - wiki.sdbx.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sdbx.dev' will expire on Tue 25 May 2021 01:14:00 GMT +0000. [17:59:37] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Tue 25 May 2021 00:42:37 GMT +0000. [17:59:38] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Tue 25 May 2021 00:38:03 GMT +0000. [17:59:40] RECOVERY - www.baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Mon 07 Jun 2021 23:09:54 GMT +0000. [17:59:41] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Tue 25 May 2021 01:10:02 GMT +0000. [17:59:43] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Tue 25 May 2021 00:36:00 GMT +0000. [17:59:43] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Tue 25 May 2021 01:17:42 GMT +0000. [17:59:44] Fire in a data centre = definitely not good and a critical incident [17:59:47] RECOVERY - wiki.cdntennis.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cdntennis.ca' will expire on Tue 25 May 2021 00:58:53 GMT +0000. [17:59:48] RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Tue 25 May 2021 00:37:53 GMT +0000. [17:59:51] RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Tue 25 May 2021 00:41:39 GMT +0000. [17:59:52] RECOVERY - www.sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Tue 08 Jun 2021 01:02:24 GMT +0000. [17:59:52] RECOVERY - wiki.redeemer.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.redeemer.live' will expire on Tue 25 May 2021 01:13:18 GMT +0000. [17:59:55] RECOVERY - wiki.ripto.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ripto.gq' will expire on Tue 25 May 2021 01:13:40 GMT +0000. [18:00:05] RECOVERY - wiki.fourta.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fourta.org' will expire on Tue 25 May 2021 19:16:04 GMT +0000. [18:00:06] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Tue 25 May 2021 00:56:38 GMT +0000. [18:00:06] RECOVERY - wiki.kali-team.cn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kali-team.cn' will expire on Tue 25 May 2021 01:08:20 GMT +0000. [18:00:12] RECOVERY - wiki.openhatch.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Tue 25 May 2021 01:12:18 GMT +0000. [18:00:12] RECOVERY - wiki.funkey-project.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.funkey-project.com' will expire on Tue 25 May 2021 01:03:42 GMT +0000. [18:00:13] RECOVERY - wiki.azimuth.navy - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.azimuth.navy' will expire on Tue 25 May 2021 00:58:19 GMT +0000. [18:00:14] RECOVERY - iesd.mobagenie.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'iesd.mobagenie.my.id' will expire on Tue 25 May 2021 00:44:15 GMT +0000. [18:00:16] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 25 May 2021 00:52:32 GMT +0000. [18:00:17] RECOVERY - studentwiki.ddns.net - LetsEncrypt on sslhost is OK: OK - Certificate 'studentwiki.ddns.net' will expire on Tue 25 May 2021 00:54:13 GMT +0000. [18:00:18] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Tue 25 May 2021 01:09:45 GMT +0000. [18:00:18] dmehus: quite a lot of damage. They're expecting up to 2 weeks downtime [18:00:18] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Tue 25 May 2021 01:16:17 GMT +0000. [18:00:19] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 25 May 2021 00:50:58 GMT +0000. [18:00:21] RECOVERY - viileapedia.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'viileapedia.ga' will expire on Tue 25 May 2021 00:56:11 GMT +0000. [18:00:22] RECOVERY - savagepedia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'savagepedia.wiki' will expire on Tue 25 May 2021 00:53:12 GMT +0000. [18:00:24] RECOVERY - wiki.patriam.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.patriam.cc' will expire on Tue 25 May 2021 01:12:45 GMT +0000. [18:00:26] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Tue 25 May 2021 00:36:16 GMT +0000. [18:00:27] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Tue 25 May 2021 00:54:06 GMT +0000. [18:00:28] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Mon 07 Jun 2021 22:52:38 GMT +0000. [18:00:28] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Tue 25 May 2021 00:48:31 GMT +0000. [18:00:28] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Tue 25 May 2021 01:11:37 GMT +0000. [18:00:32] RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Tue 25 May 2021 00:47:55 GMT +0000. [18:00:33] RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 25 May 2021 00:36:07 GMT +0000. [18:00:33] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 25 May 2021 00:47:30 GMT +0000. [18:00:33] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Tue 25 May 2021 00:46:13 GMT +0000. [18:00:35] RECOVERY - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beaconspace.unrestrictedlorefare.com' will expire on Tue 25 May 2021 00:36:55 GMT +0000. [18:00:36] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Tue 25 May 2021 00:52:00 GMT +0000. [18:00:37] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Tue 25 May 2021 00:53:51 GMT +0000. [18:00:38] RECOVERY - wiki.seamly.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.net' will expire on Tue 25 May 2021 01:14:07 GMT +0000. [18:00:41] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 25 May 2021 19:25:57 GMT +0000. [18:00:44] RECOVERY - indiancannabis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'indiancannabis.wiki' will expire on Tue 25 May 2021 00:44:30 GMT +0000. [18:00:48] RECOVERY - idleon.info - LetsEncrypt on sslhost is OK: OK - Certificate 'idleon.info' will expire on Tue 25 May 2021 00:44:05 GMT +0000. [18:00:50] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 25 May 2021 19:25:57 GMT +0000. [18:00:51] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Tue 25 May 2021 00:54:36 GMT +0000. [18:00:52] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Tue 25 May 2021 00:36:24 GMT +0000. [18:00:52] RECOVERY - wiki.xysspon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xysspon.com' will expire on Tue 25 May 2021 19:20:13 GMT +0000. [18:00:54] RECOVERY - towarzystwo.prawa.kredytowego.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'towarzystwo.prawa.kredytowego.pl' will expire on Tue 08 Jun 2021 00:44:38 GMT +0000. [18:01:00] RECOVERY - wiki.velaan.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.velaan.org' will expire on Tue 25 May 2021 01:17:31 GMT +0000. [18:01:00] RECOVERY - wiki.twilightsignal.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.twilightsignal.com' will expire on Tue 25 May 2021 01:16:57 GMT +0000. [18:01:01] RECOVERY - wiki.campaign-labour.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.campaign-labour.org' will expire on Mon 07 Jun 2021 23:48:54 GMT +0000. [18:01:01] RECOVERY - wiki.dom64.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dom64.xyz' will expire on Tue 25 May 2021 00:59:54 GMT +0000. [18:01:02] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Tue 25 May 2021 01:15:48 GMT +0000. [18:01:03] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Tue 25 May 2021 01:10:51 GMT +0000. [18:01:04] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Tue 25 May 2021 00:38:03 GMT +0000. [18:01:06] RECOVERY - wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Tue 25 May 2021 19:21:09 GMT +0000. [18:01:06] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Tue 25 May 2021 01:16:50 GMT +0000. [18:01:07] RECOVERY - bharatwiki.online - LetsEncrypt on sslhost is OK: OK - Certificate 'bharatwiki.online' will expire on Tue 25 May 2021 00:37:15 GMT +0000. [18:01:11] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Tue 25 May 2021 01:08:00 GMT +0000. [18:01:13] RECOVERY - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pablojprieto.com.es' will expire on Tue 25 May 2021 01:12:33 GMT +0000. [18:01:13] RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Mon 07 Jun 2021 22:48:26 GMT +0000. [18:01:14] RECOVERY - bluevies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluevies.org' will expire on Mon 07 Jun 2021 23:20:59 GMT +0000. [18:01:15] RECOVERY - wiki.teamrelectric.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teamrelectric.ca' will expire on Tue 25 May 2021 01:15:36 GMT +0000. [18:01:19] RECOVERY - www.fraterniterebellion.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fraterniterebellion.com' will expire on Tue 25 May 2021 01:03:16 GMT +0000. [18:01:20] RECOVERY - pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pornwiki.org' will expire on Mon 07 Jun 2021 23:08:24 GMT +0000. [18:01:26] RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Mon 07 Jun 2021 23:25:08 GMT +0000. [18:01:26] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Tue 25 May 2021 01:18:06 GMT +0000. [18:01:27] RECOVERY - wiki.yunachannel.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yunachannel.com' will expire on Tue 25 May 2021 19:20:35 GMT +0000. [18:01:36] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Tue 25 May 2021 01:04:48 GMT +0000. [18:01:36] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Tue 25 May 2021 00:48:17 GMT +0000. [18:01:37] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 25 May 2021 00:46:51 GMT +0000. [18:01:38] RECOVERY - erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com' will expire on Mon 07 Jun 2021 23:16:38 GMT +0000. [18:01:40] RECOVERY - wiki.xfbs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xfbs.net' will expire on Tue 25 May 2021 19:20:06 GMT +0000. [18:01:42] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Tue 25 May 2021 00:47:48 GMT +0000. [18:01:46] RECOVERY - franchise.franchising.org.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'franchise.franchising.org.ua' will expire on Tue 25 May 2021 00:42:04 GMT +0000. [18:01:47] RECOVERY - wiki.usagihime.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.usagihime.ml' will expire on Tue 25 May 2021 01:17:16 GMT +0000. [18:01:47] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Mon 07 Jun 2021 23:30:57 GMT +0000. [18:01:49] RECOVERY - lukaibrisimovic.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lukaibrisimovic.org' will expire on Mon 07 Jun 2021 23:23:21 GMT +0000. [18:01:50] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Tue 25 May 2021 01:11:57 GMT +0000. [18:01:50] RECOVERY - www.tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tradwiki.org' will expire on Tue 25 May 2021 00:55:47 GMT +0000. [18:01:56] RECOVERY - guia.esporo.net - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.esporo.net' will expire on Tue 25 May 2021 00:42:44 GMT +0000. [18:02:00] RhinosF1, ack, yeah I saw that. 10,000 new servers to be installed in one of their data centres, too [18:02:02] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Tue 25 May 2021 01:13:11 GMT +0000. [18:02:03] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Tue 25 May 2021 00:59:32 GMT +0000. [18:02:03] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Tue 25 May 2021 00:52:25 GMT +0000. [18:02:07] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Tue 25 May 2021 01:09:24 GMT +0000. [18:02:10] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Tue 25 May 2021 01:12:57 GMT +0000. [18:02:11] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Tue 25 May 2021 01:18:13 GMT +0000. [18:02:12] RECOVERY - housing.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'housing.wiki' will expire on Tue 25 May 2021 00:43:50 GMT +0000. [18:02:18] RECOVERY - www.bluevies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluevies.org' will expire on Mon 07 Jun 2021 23:20:59 GMT +0000. [18:02:21] RECOVERY - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jacksonheights.nyc' will expire on Tue 25 May 2021 01:07:46 GMT +0000. [18:02:21] RECOVERY - wiki.hrznstudio.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hrznstudio.com' will expire on Mon 07 Jun 2021 23:36:21 GMT +0000. [18:02:21] RECOVERY - es.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Tue 25 May 2021 00:52:18 GMT +0000. [18:02:22] RECOVERY - mediawiki.ecfin.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'mediawiki.ecfin.eu' will expire on Tue 25 May 2021 00:47:01 GMT +0000. [18:02:29] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Mon 07 Jun 2021 22:52:38 GMT +0000. [18:02:30] RECOVERY - wiki.candela.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.candela.digital' will expire on Tue 25 May 2021 00:58:41 GMT +0000. [18:02:31] RECOVERY - wiki.fraterniterebellion.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fraterniterebellion.com' will expire on Tue 25 May 2021 01:03:16 GMT +0000. [18:02:31] RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Tue 25 May 2021 00:44:37 GMT +0000. [18:02:32] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Tue 25 May 2021 00:40:48 GMT +0000. [18:02:36] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Mon 07 Jun 2021 22:52:26 GMT +0000. [18:02:37] RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Mon 07 Jun 2021 23:59:12 GMT +0000. [18:02:38] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Tue 25 May 2021 00:49:20 GMT +0000. [18:02:39] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Mon 07 Jun 2021 22:53:18 GMT +0000. [18:02:41] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Tue 25 May 2021 01:13:46 GMT +0000. [18:02:43] RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 25 May 2021 00:47:30 GMT +0000. [18:02:48] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Tue 25 May 2021 00:50:39 GMT +0000. [18:02:48] RECOVERY - wiki.ostans.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ostans.com' will expire on Tue 25 May 2021 01:12:25 GMT +0000. [18:02:55] RECOVERY - ipv6bolivia.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'ipv6bolivia.tk' will expire on Tue 25 May 2021 00:44:56 GMT +0000. [18:02:56] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Tue 25 May 2021 00:47:41 GMT +0000. [18:02:58] RECOVERY - www.wikilukas.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wikilukas.tk' will expire on Tue 25 May 2021 19:21:17 GMT +0000. [18:03:02] RECOVERY - madzebrascience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madzebrascience.wiki' will expire on Tue 25 May 2021 00:46:42 GMT +0000. [18:03:02] RECOVERY - civclassic.civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civclassic.civwiki.org' will expire on Tue 25 May 2021 00:37:31 GMT +0000. [18:03:04] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Tue 25 May 2021 00:59:12 GMT +0000. [18:03:09] RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Tue 25 May 2021 00:43:27 GMT +0000. [18:03:09] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'es.gyaanipedia.com' will expire on Tue 25 May 2021 00:41:00 GMT +0000. [18:03:10] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Mon 07 Jun 2021 23:54:57 GMT +0000. [18:03:11] RECOVERY - wiki.raghuveer.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.raghuveer.net' will expire on Tue 25 May 2021 01:13:04 GMT +0000. [18:03:11] RECOVERY - innersphere.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'innersphere.tk' will expire on Tue 25 May 2021 00:44:44 GMT +0000. [18:03:13] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Tue 25 May 2021 00:39:08 GMT +0000. [18:03:13] RECOVERY - battlecats.f5.si - LetsEncrypt on sslhost is OK: OK - Certificate 'battlecats.f5.si' will expire on Tue 25 May 2021 00:36:48 GMT +0000. [18:03:15] RECOVERY - wiki.thehall.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thehall.xyz' will expire on Tue 25 May 2021 01:16:00 GMT +0000. [18:03:15] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Tue 25 May 2021 01:11:12 GMT +0000. [18:03:15] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Tue 25 May 2021 01:11:50 GMT +0000. [18:03:16] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Tue 25 May 2021 00:59:04 GMT +0000. [18:03:20] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Mon 07 Jun 2021 23:04:40 GMT +0000. [18:03:21] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Tue 25 May 2021 00:37:05 GMT +0000. [18:03:22] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Tue 25 May 2021 00:55:08 GMT +0000. [18:03:24] RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Tue 25 May 2021 00:36:38 GMT +0000. [18:03:25] RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Tue 25 May 2021 00:56:31 GMT +0000. [18:03:30] RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Tue 25 May 2021 00:52:18 GMT +0000. [18:03:30] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Tue 25 May 2021 00:41:56 GMT +0000. [18:03:30] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Tue 25 May 2021 01:13:54 GMT +0000. [18:03:34] RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Tue 25 May 2021 01:15:00 GMT +0000. [18:03:35] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Tue 25 May 2021 00:40:40 GMT +0000. [18:03:37] RECOVERY - www.erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com' will expire on Mon 07 Jun 2021 23:16:38 GMT +0000. [18:03:38] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 25 May 2021 00:50:58 GMT +0000. [18:03:38] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Tue 25 May 2021 00:54:28 GMT +0000. [18:03:39] RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Mon 07 Jun 2021 23:25:08 GMT +0000. [18:03:39] RECOVERY - cubashpedia.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'cubashpedia.cf' will expire on Tue 25 May 2021 00:37:45 GMT +0000. [18:03:40] I wonder what the cause of the fire was, I'd imagine there would be safeguards against that considering they're dealing with server racks [18:03:41] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.com' will expire on Tue 25 May 2021 00:39:41 GMT +0000. [18:03:43] RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lg.gyaanipedia.com' will expire on Tue 25 May 2021 00:46:27 GMT +0000. [18:03:43] RECOVERY - rupowerrangersfanon.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'rupowerrangersfanon.tk' will expire on Tue 25 May 2021 00:52:51 GMT +0000. [18:03:44] RECOVERY - holonet.pw - LetsEncrypt on sslhost is OK: OK - Certificate 'holonet.pw' will expire on Tue 25 May 2021 00:43:42 GMT +0000. [18:03:44] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Tue 25 May 2021 00:49:47 GMT +0000. [18:03:44] RECOVERY - wiki.wysteria.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wysteria.ga' will expire on Tue 25 May 2021 19:19:53 GMT +0000. [18:03:46] RECOVERY - www.wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Tue 25 May 2021 19:21:09 GMT +0000. [18:03:49] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Tue 25 May 2021 00:52:18 GMT +0000. [18:03:52] RECOVERY - chouverse.cludisciples.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'chouverse.cludisciples.cyou' will expire on Tue 25 May 2021 00:37:24 GMT +0000. [18:04:06] !log renerate backups on bacula2 [18:04:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:00] *regenerate [18:05:01] typo [18:05:05] RECOVERY - bacula2 PowerDNS Recursor on bacula2 is OK: DNS OK: 0.421 seconds response time. miraheze.org returns 2607:5300:205:200::1c30,51.222.25.132 [18:05:20] RECOVERY - bacula2 Disk Space on bacula2 is OK: DISK OK - free space: / 780419 MB (81% inode=99%); [18:05:24] Reception123: I'm sure they'll investigate [18:05:51] RECOVERY - dreamsit.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'dreamsit.com.br' will expire on Tue 25 May 2021 00:38:56 GMT +0000. [18:05:54] yeah, though I am curious [18:06:24] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 25 May 2021 19:25:57 GMT +0000. [18:06:32] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:06:37] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:07:13] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:07:30] > I wonder what the cause of the fire was, I'd imagine there would be safeguards against that considering they're dealing with server racks [18:07:31] Yeah... [18:08:23] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [18:15:59] RECOVERY - cp10 Varnish Backends on cp10 is OK: [b'Backend name Admin Probe Last change', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mon2 probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mw8 probe 5/5 good Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mw9 probe 5/5 good Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mw10 probe 5/5 good Sun, 07 Ma [18:15:59] 1 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mw11 probe 5/5 good Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.jobrunner3 probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.test3 probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mw8_test probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a [18:15:59] 9_test probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mw10_test probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mw11_test probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'vcl-998a631b-23aa-432c-8e99-be45ebd38a4a.mediawiki probe healthy Sun, 07 Mar 2021 21:44:45 GMT', b'']All 7 backends are healthy [18:16:19] [02mw-config] 07R4356th commented on pull request 03#3761: Better canned responses - 13https://git.io/JqZ68 [18:17:34] [02mw-config] 07R4356th commented on pull request 03#3761: Better canned responses - 13https://git.io/JqZ6g [18:20:32] > wow now icinga-miraheze wakes up just as we were discussing disable_event [18:20:32] > now that's a funny coincidence [18:20:32] lol, yes! :P [18:21:56] PROBLEM - cp10 Varnish Backends on cp10 is WARNING: No backends detected. If this is an error, see readme.txt [18:23:14] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[icinga2] [18:23:29] Huh [18:23:51] paladox: can you check them 2 alerts? [18:23:55] RECOVERY - cp10 Varnish Backends on cp10 is OK: 1 backends are down. mon22 backends are down. mon2 mw83 backends are down. mon2 mw8 mw94 backends are down. mon2 mw8 mw9 mw105 backends are down. mon2 mw8 mw9 mw10 mw116 backends are down. mon2 mw8 mw9 mw10 mw11 jobrunner37 backends are down. mon2 mw8 mw9 mw10 mw11 jobrunner3 mediawikiAll 7 backends are healthy [18:23:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZiJ [18:23:59] [02miraheze/puppet] 07paladox 0356106df - varnish: Fix check_varnishbackends icinga python script [18:24:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZiJ [18:24:02] [02miraheze/puppet] 07paladox 0356106df - varnish: Fix check_varnishbackends icinga python script [18:24:03] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:24:05] [02puppet] 07paladox opened pull request 03#1698: varnish: Fix check_varnishbackends icinga python script - 13https://git.io/JqZiU [18:24:41] [02mw-config] 07dmehus commented on pull request 03#3761: Better canned responses - 13https://git.io/JqZik [18:25:21] [02puppet] 07paladox closed pull request 03#1698: varnish: Fix check_varnishbackends icinga python script - 13https://git.io/JqZiU [18:25:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZiq [18:25:24] [02miraheze/puppet] 07paladox 03abc285f - varnish: Fix check_varnishbackends icinga python script (#1698) [18:25:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZiq [18:25:27] [02miraheze/puppet] 07paladox 03abc285f - varnish: Fix check_varnishbackends icinga python script (#1698) [18:25:29] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:25:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:25:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:26:26] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [18:26:31] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [18:26:33] RECOVERY - cp11 Varnish Backends on cp11 is OK: All 7 backends are healthy [18:27:14] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:27:35] [02puppet] 07Universal-Omega edited pull request 03#1680: Varnish: Add X-Robots-Tag http header (Should fix T6680) - 13https://git.io/JqTN1 [18:43:37] [02mw-config] 07R4356th commented on pull request 03#3761: Better canned responses - 13https://git.io/JqZP3 [19:00:25] Our cache hit percentage dropped significantly from Feb 25th [19:00:33] From ~50% to ~25% [19:01:08] what's the likely explanation behind that? [19:04:08] Unsure, data suggests it happened between my DNS recursor change and immediately before paladox's change to Varnish [19:04:59] Does DNS stand for Data Net Security? [19:05:02] I think that it would more likely be my change since that directly touches varnish (i'm guessing here) [19:05:30] Domain Name System [19:05:37] Oh. [19:05:52] paladox: potentially, could we revert it and see if it has any change on cache hits? [19:06:16] We can though restarting varnish will fail apparently. [19:06:34] hm? [19:08:11] well when ever a change to default.vcl was done and we notified varnish to restart the service would fail. I think its because we have to change the service type to "simple" rather than "forking". At least the debian version does simple rather than using pids. [19:08:41] including doing "service varnish restart" failed. [19:08:46] i'll work on the revert [19:10:14] actually [19:10:22] JohnLewis https://github.com/miraheze/puppet/commit/da00b371c49638e22182efe8d09f7b9242d80e98#diff-7ced05666d7aaa7ddf3e98ea83352db0830ec45950fbd243a4da816e1cfb170f was done only like 5 days ago [19:10:22] [ varnish: Convert to systemd::service and also fix init script (#1677) · miraheze/puppet@da00b37 · GitHub ] - github.com [19:10:28] https://grafana.miraheze.org/d/LaSguOMZz/varnish?orgId=1&from=now-30d&to=now [19:10:29] [ Grafana ] - grafana.miraheze.org [19:10:49] shows the issue started on the feb 25th [19:10:50] paladox: I said Feb 25th [19:10:54] yeh [19:11:02] https://github.com/miraheze/puppet/commit/19ecf0c2d7888a0f8973efc88a428e85f9c6f126 [19:11:03] [ varnish: Centralise Access-Control-Allow-Origin header (#1663) · miraheze/puppet@19ecf0c · GitHub ] - github.com [19:11:11] oh [19:12:16] Although looking at that change again, I'm unsure how it could have [19:12:36] but it started about 3 hours after I pushed my changes - so there's no correlation [19:13:27] i don't understand how that could cause it tbh [19:13:32] but the times match up [19:17:58] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqZ1u [19:18:00] [02miraheze/CreateWiki] 07Universal-Omega 033b620c5 - Fix echo notifications for comments and declined requests [19:18:01] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [19:18:01] expirations just stop, as if nothing is being cached anymore [19:18:02] [02CreateWiki] 07Universal-Omega opened pull request 03#201: Fix echo notifications for comments and declined requests - 13https://git.io/JqZ1z [19:19:06] miraheze/CreateWiki - Universal-Omega the build passed. [19:20:53] JohnLewis the only thing i could say is remotely related is the removal of "expires 1w;" [19:21:21] but that's nginx, not Varnish [19:22:49] yeh, but the static files will use local web browser cache [19:22:51] for 1k [19:22:54] *1w [19:23:46] Hm, true, so more requests would be served which would account for a reduction is caching hits as more uncached hits are served [19:24:28] i can re-add it for now [19:24:52] aha [19:24:58] Expires controls the caching header [19:25:13] So when it's not present, there's no cache control header - so Varnish treats it as uncachable [19:25:30] It's nothing to do with more requests, it's just Varnish assuming it's uncachable [19:26:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZMq [19:26:15] [02miraheze/puppet] 07paladox 037b259d4 - mediawiki: Add expires field to static.miraheze.org location [19:26:16] Please do re-add it, because it's essential for caching [19:26:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZMq [19:26:18] [02miraheze/puppet] 07paladox 037b259d4 - mediawiki: Add expires field to static.miraheze.org location [19:26:19] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:26:21] [02puppet] 07paladox opened pull request 03#1699: mediawiki: Add expires field to static.miraheze.org location - 13https://git.io/JqZMm [19:27:41] [02puppet] 07paladox closed pull request 03#1699: mediawiki: Add expires field to static.miraheze.org location - 13https://git.io/JqZMm [19:27:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZMn [19:27:43] [02miraheze/puppet] 07paladox 03557bd29 - mediawiki: Add expires field to static.miraheze.org location (#1699) [19:27:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZMn [19:27:46] [02miraheze/puppet] 07paladox 03557bd29 - mediawiki: Add expires field to static.miraheze.org location (#1699) [19:27:48] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:27:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:27:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:28:15] Can't everything on static be cached though? [19:28:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZMB [19:28:32] [02miraheze/puppet] 07paladox 032b406b8 - Add Access-Control-Allow-Origin * for stl & STL [19:28:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZMB [19:28:35] [02miraheze/puppet] 07paladox 032b406b8 - Add Access-Control-Allow-Origin * for stl & STL [19:28:38] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 461440 files, 42.36GB, 2021-03-10 19:27:00 (1.6 minutes ago) [19:29:06] JohnLewis oh! [19:29:42] well not exactly, i don't think we want dumps being cached nor sitemaps? [19:32:24] exclude those directories? [19:32:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZMD [19:32:33] [02miraheze/puppet] 07paladox 03a59e3a4 - mediawiki: Add expires field to all files other than dumps and sitemaps [19:32:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZMD [19:32:36] [02miraheze/puppet] 07paladox 03a59e3a4 - mediawiki: Add expires field to all files other than dumps and sitemaps [19:32:38] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:32:39] [02puppet] 07paladox opened pull request 03#1700: mediawiki: Add expires field to all files other than dumps and sitemaps - 13https://git.io/JqZMy [19:37:58] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:38:29] [02CreateWiki] 07Universal-Omega edited pull request 03#201: Fix echo notifications for comments and declined requests - 13https://git.io/JqZ1z [19:38:50] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:39:20] dmehus: ^ [19:39:48] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:42:28] [02CreateWiki] 07Universal-Omega edited pull request 03#201: Fix echo notifications for comments and declined requests - 13https://git.io/JqZ1z [19:43:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZDZ [19:43:27] [02miraheze/puppet] 07paladox 03f585f32 - Update mediawiki.conf.erb [19:43:29] [02puppet] 07paladox synchronize pull request 03#1700: mediawiki: Add expires field to all files other than dumps and sitemaps - 13https://git.io/JqZMy [19:43:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZDZ [19:43:32] [02miraheze/puppet] 07paladox 03f585f32 - Update mediawiki.conf.erb [19:44:50] JohnLewis https://github.com/miraheze/puppet/pull/1700 [19:44:51] [ mediawiki: Add expires field to all files other than dumps and sitemaps by paladox · Pull Request #1700 · miraheze/puppet · GitHub ] - github.com [19:44:51] does this look good to you? [19:45:30] [02puppet] 07paladox synchronize pull request 03#1700: mediawiki: Add expires field to all files other than dumps and sitemaps - 13https://git.io/JqZMy [19:45:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZDB [19:45:33] [02miraheze/puppet] 07paladox 0312ca6d4 - Update mediawiki.conf.erb [19:45:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZDB [19:45:36] [02miraheze/puppet] 07paladox 0312ca6d4 - Update mediawiki.conf.erb [19:45:59] looks okay [19:46:17] thanks! [19:46:37] [02puppet] 07paladox closed pull request 03#1700: mediawiki: Add expires field to all files other than dumps and sitemaps - 13https://git.io/JqZMy [19:46:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZDu [19:46:40] [02miraheze/puppet] 07paladox 03e698c13 - mediawiki: Add expires field to all files other than dumps and sitemaps (#1700) [19:46:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZDu [19:46:43] [02miraheze/puppet] 07paladox 03e698c13 - mediawiki: Add expires field to all files other than dumps and sitemaps (#1700) [19:46:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:46:46] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:46:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:48:02] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:52:25] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 10.68, 5.12, 2.42 [19:54:00] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:54:18] RECOVERY - jobrunner3 MirahezeRenewSsl on jobrunner3 is OK: TCP OK - 0.001 second response time on 51.195.236.220 port 5000 [19:56:26] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.61, 2.96, 2.21 [20:01:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZy2 [20:01:32] [02miraheze/puppet] 07paladox 037f825a5 - varnish: Set header Age to 0 for /w/load.php [20:01:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZy2 [20:01:35] [02miraheze/puppet] 07paladox 037f825a5 - varnish: Set header Age to 0 for /w/load.php [20:01:49] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZyV [20:01:50] [02miraheze/ssl] 07Reception123 03cf6cd91 - try to fix pornwiki.org issue [20:02:53] Reception123 the domain you are redirecting from must have it's sslname defined rather than the one you are redirecting to [20:03:00] unless www. is in the same cert as en.? [20:03:51] hmm, it seemed that the domains weren't working but they now work [20:03:54] so I guess that change isn't needed [20:05:08] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:05:50] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.64 ms [20:07:48] 2 reports of images not loading properly [20:07:56] (and also they're not loading on Popups for me) [20:09:53] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 113.68 ms [20:12:22] Lake: link to the image/page? [20:12:53] one of the reports: https://tuscriaturas.miraheze.org/wiki/Nunsha_la_Valiente [20:12:55] [ Nunsha la Valiente - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [20:13:07] and here on my wiki (on popups specifically): https://pokemundo.miraheze.org/wiki/Predefini%C3%A7%C3%A3o:Evobox [20:13:08] [ Predefinição:Evobox - Pokémundo ] - pokemundo.miraheze.org [20:19:02] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 140.54 ms [20:21:04] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 113.76 ms [20:23:15] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[icinga2] [20:25:09] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 145.48 ms [20:31:33] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ9G [20:31:35] [02miraheze/ssl] 07paladox 03c3ad1b8 - Remove duplicate wiki.meregos.com [20:33:14] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:37:48] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 210s [20:39:48] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 41s [20:42:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqZ9N [20:42:46] [02miraheze/mw-config] 07Universal-Omega 03a00003c - Add multiple DiscordNotifications configs to ManageWikiSettings [20:42:48] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [20:43:11] [02mw-config] 07Universal-Omega opened pull request 03#3770: Add multiple DiscordNotifications configs to ManageWikiSettings (T6942) - 13https://git.io/JqZ9A [20:44:27] miraheze/mw-config - Universal-Omega the build passed. [20:46:27] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqZH3 [20:46:28] [02miraheze/mediawiki] 07paladox 0305c78ae - Revert "Bump extensions/Disambiguator from `e481d43` to `a6956cc` (#1247)" [20:47:33] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±2] 13https://git.io/JqZHC [20:47:35] [02miraheze/mediawiki] 07paladox 03c86aa74 - Downgrade Disambiguator to REL1_35 [20:47:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqZHl [20:47:44] [02miraheze/mw-config] 07Universal-Omega 0396743ad - Update LocalSettings.php [20:47:45] [02mw-config] 07Universal-Omega synchronize pull request 03#3770: Add multiple DiscordNotifications configs to ManageWikiSettings (T6942) - 13https://git.io/JqZ9A [20:48:22] RECOVERY - bacula2 Bacula Databases db12 on bacula2 is OK: OK: Full, 302318 files, 53.38GB, 2021-03-10 20:47:00 (1.4 minutes ago) [20:48:48] miraheze/mw-config - Universal-Omega the build passed. [20:50:09] RECOVERY - mon2 Check correctness of the icinga configuration on mon2 is OK: Icinga configuration is correct [20:51:22] [02mw-config] 07Universal-Omega closed pull request 03#3770: Add multiple DiscordNotifications configs to ManageWikiSettings (T6942) - 13https://git.io/JqZ9A [20:51:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqZHo [20:51:25] [02miraheze/mw-config] 07Universal-Omega 03d75b55a - Add multiple DiscordNotifications configs to ManageWikiSettings (T6942) (#3770) [20:51:27] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [20:51:28] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:51:36] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 100.64 ms [20:52:23] miraheze/mw-config - Universal-Omega the build passed. [20:59:38] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 2 failures [21:07:09] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 145.40 ms [21:17:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZ7I [21:17:03] [02miraheze/puppet] 07paladox 03c79dfcc - mediawiki: Fix thumb generation if image is 404 [21:17:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JqZ7I [21:17:06] [02miraheze/puppet] 07paladox 03c79dfcc - mediawiki: Fix thumb generation if image is 404 [21:17:08] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:17:09] [02puppet] 07paladox opened pull request 03#1701: mediawiki: Fix thumb generation if image is 404 - 13https://git.io/JqZ7L [21:17:33] [02puppet] 07paladox closed pull request 03#1701: mediawiki: Fix thumb generation if image is 404 - 13https://git.io/JqZ7L [21:17:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ7q [21:17:36] [02miraheze/puppet] 07paladox 0362e65ed - mediawiki: Fix thumb generation if image is 404 (#1701) [21:17:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ7q [21:17:39] [02miraheze/puppet] 07paladox 0362e65ed - mediawiki: Fix thumb generation if image is 404 (#1701) [21:17:41] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:17:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:17:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:19:22] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core] [21:20:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ7n [21:20:09] [02miraheze/puppet] 07paladox 03efb083c - Fix [21:20:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ7n [21:20:12] [02miraheze/puppet] 07paladox 03efb083c - Fix [21:21:45] @Lake should be fixed [21:23:41] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 114.33 ms [21:24:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ7u [21:25:00] [02miraheze/puppet] 07paladox 03931844d - Update mediawiki.conf.erb [21:25:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ7u [21:25:03] [02miraheze/puppet] 07paladox 03931844d - Update mediawiki.conf.erb [21:27:22] [02puppet] 07Universal-Omega synchronize pull request 03#1680: Varnish: Add X-Robots-Tag http header (Should fix T6680) - 13https://git.io/JqTN1 [21:28:35] [02puppet] 07Universal-Omega closed pull request 03#1680: Varnish: Add X-Robots-Tag http header (Should fix T6680) - 13https://git.io/JqTN1 [21:30:23] [02puppet] 07Universal-Omega opened pull request 03#1702: Varnish: Add X-Robots-Tag http header (Should fix T6680) - 13https://git.io/JqZ7S [21:31:13] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 56 minutes ago with 2 failures [21:33:01] [02puppet] 07Universal-Omega edited pull request 03#1702: Varnish: Add X-Robots-Tag http header (Should fix T6680) - 13https://git.io/JqZ7S [21:34:08] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.12 ms [21:39:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ5s [21:39:50] [02miraheze/puppet] 07paladox 039ae476c - mediawiki: Fix Nginx config [21:39:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ5s [21:39:53] [02miraheze/puppet] 07paladox 039ae476c - mediawiki: Fix Nginx config [21:40:21] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 107.25 ms [21:41:07] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core] [21:54:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ5A [21:54:06] [02miraheze/puppet] 07paladox 03a9e9407 - mediawiki: Fix Nginx config part 2 [21:54:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZ5A [21:54:09] [02miraheze/puppet] 07paladox 03a9e9407 - mediawiki: Fix Nginx config part 2 [22:11:08] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqZdo [22:11:09] [02miraheze/MirahezeMagic] 07Universal-Omega 0311964bd - Add miraheze-sitenotice-* messages [22:11:11] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#214: Move SiteNoticeAfter hooks from mw-config to MirahezeMagic - 13https://git.io/JtbsK [22:16:42] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 0317 commits to 03Universal-Omega-patch-1 [+0/-2/±42] 13https://git.io/JqZdb [22:16:44] [02miraheze/MirahezeMagic] 07Universal-Omega 033251f1c - Merge branch 'master' into Universal-Omega-patch-1 [22:16:45] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#214: Move SiteNoticeAfter hooks from mw-config to MirahezeMagic - 13https://git.io/JtbsK [22:17:44] miraheze/MirahezeMagic - Universal-Omega the build passed. [22:19:33] !log started replica process for c4, failed because binary logs on the master have been purged since [22:19:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:23:45] paladox: around by chance? [22:23:50] SPF|Cloud hi [22:23:51] yeh [22:23:53] hey :) [22:24:03] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 5.50, 3.22, 1.39 [22:24:39] critical load on a replica that can't replicate? [22:25:31] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=dbbackup2.miraheze.org&var-port=9100 [22:25:32] [ Grafana ] - grafana.miraheze.org [22:25:33] (and my !log was wrong, should have been c3) [22:25:33] I/o [22:25:56] anyways, my question was if you could (re)clone the two remaining clusters as part of https://phabricator.miraheze.org/T6071 [22:25:57] [ ⚓ T6071 Set up replicas for all database clusters ] - phabricator.miraheze.org [22:26:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is UNKNOWN: NRPE: Unable to read output [22:26:40] I have described the procedure (unfortunately still not that automated) at https://meta.miraheze.org/wiki/Tech:MariaDB#Master-replica_replication, you do not have to sit behind your laptop for a few hours [22:26:41] [ Tech:MariaDB - Miraheze Meta ] - meta.miraheze.org [22:27:23] just start the transfer for both clusters, you can apply the correct master settings (e.g. binlog position) tomorrow [22:27:23] sure [22:28:03] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 0.66, 2.50, 1.58 [22:28:40] SPF|Cloud so you are doing c4? [22:28:48] PROBLEM - dbbackup2 MariaDB c3 on dbbackup2 is CRITICAL: Can't connect to MySQL server on 'dbbackup2.miraheze.org' (111) [22:28:53] service seems dead [22:28:56] oh [22:29:11] !log stop mariadb c3 replica [22:29:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:19] so, is c3 for db11, c4 for db12 and c5 for db13? [22:29:28] (for c4) what I did was creating a directory called /home/dbcopy/db13 on dbbackup1, chmod 700 the directory and set dbcopy:dbcopy as owner:group [22:29:39] read the yaml file :) [22:29:51] https://github.com/miraheze/puppet/blob/master/hieradata/hosts/dbbackup1.yaml https://github.com/miraheze/puppet/blob/master/hieradata/hosts/dbbackup2.yaml [22:29:51] [ puppet/dbbackup1.yaml at master · miraheze/puppet · GitHub ] - github.com [22:29:52] [ puppet/dbbackup2.yaml at master · miraheze/puppet · GitHub ] - github.com [22:30:14] i have, i'm trying to figure out which db cluster goes with which instance [22:30:28] well, per host a list of clusters is given as you can see [22:30:39] oh [22:30:46] role::dbbackup::clusters contains the clusters per host [22:30:57] cp2 === db11, c3 === db12 and c4 === db13? [22:31:07] correct [22:31:12] thanks! [22:31:25] hosts/dbbackup1.yaml has clusters c2 and c4, hosts/dbbackup2.yaml has c3 only [22:31:55] c2 has been done and replicates fine, c4 was not done yet and c3 is broken (binary logs too old) [22:32:16] ok [22:32:18] doing c4 [22:32:29] if you rm -rf /srv/mariadb.c3 on dbbackup2 to start fresh, you can follow the on-wiki procedure [22:32:38] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4233 MB (10% inode=96%); [22:33:47] !log starting db13 backup to dbbackup1 [22:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:17] I saw you have been working on https://phabricator.miraheze.org/T6830, thanks :) [22:34:18] [ ⚓ T6830 Add icinga/prometheus monitoring for multi-instance ] - phabricator.miraheze.org [22:34:43] yup! [22:35:13] https://grafana.miraheze.org/d/MQWgroiiz/mariadb-overview?orgId=1&refresh=1m the graphs for db* are gone though :( [22:35:13] [ Grafana ] - grafana.miraheze.org [22:35:35] but I'll see if I can find the cause [22:35:45] oh [22:35:46] damn [22:36:11] I'll take care of prometheus, the backups are more important [22:36:25] you have seen what has happened in Strasbourg... [22:38:09] yeh [22:38:26] i've started the backup for db13 [22:40:25] ah, the definitions for dbX don't exist in prometheus.yml [22:40:35] found the issue [22:40:37] yeh [22:40:54] aha [22:41:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZbR [22:41:56] [02miraheze/puppet] 07paladox 03a522f4e - prometheus: Fix query for finding mariadb hosts [22:41:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZbR [22:41:59] [02miraheze/puppet] 07paladox 03a522f4e - prometheus: Fix query for finding mariadb hosts [22:42:21] SPF|Cloud ^ [22:42:24] yea, role::db has prometheus::mysqld_exporter::instance [22:42:36] nice fix :) [22:43:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZba [22:43:14] [02miraheze/puppet] 07paladox 03ae73a08 - prometheus: Remove dbbackup from mariadb query [22:43:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZba [22:43:17] [02miraheze/puppet] 07paladox 03ae73a08 - prometheus: Remove dbbackup from mariadb query [22:43:28] Hm? [22:43:43] oh, right, backup servers are defined manually [22:43:44] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 238729 files, 84.61GB, 2021-03-10 22:42:00 (1.7 minutes ago) [22:43:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZbV [22:44:00] [02miraheze/puppet] 07paladox 039424af4 - Fix [22:44:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZbV [22:44:03] [02miraheze/puppet] 07paladox 039424af4 - Fix [22:44:31] SPF|Cloud yeh, i did it manually for now. [22:44:55] seems fine for now, the monitoring is important [22:46:10] eventually you will be able to automate this, perhaps we should use a hiera variable comparable to role::dbbackup::clusters that is used for generating the exporter list [22:50:16] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 5509 files, 22.13MB, 2021-03-10 22:50:00 (15.0 seconds ago) [22:54:36] [02puppet] 07Universal-Omega edited pull request 03#1702: Varnish: Add X-Robots-Tag http header (Should fix T6680) - 13https://git.io/JqZ7S [23:03:58] PROBLEM - phab2 Current Load on phab2 is WARNING: WARNING - load average: 1.76, 1.49, 0.90 [23:06:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZNj [23:06:40] [02miraheze/mw-config] 07Universal-Omega 03e79dbe1 - Use $wmgUseTheme check on theme variables for Gamepress [23:07:40] miraheze/mw-config - Universal-Omega the build passed. [23:07:54] RECOVERY - phab2 Current Load on phab2 is OK: OK - load average: 1.64, 1.60, 1.08 [23:10:37] RECOVERY - bacula2 Bacula Phabricator Static on bacula2 is OK: OK: Full, 87218 files, 3.984GB, 2021-03-10 23:10:00 (37.0 seconds ago) [23:12:18] night [23:13:29] night, SPF|Cloud [23:13:32] Early night for you [23:13:37] It's not even midnight :P [23:13:40] it is [23:13:43] heh [23:13:45] 00:13 [23:13:48] oh [23:13:50] right [23:13:53] you're UTC +1 [23:14:00] correct [23:14:22] there is some work I need to finish 'today', but waking up early may be the better way to do it [23:15:36] but asking p.aladox to fix the database replication was a high priority task for now [23:16:39] you may have seen one of OVH's data centers was on fire today [23:25:06] yeah [23:58:58] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nowchess.org All nameservers failed to answer the query.