[00:03:43] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.20, 21.04, 16.68 [00:05:41] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.41, 17.60, 15.98 [00:08:17] RECOVERY - wiki.kali-team.cn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kali-team.cn' will expire on Sun 17 Jan 2021 22:52:04 GMT +0000. [00:24:41] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (1 critical updates). [00:35:23] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (1 critical updates). [00:35:37] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (1 critical updates). [01:15:08] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 92 packages available for upgrade (1 critical updates). [01:56:53] PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 93 packages available for upgrade (1 critical updates). [02:06:36] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [02:11:53] PROBLEM - holonet.pw - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'holonet.pw' expires in 15 day(s) (Thu 05 Nov 2020 02:05:46 GMT +0000). [02:16:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBK2 [02:16:38] [02miraheze/ssl] 07MirahezeSSLBot 03fd946c1 - Bot: Update SSL cert for holonet.pw [02:24:50] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 15 day(s) (Thu 05 Nov 2020 02:16:19 GMT +0000). [02:25:32] RECOVERY - holonet.pw - LetsEncrypt on sslhost is OK: OK - Certificate 'holonet.pw' will expire on Mon 18 Jan 2021 01:16:30 GMT +0000. [02:27:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTB6R [02:27:08] [02miraheze/ssl] 07MirahezeSSLBot 03a9b4f5e - Bot: Update SSL cert for enc.for.uz [02:33:58] PROBLEM - studentwiki.ddns.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'studentwiki.ddns.net' expires in 15 day(s) (Thu 05 Nov 2020 02:25:24 GMT +0000). [02:37:46] PROBLEM - wiki.bonessdiving.club - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bonessdiving.club' expires in 15 day(s) (Thu 05 Nov 2020 02:30:02 GMT +0000). [02:37:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBim [02:37:56] [02miraheze/ssl] 07MirahezeSSLBot 030ada076 - Bot: Update SSL cert for studentwiki.ddns.net [02:38:28] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Mon 18 Jan 2021 01:26:59 GMT +0000. [02:39:16] hi [02:41:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBiR [02:41:50] [02miraheze/ssl] 07MirahezeSSLBot 0394f2573 - Bot: Update SSL cert for wiki.bonessdiving.club [02:42:08] hello @hispano76 [02:42:27] :) [02:46:22] PROBLEM - wiki.campaign-labour.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.campaign-labour.org' expires in 15 day(s) (Thu 05 Nov 2020 02:37:41 GMT +0000). [02:47:39] RECOVERY - studentwiki.ddns.net - LetsEncrypt on sslhost is OK: OK - Certificate 'studentwiki.ddns.net' will expire on Mon 18 Jan 2021 01:37:49 GMT +0000. [02:50:14] PROBLEM - wiki.dobots.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dobots.nl' expires in 15 day(s) (Thu 05 Nov 2020 02:41:27 GMT +0000). [02:53:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBih [02:53:07] [02miraheze/ssl] 07MirahezeSSLBot 03c92cd30 - Bot: Update SSL cert for wiki.campaign-labour.org [02:53:52] PROBLEM - wiki.ngscott.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ngscott.net' expires in 15 day(s) (Thu 05 Nov 2020 02:49:59 GMT +0000). [02:54:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBPU [02:54:26] [02miraheze/ssl] 07MirahezeSSLBot 03e224ace - Bot: Update SSL cert for wiki.dobots.nl [02:58:12] RECOVERY - wiki.bonessdiving.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bonessdiving.club' will expire on Mon 18 Jan 2021 01:41:42 GMT +0000. [03:01:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBPo [03:01:42] [02miraheze/ssl] 07MirahezeSSLBot 031ca6aca - Bot: Update SSL cert for wiki.ngscott.net [03:02:36] PROBLEM - wiki.pupilliam.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pupilliam.com' expires in 15 day(s) (Thu 05 Nov 2020 02:59:24 GMT +0000). [03:03:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBP1 [03:03:34] [02miraheze/ssl] 07MirahezeSSLBot 03a4fd446 - Bot: Update SSL cert for wiki.pupilliam.com [03:03:35] RECOVERY - wiki.dobots.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dobots.nl' will expire on Mon 18 Jan 2021 01:54:18 GMT +0000. [03:04:21] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Thu 05 Nov 2020 02:55:38 GMT +0000). [03:06:25] RECOVERY - wiki.campaign-labour.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.campaign-labour.org' will expire on Mon 18 Jan 2021 01:52:58 GMT +0000. [03:09:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBXf [03:09:18] [02miraheze/ssl] 07MirahezeSSLBot 03e4273f6 - Bot: Update SSL cert for wiki.whentheycry.org [03:14:18] RECOVERY - wiki.ngscott.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ngscott.net' will expire on Mon 18 Jan 2021 02:01:34 GMT +0000. [03:14:50] PROBLEM - wiki.rmbrk.sk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rmbrk.sk' expires in 15 day(s) (Thu 05 Nov 2020 03:06:54 GMT +0000). [03:16:12] RECOVERY - wiki.pupilliam.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pupilliam.com' will expire on Mon 18 Jan 2021 02:03:26 GMT +0000. [03:16:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTBXa [03:16:57] [02miraheze/ssl] 07MirahezeSSLBot 03c70bacd - Bot: Update SSL cert for wiki.rmbrk.sk [03:23:34] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.28, 19.24, 17.29 [03:24:45] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Mon 18 Jan 2021 02:09:10 GMT +0000. [03:25:32] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.84, 17.00, 16.72 [03:28:23] RECOVERY - wiki.rmbrk.sk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rmbrk.sk' will expire on Mon 18 Jan 2021 02:16:49 GMT +0000. [04:19:11] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (1 critical updates). [04:31:09] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (1 critical updates). [05:29:04] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (1 critical updates). [06:00:29] RECOVERY - services1 APT on services1 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [06:02:35] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [06:13:05] RECOVERY - services2 APT on services2 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [06:13:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.51, 18.78, 15.96 [06:14:49] RECOVERY - mw4 APT on mw4 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [06:15:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.73, 21.55, 17.37 [06:17:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.01, 19.14, 17.01 [06:21:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.81, 20.77, 18.23 [06:23:09] RECOVERY - cp7 APT on cp7 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [06:25:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.93, 19.73, 18.48 [06:26:02] RECOVERY - cp3 APT on cp3 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [06:29:53] RECOVERY - mon1 APT on mon1 is OK: APT OK: 11 packages available for upgrade (0 critical updates). [06:31:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.13, 21.69, 19.46 [06:31:43] RECOVERY - mw7 APT on mw7 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [06:33:01] RECOVERY - cp6 APT on cp6 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [06:33:26] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.28, 20.22, 19.18 [06:42:54] RECOVERY - mw5 APT on mw5 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [06:45:05] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:53:07] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:56:55] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 92 packages available for upgrade (0 critical updates). [06:59:08] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 91 packages available for upgrade (0 critical updates). [07:43:46] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:44:15] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 5.21, 4.00, 2.01 [07:45:43] RECOVERY - ns1 APT on ns1 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [07:46:12] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 0.11, 1.85, 1.58 [07:48:09] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.00, 0.86, 1.24 [08:38:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.77.107.210/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb [08:39:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 2607:5300:205:200::2ac4/cpweb [08:40:12] PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found [08:40:13] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found [08:40:13] PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found [08:40:14] PROBLEM - mediawiki.ecfin.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mediawiki.ecfin.eu could not be found [08:40:14] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [08:40:35] PROBLEM - ndg.nenawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ndg.nenawiki.org could not be found [08:40:40] PROBLEM - wiki.erisly.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.erisly.com could not be found [08:40:45] PROBLEM - gu.gyaanipedia.co.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gu.gyaanipedia.co.in could not be found [08:40:45] PROBLEM - la.gyaanipedia.co.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for la.gyaanipedia.co.in could not be found [08:40:46] PROBLEM - wiki.casadocarvalho.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.casadocarvalho.net could not be found [08:40:47] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [08:40:47] PROBLEM - pwiki.arkcls.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.arkcls.com could not be found [08:40:47] PROBLEM - wiki.sketch.town - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sketch.town could not be found [08:40:47] PROBLEM - holonet.pw - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for holonet.pw could not be found [08:40:50] PROBLEM - www.mashedpotaytoes.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mashedpotaytoes.com could not be found [08:40:50] PROBLEM - wiki.consentcraft.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.consentcraft.uk could not be found [08:40:51] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minecraftathome.com could not be found [08:40:52] PROBLEM - wiki.xfbs.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xfbs.net could not be found [08:40:53] PROBLEM - enc.for.uz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for enc.for.uz could not be found [08:40:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:40:55] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [08:40:56] PROBLEM - tallguysfree.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tallguysfree.miraheze.org could not be found [08:40:56] PROBLEM - wiki.velaan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.velaan.org could not be found [08:40:58] PROBLEM - commonswiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commonswiki.miraheze.org could not be found [08:40:59] PROBLEM - wiki.patriam.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.patriam.cc could not be found [08:41:00] PROBLEM - rivworld.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rivworld.wiki could not be found [08:41:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:41:02] PROBLEM - dcmultiverse.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dcmultiverse.miraheze.org could not be found [08:41:05] PROBLEM - wiki.bullshit.systems - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.bullshit.systems could not be found [08:47:07] RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: rDNS OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp6.miraheze.org [08:47:09] RECOVERY - orain.org - reverse DNS on sslhost is OK: rDNS OK - orain.org reverse DNS resolves to cp6.miraheze.org [08:47:10] RECOVERY - mediawiki.ecfin.eu - reverse DNS on sslhost is OK: rDNS OK - mediawiki.ecfin.eu reverse DNS resolves to cp7.miraheze.org [08:47:11] RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: rDNS OK - n64brew.dev reverse DNS resolves to cp7.miraheze.org [08:47:12] RECOVERY - wiki.ssij.se - reverse DNS on sslhost is OK: rDNS OK - wiki.ssij.se reverse DNS resolves to cp7.miraheze.org [08:47:16] RECOVERY - ndg.nenawiki.org - reverse DNS on sslhost is OK: rDNS OK - ndg.nenawiki.org reverse DNS resolves to cp6.miraheze.org [08:47:25] RECOVERY - wiki.erisly.com - reverse DNS on sslhost is OK: rDNS OK - wiki.erisly.com reverse DNS resolves to cp7.miraheze.org [08:47:29] RECOVERY - gu.gyaanipedia.co.in - reverse DNS on sslhost is OK: rDNS OK - gu.gyaanipedia.co.in reverse DNS resolves to cp7.miraheze.org [08:47:34] RECOVERY - la.gyaanipedia.co.in - reverse DNS on sslhost is OK: rDNS OK - la.gyaanipedia.co.in reverse DNS resolves to cp6.miraheze.org [08:47:35] RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: rDNS OK - guia.esporo.net reverse DNS resolves to cp6.miraheze.org [08:47:36] RECOVERY - tallguysfree.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - tallguysfree.miraheze.org reverse DNS resolves to cp7.miraheze.org [08:47:36] RECOVERY - wiki.velaan.org - reverse DNS on sslhost is OK: rDNS OK - wiki.velaan.org reverse DNS resolves to cp6.miraheze.org [08:47:37] RECOVERY - wiki.casadocarvalho.net - reverse DNS on sslhost is OK: rDNS OK - wiki.casadocarvalho.net reverse DNS resolves to cp7.miraheze.org [08:47:38] RECOVERY - wiki.sketch.town - reverse DNS on sslhost is OK: rDNS OK - wiki.sketch.town reverse DNS resolves to cp7.miraheze.org [08:47:38] RECOVERY - holonet.pw - reverse DNS on sslhost is OK: rDNS OK - holonet.pw reverse DNS resolves to cp7.miraheze.org [08:47:38] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp7.miraheze.org [08:47:39] RECOVERY - pwiki.arkcls.com - reverse DNS on sslhost is OK: rDNS OK - pwiki.arkcls.com reverse DNS resolves to cp6.miraheze.org [08:47:39] RECOVERY - commonswiki.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - commonswiki.miraheze.org reverse DNS resolves to cp6.miraheze.org [08:47:41] RECOVERY - wiki.patriam.cc - reverse DNS on sslhost is OK: rDNS OK - wiki.patriam.cc reverse DNS resolves to cp6.miraheze.org [08:47:47] RECOVERY - rivworld.wiki - reverse DNS on sslhost is OK: rDNS OK - rivworld.wiki reverse DNS resolves to cp7.miraheze.org [08:47:47] RECOVERY - www.mashedpotaytoes.com - reverse DNS on sslhost is OK: rDNS OK - www.mashedpotaytoes.com reverse DNS resolves to cp6.miraheze.org [08:47:48] RECOVERY - wiki.consentcraft.uk - reverse DNS on sslhost is OK: rDNS OK - wiki.consentcraft.uk reverse DNS resolves to cp6.miraheze.org [08:47:48] RECOVERY - wiki.minecraftathome.com - reverse DNS on sslhost is OK: rDNS OK - wiki.minecraftathome.com reverse DNS resolves to cp6.miraheze.org [08:47:48] RECOVERY - wiki.xfbs.net - reverse DNS on sslhost is OK: rDNS OK - wiki.xfbs.net reverse DNS resolves to cp7.miraheze.org [08:47:49] RECOVERY - dcmultiverse.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - dcmultiverse.miraheze.org reverse DNS resolves to cp6.miraheze.org [08:47:53] RECOVERY - enc.for.uz - reverse DNS on sslhost is OK: rDNS OK - enc.for.uz reverse DNS resolves to cp7.miraheze.org [08:47:55] RECOVERY - wiki.bullshit.systems - reverse DNS on sslhost is OK: rDNS OK - wiki.bullshit.systems reverse DNS resolves to cp7.miraheze.org [09:12:14] PROBLEM - cp9 APT on cp9 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (1 critical updates). [09:20:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTR3X [09:20:21] [02miraheze/services] 07MirahezeSSLBot 036991002 - BOT: Updating services config for wikis [09:32:40] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.22, 20.58, 17.64 [09:34:23] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 4.69, 4.31, 2.08 [09:34:40] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.33, 18.88, 17.35 [09:36:22] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 0.10, 2.00, 1.62 [09:38:19] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.04, 0.93, 1.27 [11:43:43] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.35, 20.55, 18.11 [11:49:39] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.02, 20.40, 18.87 [12:11:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.20, 21.28, 20.06 [12:15:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.81, 18.51, 19.27 [12:43:51] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.72, 19.27, 18.56 [12:45:49] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.96, 17.35, 17.93 [12:54:40] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.79, 23.27, 20.20 [12:56:38] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.10, 22.73, 20.42 [12:58:38] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.22, 20.03, 19.71 [13:02:34] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.15, 20.53, 19.98 [13:06:34] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.86, 18.86, 19.50 [13:18:03] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 22.36, 12.97, 7.05 [13:18:32] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.70, 28.26, 22.72 [13:23:59] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.83, 7.88, 6.98 [13:25:56] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.33, 6.04, 6.41 [13:28:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.17, 21.40, 22.62 [13:32:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.26, 23.19, 22.98 [13:34:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.96, 22.75, 22.83 [13:40:12] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.43, 22.91, 22.83 [13:42:25] .op [13:42:25] IrcTest: Access Denied. If in error, please contact the channel founder. [13:42:28] .op [13:42:28] IrcTest: Access Denied. If in error, please contact the channel founder. [13:42:44] Reception123 Hello, i can use Toolforge??? [13:42:51] !log reset an account token [13:42:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:44:16] .op [13:46:06] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.86, 21.01, 22.24 [13:50:52] PROBLEM - ping6 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 463.96 ms [13:52:51] RECOVERY - ping6 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 18.49 ms [13:55:56] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.90, 18.22, 20.34 [13:59:51] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.53, 20.16, 20.66 [14:06:37] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (1 critical updates). [14:11:43] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.57, 21.96, 21.17 [14:12:13] Thanks John! [14:13:49] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:29] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 4.22, 5.47, 2.82 [14:15:49] RECOVERY - ns1 APT on ns1 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [14:18:25] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.00, 1.13, 1.70 [14:35:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.27, 22.57, 23.87 [14:49:25] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.32, 21.81, 22.44 [14:57:06] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 20.88, 14.27, 7.98 [15:01:52] PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (1 critical updates). [15:07:15] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.48, 6.01, 7.36 [15:09:15] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 18.33, 10.64, 8.85 [15:15:18] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 5.24, 7.10, 7.82 [15:19:16] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.93, 4.65, 6.63 [15:35:06] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.37, 6.39, 5.51 [15:35:20] PROBLEM - wiki.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fraterniterebellion.com could not be found [15:35:25] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [15:35:26] PROBLEM - wiki.starship.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.starship.digital could not be found [15:35:27] PROBLEM - wikibeats.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikibeats.net could not be found [15:35:30] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lefrenchmelee.fr could not be found [15:35:36] PROBLEM - intp.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for intp.miraheze.org could not be found [15:37:02] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.31, 5.90, 5.43 [15:37:55] ^ @Reception123, do the above custom domain DNS errors possibly mean the domains are no longer pointing to us? 🤔 [15:41:01] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (1 critical updates). [15:42:05] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp7.miraheze.org [15:42:10] RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: rDNS OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp7.miraheze.org [15:42:12] RECOVERY - wiki.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - wiki.fraterniterebellion.com reverse DNS resolves to cp7.miraheze.org [15:42:23] RECOVERY - intp.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - intp.miraheze.org reverse DNS resolves to cp7.miraheze.org [15:42:27] RECOVERY - wiki.starship.digital - reverse DNS on sslhost is OK: rDNS OK - wiki.starship.digital reverse DNS resolves to cp7.miraheze.org [15:42:29] RECOVERY - wikibeats.net - reverse DNS on sslhost is OK: rDNS OK - wikibeats.net reverse DNS resolves to cp7.miraheze.org [15:44:15] dmehus: sometimes it seems to say that and then goes back to ok [15:46:24] @Reception123 Ah, weird, yeah. I noticed there was a miraheze.org subdomain in there afterward, so kind of thought it would likely self-correct. [16:07:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.55, 21.09, 23.90 [16:11:17] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.44, 24.11, 24.36 [16:20:50] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (1 critical updates). [16:30:24] @Doug: rDNS could not be found is less worrying. Could just be slowness. [16:31:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.04, 20.14, 23.64 [16:33:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.29, 22.86, 24.18 [16:35:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.65, 20.77, 23.26 [16:41:12] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.72, 24.38, 23.59 [16:42:08] @RhinosF1 Thanks. 🙂 [16:42:40] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 215s [16:46:40] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [16:55:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.97, 22.03, 23.73 [17:01:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.89, 20.66, 22.22 [17:03:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.96, 20.70, 22.08 [17:13:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.28, 22.25, 21.70 [17:15:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.48, 20.45, 21.11 [17:21:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.44, 22.14, 21.45 [17:25:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.66, 19.45, 20.69 [17:27:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.61, 17.73, 19.93 [17:43:09] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (1 critical updates). [17:53:45] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.80, 19.98, 18.79 [17:55:45] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.38, 19.49, 18.78 [17:59:52] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.85, 23.32, 20.50 [18:01:50] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.00, 20.23, 19.70 [18:25:34] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.04, 20.86, 18.98 [18:29:31] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.38, 18.67, 18.60 [20:01:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.27, 21.00, 19.08 [20:03:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.58, 19.45, 18.75 [20:10:58] RECOVERY - bacula2 Bacula Static 1 on bacula2 is OK: OK: Full, 11963030 files, 597.6GB, 2020-10-20 20:09:00 (1.9 minutes ago) [20:40:05] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.79, 18.18, 16.53 [20:42:03] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.83, 17.60, 16.54 [20:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JT0Ve [20:55:14] [02miraheze/services] 07MirahezeSSLBot 0307d423b - BOT: Updating services config for wikis [21:08:46] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.29, 21.95, 17.74 [21:10:46] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.29, 19.36, 17.25 [21:21:44] is anyone else getting an exception when trying to approve a wiki [21:24:12] What is the message [21:24:21] I can check server logs [21:25:34] Naleksuh [21:25:43] [3111c578df8dadbcb3001d30] 2020-10-20 21:21:29: Fatal exception of type "MWException" [21:25:45] zpiix [21:25:46] zppix [21:25:48] Looking [21:32:35] hrm, yeah I'm curious to see what that error traces as [21:33:23] I gotta do it the old fashioned way graylog confuses me so i got to figure out what mw he hit [21:33:43] ah [21:37:19] Can you link me to request naleksuh [21:39:24] Actually found ir [21:40:07] The request cannot be approved due to technical limitations, no non alphanumeric chars can be in the db name [21:40:28] Zppix graylog is not in operation for prod yet [21:40:30] only test2 [21:40:37] Oh [21:40:42] That explains it xD [21:47:20] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 17.36, 10.34, 5.02 [21:47:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 36.65, 26.28, 18.99 [21:47:50] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:48:32] @Zppix ah that makes sense. Non-alphanumeric characters. Well at least we know it's not a bug, though it would be nice if we maybe had CreateWiki return an actual error message instead of a fatal exception. [21:48:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.222.27.129/cpweb [21:49:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb [21:49:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.36, 23.32, 18.80 [21:49:51] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15670 bytes in 1.012 second response time [21:50:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JT0K5 [21:50:19] [02miraheze/services] 07MirahezeSSLBot 0319ba701 - BOT: Updating services config for wikis [21:51:01] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.23, 20.88, 16.65 [21:51:01] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:51:18] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 5.81, 7.95, 5.30 [21:52:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:53:01] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.09, 19.43, 16.68 [21:53:15] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 7.68, 8.78, 5.96 [21:53:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 23.29, 24.60, 20.37 [21:55:15] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.92, 6.36, 5.42 [21:55:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.05, 21.47, 19.68 [21:57:14] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.80, 17.93, 18.62 [22:01:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.13, 21.19, 19.87 [22:01:15] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 6.07, 6.87, 5.83 [22:03:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.09, 19.57, 19.48 [22:03:15] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.24, 5.21, 5.35 [22:46:41] @Doug its fatalling due to the db erroring when it tries to create the db [22:46:50] Its not the ext erroring [23:06:11] > @Doug its fatalling due to the db erroring when it tries to create the db @Zppix Yeah, I get that. I was just thinking if we had a friendly error message that prohibited the non-alphanumeric wiki database name/URL before it even tried to create the database, that would solve that.