[00:03:17] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.87, 20.99, 16.36 [00:05:16] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.29, 18.62, 16.03 [00:07:22] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (15 critical updates). [00:34:28] PROBLEM - wiki.twilightsignal.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.twilightsignal.com' expires in 15 day(s) (Sat 12 Sep 2020 00:29:23 GMT +0000). [00:35:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkwI [00:35:54] [02miraheze/ssl] 07MirahezeSSLBot 032e19018 - Bot: Update SSL cert for wiki.twilightsignal.com [00:42:58] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 8.06, 5.01, 2.27 [00:43:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:22] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:46:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [00:46:19] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [00:48:15] RECOVERY - wiki.twilightsignal.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.twilightsignal.com' will expire on Tue 24 Nov 2020 23:35:45 GMT +0000. [00:50:56] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 0.44, 0.88, 1.84 [00:52:55] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.06, 0.49, 1.47 [01:01:21] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Sat 12 Sep 2020 00:58:17 GMT +0000). [01:02:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkwb [01:02:44] [02miraheze/ssl] 07MirahezeSSLBot 039a6d87f - Bot: Update SSL cert for nonbinary.wiki [01:06:47] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Sat 12 Sep 2020 00:59:50 GMT +0000). [01:07:08] PROBLEM - pl.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Sat 12 Sep 2020 00:58:17 GMT +0000). [01:10:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkrf [01:10:34] [02miraheze/ssl] 07MirahezeSSLBot 03ef75b6a - Bot: Update SSL cert for data.nonbinary.wiki [01:13:50] RECOVERY - pl.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Wed 25 Nov 2020 00:02:34 GMT +0000. [01:15:03] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Wed 25 Nov 2020 00:02:34 GMT +0000. [01:16:24] PROBLEM - www.schulwiki.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.schulwiki.de' expires in 15 day(s) (Sat 12 Sep 2020 01:12:20 GMT +0000). [01:16:28] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vedopedia.witches-empire.com' expires in 15 day(s) (Sat 12 Sep 2020 01:07:42 GMT +0000). [01:23:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkrC [01:23:23] [02miraheze/ssl] 07MirahezeSSLBot 03dbdfb37 - Bot: Update SSL cert for www.schulwiki.de [01:24:44] PROBLEM - wiki.mnzp.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mnzp.xyz' expires in 15 day(s) (Sat 12 Sep 2020 01:17:17 GMT +0000). [01:25:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkrl [01:25:56] [02miraheze/ssl] 07MirahezeSSLBot 03f6c0c76 - Bot: Update SSL cert for vedopedia.witches-empire.com [01:26:50] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Wed 25 Nov 2020 00:10:25 GMT +0000. [01:28:33] PROBLEM - www.bestpractice.technology - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bestpractice.technology' expires in 15 day(s) (Sat 12 Sep 2020 01:24:21 GMT +0000). [01:29:45] PROBLEM - www.openonderwijs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.openonderwijs.org' expires in 15 day(s) (Sat 12 Sep 2020 01:27:05 GMT +0000). [01:31:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkrV [01:31:51] [02miraheze/ssl] 07MirahezeSSLBot 0355d5926 - Bot: Update SSL cert for wiki.mnzp.xyz [01:36:35] PROBLEM - wiki.candela.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.candela.digital' expires in 15 day(s) (Sat 12 Sep 2020 01:31:59 GMT +0000). [01:36:57] RECOVERY - www.schulwiki.de - LetsEncrypt on sslhost is OK: OK - Certificate 'www.schulwiki.de' will expire on Wed 25 Nov 2020 00:23:13 GMT +0000. [01:37:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkrP [01:37:06] [02miraheze/ssl] 07MirahezeSSLBot 038bc718c - Bot: Update SSL cert for www.openonderwijs.org [01:37:09] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Wed 25 Nov 2020 00:25:48 GMT +0000. [01:37:36] PROBLEM - wiki.itsuali.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.itsuali.tk' expires in 15 day(s) (Sat 12 Sep 2020 01:31:10 GMT +0000). [01:40:09] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Sat 12 Sep 2020 01:36:24 GMT +0000). [01:40:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkrF [01:40:37] [02miraheze/ssl] 07MirahezeSSLBot 038d45641 - Bot: Update SSL cert for wiki.candela.digital [01:43:34] RECOVERY - www.openonderwijs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.openonderwijs.org' will expire on Wed 25 Nov 2020 00:36:57 GMT +0000. [01:45:39] RECOVERY - wiki.mnzp.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mnzp.xyz' will expire on Wed 25 Nov 2020 00:31:42 GMT +0000. [01:46:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8d456414ec2b...60fd3f62b8bc [01:46:11] [ Comparing 8d456414ec2b...60fd3f62b8bc · miraheze/ssl · GitHub ] - github.com [01:46:12] [02miraheze/ssl] 07MirahezeSSLBot 0360fd3f6 - Bot: Update SSL cert for vmcodex.net [01:47:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkov [01:47:05] [02miraheze/ssl] 07MirahezeSSLBot 03203c1f0 - Bot: Update SSL cert for www.bestpractice.technology [01:48:12] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Sat 12 Sep 2020 01:39:30 GMT +0000). [01:50:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoI [01:50:59] [02miraheze/ssl] 07MirahezeSSLBot 03559eb29 - Bot: Update SSL cert for wiki.itsuali.tk [01:52:01] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 15 day(s) (Sat 12 Sep 2020 01:47:52 GMT +0000). [01:52:29] PROBLEM - www.evanswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.evanswiki.org' expires in 15 day(s) (Sat 12 Sep 2020 01:46:07 GMT +0000). [01:52:33] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tensegritywiki.com' expires in 15 day(s) (Sat 12 Sep 2020 01:47:17 GMT +0000). [01:53:42] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Wed 25 Nov 2020 00:45:58 GMT +0000. [01:54:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUko3 [01:54:46] [02miraheze/ssl] 07MirahezeSSLBot 03ecda15d - Bot: Update SSL cert for vitriol.spoon.army [01:55:29] RECOVERY - www.bestpractice.technology - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bestpractice.technology' will expire on Wed 25 Nov 2020 00:46:56 GMT +0000. [01:56:50] RECOVERY - wiki.candela.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.candela.digital' will expire on Wed 25 Nov 2020 00:40:24 GMT +0000. [01:58:38] PROBLEM - sahitya.shaunak.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sahitya.shaunak.in' expires in 15 day(s) (Sat 12 Sep 2020 01:49:57 GMT +0000). [02:00:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoC [02:00:38] [02miraheze/ssl] 07MirahezeSSLBot 0374035c5 - Bot: Update SSL cert for www.evanswiki.org [02:01:47] RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Wed 25 Nov 2020 00:54:36 GMT +0000. [02:04:22] RECOVERY - wiki.itsuali.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.itsuali.tk' will expire on Wed 25 Nov 2020 00:50:51 GMT +0000. [02:05:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUko0 [02:05:36] [02miraheze/ssl] 07MirahezeSSLBot 03f1c7031 - Bot: Update SSL cert for sahitya.shaunak.in [02:09:28] PROBLEM - www.modesofdiscourse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.modesofdiscourse.com' expires in 15 day(s) (Sat 12 Sep 2020 02:02:52 GMT +0000). [02:12:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoa [02:12:27] [02miraheze/ssl] 07MirahezeSSLBot 03e7af10b - Bot: Update SSL cert for tensegritywiki.com [02:12:34] RECOVERY - sahitya.shaunak.in - LetsEncrypt on sslhost is OK: OK - Certificate 'sahitya.shaunak.in' will expire on Wed 25 Nov 2020 01:05:27 GMT +0000. [02:13:06] RECOVERY - www.evanswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.evanswiki.org' will expire on Wed 25 Nov 2020 01:00:29 GMT +0000. [02:14:18] PROBLEM - wiki.casual4casuals.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.casual4casuals.com' expires in 15 day(s) (Sat 12 Sep 2020 02:09:04 GMT +0000). [02:14:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoo [02:14:53] [02miraheze/ssl] 07MirahezeSSLBot 0313b4e1c - Bot: Update SSL cert for www.modesofdiscourse.com [02:17:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoX [02:17:24] [02miraheze/ssl] 07MirahezeSSLBot 030bb7951 - Bot: Update SSL cert for nonciclopedia.org [02:20:33] PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mikrodev.com' expires in 15 day(s) (Sat 12 Sep 2020 02:16:12 GMT +0000). [02:20:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoH [02:20:38] [02miraheze/ssl] 07MirahezeSSLBot 03f5980dd - Bot: Update SSL cert for wiki.casual4casuals.com [02:22:52] RECOVERY - www.modesofdiscourse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.modesofdiscourse.com' will expire on Wed 25 Nov 2020 01:14:44 GMT +0000. [02:23:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoF [02:23:43] [02miraheze/ssl] 07MirahezeSSLBot 033cf71f9 - Bot: Update SSL cert for wiki.mikrodev.com [02:24:46] PROBLEM - wiki.ripto.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ripto.gq' expires in 15 day(s) (Sat 12 Sep 2020 02:20:19 GMT +0000). [02:25:41] PROBLEM - wiki.newfil.com.au - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.newfil.com.au' expires in 15 day(s) (Sat 12 Sep 2020 02:20:51 GMT +0000). [02:26:48] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 25 Nov 2020 01:17:16 GMT +0000. [02:27:17] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Wed 25 Nov 2020 01:12:18 GMT +0000. [02:28:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkoh [02:28:16] [02miraheze/ssl] 07MirahezeSSLBot 0331459c5 - Bot: Update SSL cert for wiki.ripto.gq [02:28:27] Alert to Miraheze Staff: It looks like the MirahezeRC bot has stopped! Recent Changes are no longer available from IRC. [02:32:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkKt [02:32:35] [02miraheze/ssl] 07MirahezeSSLBot 0345fb9b8 - Bot: Update SSL cert for wiki.newfil.com.au [02:34:18] RECOVERY - wiki.casual4casuals.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.casual4casuals.com' will expire on Wed 25 Nov 2020 01:20:30 GMT +0000. [02:34:24] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Wed 25 Nov 2020 01:23:34 GMT +0000. [02:38:44] RECOVERY - wiki.ripto.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ripto.gq' will expire on Wed 25 Nov 2020 01:28:08 GMT +0000. [02:42:43] PROBLEM - crimesciencewiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crimesciencewiki.org' expires in 15 day(s) (Sat 12 Sep 2020 02:34:53 GMT +0000). [02:43:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkK8 [02:43:59] [02miraheze/ssl] 07MirahezeSSLBot 038bf9b69 - Bot: Update SSL cert for crimesciencewiki.org [02:45:46] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (15 critical updates). [02:46:12] RECOVERY - wiki.newfil.com.au - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newfil.com.au' will expire on Wed 25 Nov 2020 01:32:27 GMT +0000. [02:48:20] PROBLEM - vsfan.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vsfan.net' expires in 15 day(s) (Sat 12 Sep 2020 02:42:05 GMT +0000). [02:49:20] PROBLEM - wiki.opendominion.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.opendominion.net' expires in 15 day(s) (Sat 12 Sep 2020 02:41:53 GMT +0000). [02:49:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkKV [02:49:33] [02miraheze/ssl] 07MirahezeSSLBot 030ac77fa - Bot: Update SSL cert for vsfan.net [02:53:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkKX [02:53:16] [02miraheze/ssl] 07MirahezeSSLBot 03c5f3f87 - Bot: Update SSL cert for wiki.opendominion.net [02:56:40] RECOVERY - crimesciencewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crimesciencewiki.org' will expire on Wed 25 Nov 2020 01:43:51 GMT +0000. [02:59:22] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [02:59:27] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (15 critical updates). [03:01:50] RECOVERY - vsfan.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vsfan.net' will expire on Wed 25 Nov 2020 01:49:24 GMT +0000. [03:02:44] RECOVERY - wiki.opendominion.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.opendominion.net' will expire on Wed 25 Nov 2020 01:53:06 GMT +0000. [04:04:47] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 0.24, 1.81, 1.25 [04:06:47] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.01, 0.82, 0.97 [04:40:22] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [04:50:03] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [05:23:10] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/JUkMt [05:23:12] [02miraheze/mediawiki] 07Reception123 03df6af49 - Install JsCalendar ext. [05:27:39] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-4 [+0/-0/±4] 13https://git.io/JUkMO [05:27:41] [02miraheze/mw-config] 07Reception123 036e74498 - add JsCalendar ext. T5815 [05:27:42] [02mw-config] 07Reception123 created branch 03Reception123-patch-4 - 13https://git.io/vbvb3 [05:28:11] [02mw-config] 07Reception123 opened pull request 03#3212: add JsCalendar ext. T5815 - 13https://git.io/JUkMs [05:36:33] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.074 second response time [05:38:33] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 17197 bytes in 0.564 second response time [05:42:02] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [05:47:04] [02mw-config] 07Reception123 closed pull request 03#3212: add JsCalendar ext. T5815 - 13https://git.io/JUkMs [05:47:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JUkMP [05:47:07] [02miraheze/mw-config] 07Reception123 0362f0a0d - add JsCalendar ext. T5815 (#3212) [05:47:08] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-4 - 13https://git.io/vbvb3 [05:47:10] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-4 [05:50:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkMS [05:50:20] [02miraheze/services] 07MirahezeSSLBot 03904bdc5 - BOT: Updating services config for wikis [05:54:51] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr* [05:54:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:10] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [05:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:08] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.81, 24.06, 18.74 [05:58:30] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (15 critical updates). [05:59:56] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:00:06] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.11, 19.70, 17.76 [06:03:27] RECOVERY - mw6 APT on mw6 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:04:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.82, 21.86, 19.23 [06:05:21] RECOVERY - services2 APT on services2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:06:00] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.13, 19.75, 18.83 [06:09:08] RECOVERY - cp9 APT on cp9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:09:49] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Exec[git_pull_MediaWiki core] [06:10:29] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:11:14] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:13:46] RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:15:13] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:15:48] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:16:29] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:25:24] RECOVERY - mon1 APT on mon1 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [06:29:20] RECOVERY - mw7 APT on mw7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:34:01] RECOVERY - cp7 APT on cp7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:34:30] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:44:15] RECOVERY - mail1 APT on mail1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:50:19] RECOVERY - test2 APT on test2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:52:22] RECOVERY - services1 APT on services1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:53:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.86, 20.93, 19.08 [06:55:11] RECOVERY - mw4 APT on mw4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:55:22] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.78, 19.38, 18.74 [07:33:54] PROBLEM - wiki.senety.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.senety.com' expires in 15 day(s) (Sat 12 Sep 2020 07:27:11 GMT +0000). [07:39:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUk9b [07:39:09] [02miraheze/ssl] 07MirahezeSSLBot 03ea25270 - Bot: Update SSL cert for wiki.senety.com [07:53:59] RECOVERY - wiki.senety.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.senety.com' will expire on Wed 25 Nov 2020 06:39:00 GMT +0000. [08:05:16] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.26, 21.53, 17.82 [08:09:14] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:09:17] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.31, 21.45, 18.79 [08:15:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.53, 18.72, 18.62 [08:17:14] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:29:11] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [10:19:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUkAV [10:19:51] [02miraheze/mw-config] 07Reception123 0327adac1 - disable discord new user notifications for jadtechwiki per discord request [10:34:22] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (15 critical updates). [10:44:29] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JUkxc [10:44:30] [02miraheze/dns] 07Reception123 03ef7abb1 - add battlecats.ml zone [10:57:07] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.62, 23.77, 20.27 [11:01:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.82, 20.17, 19.48 [11:14:14] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [11:26:50] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.58, 21.66, 19.86 [11:32:46] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.97, 18.74, 19.43 [11:50:59] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JUkjH [11:51:01] [02miraheze/ssl] 07Reception123 03d75f263 - add www.battlecats.ml cert [12:02:34] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:02:50] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:02:53] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:03:06] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:03:09] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:03:42] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:03:58] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:04:18] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:04:24] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:04:30] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:05:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/services/compare/904bdc5d7361...07062ed445a4 [12:05:17] [ Comparing 904bdc5d7361...07062ed445a4 · miraheze/services · GitHub ] - github.com [12:05:17] [02miraheze/services] 07MirahezeSSLBot 0307062ed - BOT: Updating services config for wikis [13:18:03] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [13:53:29] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JUIIU [13:53:31] [02miraheze/MirahezeMagic] 07translatewiki 03e34c284 - Localisation updates from https://translatewiki.net. [13:53:32] [ Main page - translatewiki.net ] - translatewiki.net [14:07:56] PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [14:28:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.11, 19.96, 17.14 [14:30:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.61, 20.61, 17.70 [14:30:31] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUILr [14:30:33] [02miraheze/ssl] 07Reception123 03ce6e6d7 - attempt fix [14:32:19] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.77, 19.34, 17.61 [15:09:03] [02mw-config] 07Universal-Omega opened pull request 03#3213: Add Moderation namespace configs per T6055 - 13https://git.io/JUIqo [15:11:16] [02mw-config] 07Reception123 reviewed pull request 03#3213 commit - 13https://git.io/JUIq9 [15:14:54] PROBLEM - wiki.besuccess.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.besuccess.com' expires in 15 day(s) (Sat 12 Sep 2020 15:09:21 GMT +0000). [15:16:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUImv [15:16:54] [02miraheze/ssl] 07MirahezeSSLBot 03103947e - Bot: Update SSL cert for wiki.besuccess.com [15:17:41] [02mw-config] 07Universal-Omega synchronize pull request 03#3213: Add Moderation namespace configs per T6055 - 13https://git.io/JUIqo [15:20:21] [02mw-config] 07Reception123 reviewed pull request 03#3213 commit - 13https://git.io/JUImJ [15:20:23] PROBLEM - bconnected.aidanmarkham.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bconnected.aidanmarkham.com' expires in 15 day(s) (Sat 12 Sep 2020 15:15:31 GMT +0000). [15:22:43] [02mw-config] 07Universal-Omega synchronize pull request 03#3213: Add Moderation namespace configs per T6055 - 13https://git.io/JUIqo [15:25:03] [02mw-config] 07Reception123 closed pull request 03#3213: Add Moderation namespace configs per T6055 - 13https://github.com/miraheze/mw-config/pull/3213 [15:25:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIml [15:25:06] [02miraheze/mw-config] 07Universal-Omega 03e873fa6 - Add Moderation namespace configs per T6055 (#3213) * Add Moderation namespace configs per T6055 * fix comma for configs on T6055 * Fix typo for configs in T6055 [15:25:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIm8 [15:25:15] [02miraheze/ssl] 07MirahezeSSLBot 03db0ce9c - Bot: Update SSL cert for bconnected.aidanmarkham.com [15:30:13] PROBLEM - wiki.consentcraft.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.consentcraft.uk' expires in 15 day(s) (Sat 12 Sep 2020 15:22:35 GMT +0000). [15:30:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.26, 20.37, 18.10 [15:31:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIm6 [15:31:54] [02miraheze/ssl] 07MirahezeSSLBot 03912bfd6 - Bot: Update SSL cert for wiki.consentcraft.uk [15:33:48] PROBLEM - wiki.hanaoto.me - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.hanaoto.me' expires in 15 day(s) (Sat 12 Sep 2020 15:25:20 GMT +0000). [15:34:05] !log deleted and dropped doctorwhofanfilmsdatabasewiki (recreation) [15:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUImP [15:34:35] [02miraheze/ssl] 07MirahezeSSLBot 03bf4ce42 - Bot: Update SSL cert for wiki.hanaoto.me [15:36:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.41, 21.77, 19.87 [15:37:20] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Sat 12 Sep 2020 15:34:35 GMT +0000). [15:37:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bf4ce42ca038...93a4ddb4ab01 [15:37:43] [ Comparing bf4ce42ca038...93a4ddb4ab01 · miraheze/ssl · GitHub ] - github.com [15:37:44] [02miraheze/ssl] 07MirahezeSSLBot 0393a4ddb - Bot: Update SSL cert for kunwok.org [15:38:19] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.77, 19.50, 19.23 [15:43:42] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Sat 12 Sep 2020 15:36:15 GMT +0000). [15:44:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIYL [15:44:04] [02miraheze/ssl] 07MirahezeSSLBot 03d450969 - Bot: Update SSL cert for podpedia.org [15:44:44] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 15 day(s) (Sat 12 Sep 2020 15:38:02 GMT +0000). [15:45:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIYs [15:45:39] [02miraheze/ssl] 07MirahezeSSLBot 037611320 - Bot: Update SSL cert for speleo.wiki [15:49:36] PROBLEM - secularknowledge.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'secularknowledge.com' expires in 15 day(s) (Sat 12 Sep 2020 15:41:42 GMT +0000). [15:50:09] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.graalmilitary.com' expires in 15 day(s) (Sat 12 Sep 2020 15:46:52 GMT +0000). [15:54:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIYi [15:54:42] [02miraheze/ssl] 07MirahezeSSLBot 031be5953 - Bot: Update SSL cert for secularknowledge.com [16:00:01] PROBLEM - wikipuk.cl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipuk.cl' expires in 15 day(s) (Sat 12 Sep 2020 15:52:22 GMT +0000). [16:00:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIYd [16:00:38] [02miraheze/ssl] 07MirahezeSSLBot 032f3c1f1 - Bot: Update SSL cert for wiki.graalmilitary.com [16:04:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIOf [16:04:22] [02miraheze/ssl] 07MirahezeSSLBot 03a592e97 - Bot: Update SSL cert for wikipuk.cl [16:06:58] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.teessidehackspace.org.uk' expires in 15 day(s) (Sat 12 Sep 2020 15:59:44 GMT +0000). [16:09:55] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vise.dayid.org' expires in 15 day(s) (Sat 12 Sep 2020 16:03:34 GMT +0000). [16:10:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIOY [16:10:19] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kourouklides.com' expires in 15 day(s) (Sat 12 Sep 2020 16:05:12 GMT +0000). [16:10:19] [02miraheze/services] 07MirahezeSSLBot 0334f5dfa - BOT: Updating services config for wikis [16:14:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIOc [16:14:42] [02miraheze/ssl] 07MirahezeSSLBot 0312f651d - Bot: Update SSL cert for wiki.teessidehackspace.org.uk [16:15:26] PROBLEM - wiki.staraves-no.cz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.staraves-no.cz' expires in 15 day(s) (Sat 12 Sep 2020 16:12:40 GMT +0000). [16:19:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIO0 [16:19:46] [02miraheze/ssl] 07MirahezeSSLBot 03932df54 - Bot: Update SSL cert for vise.dayid.org [16:22:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JUIOK [16:22:21] [02miraheze/puppet] 07paladox 03250cf75 - fix reloading nginx after certificate update [16:22:23] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:22:25] [02puppet] 07paladox opened pull request 03#1485: fix reloading nginx after certificate update - 13https://git.io/JUIO6 [16:23:06] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 50% [16:23:07] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 59% [16:23:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIOP [16:23:30] [02miraheze/ssl] 07MirahezeSSLBot 03e405a5e - Bot: Update SSL cert for wiki.staraves-no.cz [16:23:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [16:23:35] PROBLEM - cp9 Current Load on cp9 is CRITICAL: CRITICAL - load average: 3.44, 1.86, 0.93 [16:23:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [16:25:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JUIO1 [16:25:24] [02miraheze/puppet] 07paladox 03437e5cd - Update init.pp [16:25:25] [02puppet] 07paladox synchronize pull request 03#1485: fix reloading nginx after certificate update - 13https://git.io/JUIO6 [16:25:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIOM [16:25:28] [02miraheze/ssl] 07MirahezeSSLBot 034cfee27 - Bot: Update SSL cert for wiki.kourouklides.com [16:26:33] RECOVERY - cp9 Current Load on cp9 is OK: OK - load average: 0.24, 1.10, 0.80 [16:28:59] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 29% [16:29:00] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [16:29:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:29:43] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw5 [16:29:51] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw5 [16:30:06] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [16:30:55] PROBLEM - garrettcountyguide.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'garrettcountyguide.com' expires in 15 day(s) (Sat 12 Sep 2020 16:20:47 GMT +0000). [16:31:00] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 6% [16:31:16] PROBLEM - madgenderscience.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'madgenderscience.wiki' expires in 15 day(s) (Sat 12 Sep 2020 16:23:11 GMT +0000). [16:31:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:31:39] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [16:31:48] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [16:32:02] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [16:32:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIOF [16:32:12] [02miraheze/ssl] 07MirahezeSSLBot 03653e287 - Bot: Update SSL cert for garrettcountyguide.com [16:33:31] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.27, 20.24, 17.65 [16:35:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIOp [16:35:20] [02miraheze/ssl] 07MirahezeSSLBot 032eac6e0 - Bot: Update SSL cert for madgenderscience.wiki [16:35:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.73, 22.06, 18.62 [16:36:15] PROBLEM - wiki.ameciclo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ameciclo.org' expires in 15 day(s) (Sat 12 Sep 2020 16:27:37 GMT +0000). [16:37:33] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.57, 21.91, 19.03 [16:40:06] PROBLEM - disabled.life - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'disabled.life' expires in 15 day(s) (Sat 12 Sep 2020 16:32:04 GMT +0000). [16:40:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI3J [16:40:54] [02miraheze/ssl] 07MirahezeSSLBot 038b3c96f - Bot: Update SSL cert for wiki.ameciclo.org [16:41:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.82, 22.72, 19.83 [16:45:05] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.arkcls.com' expires in 15 day(s) (Sat 12 Sep 2020 16:36:23 GMT +0000). [16:45:14] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 15 day(s) (Sat 12 Sep 2020 16:42:46 GMT +0000). [16:47:35] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.78, 22.09, 20.78 [16:48:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI3n [16:48:48] [02miraheze/ssl] 07MirahezeSSLBot 030e0df81 - Bot: Update SSL cert for disabled.life [16:51:35] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.46, 22.86, 21.29 [16:55:34] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.21, 21.64, 21.16 [16:55:46] PROBLEM - wiki.patriam.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.patriam.cc' expires in 15 day(s) (Sat 12 Sep 2020 16:51:12 GMT +0000). [16:56:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI3u [16:56:57] [02miraheze/ssl] 07MirahezeSSLBot 033bc5989 - Bot: Update SSL cert for wiki.thunis.eu [16:59:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI32 [16:59:33] [02miraheze/ssl] 07MirahezeSSLBot 03a203116 - Bot: Update SSL cert for wiki.patriam.cc [17:00:18] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Sat 12 Sep 2020 16:51:32 GMT +0000). [17:02:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.69, 17.34, 19.43 [17:03:24] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [17:03:30] PROBLEM - wiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.miraheze.org could not be found [17:04:36] PROBLEM - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pablojprieto.com.es' expires in 15 day(s) (Sat 12 Sep 2020 16:57:08 GMT +0000). [17:04:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI3K [17:04:46] [02miraheze/ssl] 07MirahezeSSLBot 0378e9bc5 - Bot: Update SSL cert for pwiki.arkcls.com [17:06:10] PROBLEM - wiki.inebriation-confederation.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.inebriation-confederation.com' expires in 15 day(s) (Sat 12 Sep 2020 17:00:27 GMT +0000). [17:07:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI3X [17:07:48] [02miraheze/ssl] 07MirahezeSSLBot 032cc2f6c - Bot: Update SSL cert for wiki.pablojprieto.com.es [17:10:05] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: rDNS OK - podpedia.org reverse DNS resolves to cp6.miraheze.org [17:10:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI3d [17:10:22] [02miraheze/services] 07MirahezeSSLBot 03ce05eae - BOT: Updating services config for wikis [17:10:28] RECOVERY - wiki.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - wiki.miraheze.org reverse DNS resolves to cp7.miraheze.org [17:12:29] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Sat 12 Sep 2020 17:07:42 GMT +0000). [17:14:12] PROBLEM - tep.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tep.wiki' expires in 15 day(s) (Sat 12 Sep 2020 17:06:06 GMT +0000). [17:14:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI3x [17:14:47] [02miraheze/ssl] 07MirahezeSSLBot 0361be0d8 - Bot: Update SSL cert for wiki.tulpa.info [17:15:29] PROBLEM - wiki.drones4nature.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.drones4nature.info' expires in 15 day(s) (Sat 12 Sep 2020 17:12:46 GMT +0000). [17:21:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIsZ [17:21:13] [02miraheze/ssl] 07MirahezeSSLBot 0334b6016 - Bot: Update SSL cert for tep.wiki [17:22:43] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 20 packages available for upgrade (13 critical updates). [17:23:02] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worlduniversityandschool.org' expires in 15 day(s) (Sat 12 Sep 2020 17:18:22 GMT +0000). [17:26:15] PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meeusen.net' expires in 15 day(s) (Sat 12 Sep 2020 17:23:06 GMT +0000). [17:29:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIsC [17:29:16] [02miraheze/ssl] 07MirahezeSSLBot 039a8e95f - Bot: Update SSL cert for wiki.drones4nature.info [17:29:43] PROBLEM - froggy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'froggy.info' expires in 15 day(s) (Sat 12 Sep 2020 17:25:52 GMT +0000). [17:33:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIs0 [17:33:15] [02miraheze/ssl] 07MirahezeSSLBot 0336b4e49 - Bot: Update SSL cert for wiki.meeusen.net [17:33:22] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nowchess.org' expires in 15 day(s) (Sat 12 Sep 2020 17:28:23 GMT +0000). [17:35:22] PROBLEM - lodge.jsnydr.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lodge.jsnydr.com' expires in 15 day(s) (Sat 12 Sep 2020 17:32:08 GMT +0000). [17:37:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIsw [17:37:53] [02miraheze/ssl] 07MirahezeSSLBot 03e23aad1 - Bot: Update SSL cert for wiki.worlduniversityandschool.org [17:38:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIso [17:38:36] [02miraheze/ssl] 07MirahezeSSLBot 0373ecc70 - Bot: Update SSL cert for wiki.inebriation-confederation.com [17:38:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIs6 [17:38:50] [02miraheze/ssl] 07MirahezeSSLBot 032f28cab - Bot: Update SSL cert for froggy.info [17:39:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIsP [17:39:37] [02miraheze/ssl] 07MirahezeSSLBot 038d756ce - Bot: Update SSL cert for wiki.autocountsoft.com [17:40:25] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 126s [17:40:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIs1 [17:40:34] [02miraheze/ssl] 07MirahezeSSLBot 03a0cd7d8 - Bot: Update SSL cert for lodge.jsnydr.com [17:40:49] PROBLEM - wiki.bullshit.systems - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bullshit.systems' expires in 15 day(s) (Sat 12 Sep 2020 17:36:07 GMT +0000). [17:42:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.37, 21.03, 19.63 [17:42:25] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [17:43:54] PROBLEM - meregos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meregos.com' expires in 15 day(s) (Sat 12 Sep 2020 17:41:45 GMT +0000). [17:47:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIsx [17:47:41] [02miraheze/ssl] 07MirahezeSSLBot 0316f1837 - Bot: Update SSL cert for wiki.nowchess.org [17:47:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIGJ [17:47:59] [02miraheze/ssl] 07MirahezeSSLBot 03e686211 - Bot: Update SSL cert for meregos.com [17:48:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.58, 20.03, 19.65 [17:50:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIGq [17:50:34] [02miraheze/ssl] 07MirahezeSSLBot 03218df7f - Bot: Update SSL cert for wiki.bullshit.systems [17:52:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.89, 22.08, 20.57 [17:54:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.56, 19.74, 19.87 [18:00:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.87, 21.17, 19.87 [18:02:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.14, 18.24, 18.98 [18:26:35] [02mw-config] 07Universal-Omega opened pull request 03#3214: Add VE useroptions for yablestudiowiki (T6107) - 13https://git.io/JUIZx [19:06:44] [02mw-config] 07Reception123 closed pull request 03#3214: Add VE useroptions for yablestudiowiki (T6107) - 13https://git.io/JUIZx [19:06:45] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIcO [19:06:47] [02miraheze/mw-config] 07Universal-Omega 031e327aa - Add VE useroptions for yablestudiowiki (T6107) (#3214) [19:14:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.76, 19.97, 17.83 [19:16:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.01, 18.03, 17.39 [19:40:36] PROBLEM - wiki.opendominion.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.opendominion.net could not be found [19:54:34] [02mw-config] 07Universal-Omega opened pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUICd [20:11:19] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.16, 21.29, 18.61 [20:13:19] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.38, 19.05, 18.11 [20:25:46] [02mw-config] 07Universal-Omega synchronize pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUICd [20:26:59] [02mw-config] 07Universal-Omega synchronize pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUICd [21:11:53] [02mw-config] 07Universal-Omega synchronize pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUICd [21:20:02] [02mw-config] 07Universal-Omega commented on pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUI8s [23:41:04] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [23:43:03] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 27%