[00:03:37] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:06:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [00:15:36] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:29:15] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [00:31:36] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:51:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [01:27:48] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:41:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [02:03:43] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:17:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [02:53:16] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:06:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [03:29:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:42:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [04:18:46] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [04:32:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [04:54:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:00:20] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.21, 1.97, 1.36 [05:02:18] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.00, 1.56, 1.28 [05:08:17] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [05:16:55] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.80, 3.50, 2.07 [05:20:46] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.60, 3.01, 2.23 [05:34:21] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 8.56, 7.35, 4.32 [05:38:18] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.55, 3.59, 3.45 [05:40:18] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.59, 2.60, 3.10 [05:57:48] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:11:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [06:16:18] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.61, 3.61, 2.11 [06:18:18] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.53, 2.76, 1.98 [06:24:26] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.23, 4.90, 2.88 [06:26:27] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.77, 3.39, 2.56 [07:48:34] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:02:10] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [08:31:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:44:51] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [09:07:08] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:20:46] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [09:36:19] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:49:56] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [10:36:35] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.95, 4.30, 2.60 [10:37:32] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mabi.myomi.net could not be found [10:37:35] PROBLEM - petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for petrawiki.org could not be found [10:37:36] PROBLEM - froggy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for froggy.info could not be found [10:37:41] PROBLEM - wiki.ciptamedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ciptamedia.org could not be found [10:37:41] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.astralprojections.org could not be found [10:37:41] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [10:38:32] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.85, 3.03, 2.33 [10:44:21] RECOVERY - wiki.ciptamedia.org - reverse DNS on sslhost is OK: rDNS OK - wiki.ciptamedia.org reverse DNS resolves to cp11.miraheze.org [10:44:21] RECOVERY - wiki.astralprojections.org - reverse DNS on sslhost is OK: rDNS OK - wiki.astralprojections.org reverse DNS resolves to cp11.miraheze.org [10:44:23] RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: rDNS OK - guia.esporo.net reverse DNS resolves to cp10.miraheze.org [10:44:25] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: rDNS OK - mabi.myomi.net reverse DNS resolves to cp11.miraheze.org [10:44:34] RECOVERY - petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - petrawiki.org reverse DNS resolves to cp10.miraheze.org [10:44:35] RECOVERY - froggy.info - reverse DNS on sslhost is OK: rDNS OK - froggy.info reverse DNS resolves to cp11.miraheze.org [10:56:34] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 2.44, 3.68, 2.44 [10:58:28] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.07, 2.85, 2.28 [11:15:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [11:28:51] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [11:37:33] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [11:45:33] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.12, 3.98, 2.34 [11:48:32] PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.gq could not be found [11:48:35] PROBLEM - www.johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.johanloopmans.nl could not be found [11:51:11] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [11:51:22] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.75, 3.71, 3.01 [11:53:18] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.38, 3.12, 2.89 [11:55:19] RECOVERY - miraheze.gq - reverse DNS on sslhost is OK: rDNS OK - miraheze.gq reverse DNS resolves to cp11.miraheze.org [11:55:22] RECOVERY - www.johanloopmans.nl - reverse DNS on sslhost is OK: rDNS OK - www.johanloopmans.nl reverse DNS resolves to cp10.miraheze.org [12:02:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.42, 21.90, 18.15 [12:03:08] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.16, 3.88, 3.13 [12:04:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 35.08, 25.75, 19.92 [12:08:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.42, 23.46, 20.36 [12:11:00] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.24, 3.38, 3.65 [12:12:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.07, 19.00, 19.37 [12:14:53] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.66, 2.08, 3.08 [12:28:23] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jsl2V [12:28:25] [02miraheze/ssl] 07Reception123 0377774db - add inourownwords.online cert [12:30:40] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 5.14, 4.30, 2.43 [12:32:34] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.37, 3.19, 2.23 [12:35:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.08, 19.20, 16.03 [12:36:27] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.41, 2.58, 2.09 [12:37:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.75, 18.28, 16.12 [12:38:27] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.96, 1.97, 1.93 [12:55:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jslrw [12:55:12] [02miraheze/services] 07MirahezeSSLBot 0391881ec - BOT: Updating services config for wikis [13:08:16] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.73, 4.17, 2.68 [13:10:15] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.33, 3.10, 2.47 [13:21:23] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:22:06] PROBLEM - www.rangpurpedia.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.rangpurpedia.xyz could not be found [13:22:10] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [13:28:51] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp11.miraheze.org [13:28:51] RECOVERY - www.rangpurpedia.xyz - reverse DNS on sslhost is OK: rDNS OK - www.rangpurpedia.xyz reverse DNS resolves to cp11.miraheze.org [13:29:45] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.46, 6.85, 4.45 [13:29:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.63, 22.52, 18.85 [13:33:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.28, 20.12, 18.73 [13:34:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [13:35:32] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.07, 3.05, 3.50 [13:37:28] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 6.25, 3.95, 3.74 [13:39:24] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.39, 3.23, 3.50 [13:41:19] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.82, 2.67, 3.25 [13:50:20] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:03:51] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [14:10:34] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.11, 25.34, 20.35 [14:12:34] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.82, 21.70, 19.61 [14:12:35] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:18:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.76, 18.50, 18.90 [14:26:06] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [14:54:58] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 8.84, 5.81, 2.96 [14:56:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.69, 21.01, 17.29 [14:58:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.36, 17.92, 16.57 [14:58:49] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.70, 2.92, 2.41 [15:06:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.81, 17.73, 16.52 [15:08:04] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.82, 17.59, 16.60 [15:15:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Js8fa [15:15:12] [02miraheze/services] 07MirahezeSSLBot 035c3a8a3 - BOT: Updating services config for wikis [15:42:25] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [15:55:42] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 3.46, 3.14, 1.83 [15:55:57] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [15:57:42] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.49, 2.49, 1.75 [16:00:15] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.32, 9.63, 5.78 [16:07:56] > ^^ ? MacFan4000, that was just @MarioMario456 setting up the Title Blacklists on multiple wikis. [16:12:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.39, 2.12, 3.64 [16:14:14] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.53, 1.88, 3.36 [16:24:14] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.75, 5.43, 4.16 [16:28:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.08, 3.01, 3.45 [16:30:14] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.37, 2.41, 3.17 [16:32:00] PROBLEM - wiki.umbrel.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.umbrel.tech could not be found [16:52:57] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.45, 3.42, 2.77 [16:54:53] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.10, 3.29, 2.82 [17:02:57] !sre Atrocious load times on `metawiki` and JavaScripts will not load [17:04:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.38, 19.20, 16.39 [17:06:20] Void's Mass Global Block script isn't loading. It usually loads after a second or two, and it does load, but it's greyed out. I just used it too, so it's not the script [17:06:32] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.25, 17.01, 15.94 [17:06:46] oh interesting [17:06:49] nevermind [17:07:04] if you don't have `flood` on, it's greyed out until you click "group enforcer" [17:07:18] but if `flood` is added manually, then it's not greyed out [17:12:21] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [17:17:43] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 8.64, 5.32, 2.64 [17:19:43] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.70, 3.75, 2.39 [17:21:43] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.58, 2.62, 2.14 [17:25:52] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [17:30:50] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.gyaanipedia.com could not be found [17:30:51] PROBLEM - www.burnout.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.burnout.wiki could not be found [17:31:08] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 6.29, 7.67, 4.68 [17:36:55] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.80, 3.16, 3.56 [17:37:40] RECOVERY - www.burnout.wiki - reverse DNS on sslhost is OK: rDNS OK - www.burnout.wiki reverse DNS resolves to cp10.miraheze.org [17:37:49] RECOVERY - bn.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - bn.gyaanipedia.com reverse DNS resolves to cp10.miraheze.org [17:38:50] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.56, 2.32, 3.20 [17:46:28] Maybe I should add a help notice on that :P [17:47:47] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki' (a few hours ago; forgot to log) on mw* [17:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:06] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [17:48:49] !log puppet3 hack to test sshfs perf optimisations [17:48:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:51:01] @Void that issue you commented on is really strange, I changed the text of the i18n message to "Testing 1234" and that worked [17:51:09] but when I change it back to next page it stops working again [17:54:08] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Js8w7 [17:54:10] [02miraheze/puppet] 07Southparkfan 032d2a537 - mariadb: give dbbackup_user extra grant [17:56:14] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.72, 4.85, 3.78 [17:58:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.82, 3.41, 3.38 [18:00:14] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.41, 2.44, 3.03 [18:01:36] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [18:04:16] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.19, 4.21, 3.63 [18:06:14] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.29, 3.01, 3.26 [18:18:13] !log revert puppet3 changes [18:18:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:19] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [18:40:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:40:52] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:41:03] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.40, 6.91, 4.07 [18:41:14] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.63, 22.59, 17.63 [18:41:44] PROBLEM - cp11 Stunnel Http for mw10 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:41:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 51.195.236.219/cpweb, 2001:41d0:800:178a::5/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:41:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.04, 5.37, 3.36 [18:42:14] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:15] that's likely me, sorry [18:42:15] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:44] should be recovering now [18:44:15] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 4.754 second response time [18:44:19] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20889 bytes in 4.061 second response time [18:45:01] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 1.581 second response time [18:45:03] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [18:45:03] PROBLEM - cubashpedia.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cubashpedia.cf could not be found [18:45:06] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [18:45:18] PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yunachannel.com could not be found [18:45:18] PROBLEM - wiki.kali-team.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kali-team.cn could not be found [18:45:20] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc-multiverse.dcwikis.com could not be found [18:45:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:45:52] RECOVERY - cp11 Stunnel Http for mw10 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 0.015 second response time [18:46:42] (or it was merely coincidence) [18:48:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:49:21] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thunis.eu could not be found [18:49:26] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.permanentfuturelab.wiki could not be found [18:49:34] PROBLEM - portalsofphereon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for portalsofphereon.com could not be found [18:49:36] PROBLEM - wiki.patriam.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.patriam.cc could not be found [18:49:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [18:50:59] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [18:51:58] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikiru.wildterra2.com reverse DNS resolves to cp12.miraheze.org [18:51:59] RECOVERY - cubashpedia.cf - reverse DNS on sslhost is OK: rDNS OK - cubashpedia.cf reverse DNS resolves to cp12.miraheze.org [18:52:05] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: rDNS OK - storytime.jdstroy.cf reverse DNS resolves to cp12.miraheze.org [18:52:07] RECOVERY - wiki.yunachannel.com - reverse DNS on sslhost is OK: rDNS OK - wiki.yunachannel.com reverse DNS resolves to cp10.miraheze.org [18:52:07] RECOVERY - wiki.kali-team.cn - reverse DNS on sslhost is OK: rDNS OK - wiki.kali-team.cn reverse DNS resolves to cp10.miraheze.org [18:52:09] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: rDNS OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp11.miraheze.org [18:53:24] PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found [18:53:26] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [18:53:28] PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for runzeppelin.ru could not be found [18:53:29] PROBLEM - sopel.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sopel.bots.miraheze.wiki could not be found [18:53:29] PROBLEM - allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.org could not be found [18:53:30] PROBLEM - private.revi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for private.revi.wiki could not be found [18:53:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:54:12] PROBLEM - wiki.umbrel.tech - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.umbrel.tech' expires in 15 day(s) (Tue 01 Jun 2021 18:51:21 GMT +0000). [18:56:08] RECOVERY - www.permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - www.permanentfuturelab.wiki reverse DNS resolves to cp11.miraheze.org [18:56:20] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.thunis.eu reverse DNS resolves to cp10.miraheze.org [18:56:28] RECOVERY - portalsofphereon.com - reverse DNS on sslhost is OK: rDNS OK - portalsofphereon.com reverse DNS resolves to cp10.miraheze.org [18:56:29] RECOVERY - wiki.patriam.cc - reverse DNS on sslhost is OK: rDNS OK - wiki.patriam.cc reverse DNS resolves to cp11.miraheze.org [18:59:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.82, 5.44, 5.22 [19:00:07] RECOVERY - mrlove.wiki - reverse DNS on sslhost is OK: rDNS OK - mrlove.wiki reverse DNS resolves to cp10.miraheze.org [19:00:08] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - miraheze.wiki reverse DNS resolves to cp10.miraheze.org [19:00:13] RECOVERY - runzeppelin.ru - reverse DNS on sslhost is OK: rDNS OK - runzeppelin.ru reverse DNS resolves to cp10.miraheze.org [19:00:14] RECOVERY - private.revi.wiki - reverse DNS on sslhost is OK: rDNS OK - private.revi.wiki reverse DNS resolves to cp10.miraheze.org [19:00:17] RECOVERY - allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.org reverse DNS resolves to cp10.miraheze.org [19:00:26] RECOVERY - sopel.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - sopel.bots.miraheze.wiki reverse DNS resolves to cp10.miraheze.org [19:01:52] PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.astralprojections.org' expires in 15 day(s) (Tue 01 Jun 2021 18:52:59 GMT +0000). [19:03:44] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.20, 5.86, 5.44 [19:05:42] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.28, 5.57, 5.39 [19:06:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Js8XC [19:06:49] [02miraheze/ssl] 07MirahezeSSLBot 03efb8b65 - Bot: Update SSL cert for wiki.astralprojections.org [19:09:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.87, 4.54, 5.00 [19:11:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.56, 19.19, 23.27 [19:13:07] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.70, 20.59, 23.23 [19:15:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.97, 18.92, 22.31 [19:18:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.32, 2.67, 3.95 [19:20:14] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.17, 3.64, 4.14 [19:21:05] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.26, 21.46, 21.98 [19:25:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.03, 21.61, 22.05 [19:26:46] https://phabricator.miraheze.org/T7318 [19:26:47] [ ⚓ T7318 CLG is down ] - phabricator.miraheze.org [19:27:02] paladox Reception123 Crepusculum [19:27:08] SPF|Cloud [19:27:14] MarioMario456: works for me. [19:27:31] There is 502 Bad Gateway for me [19:27:32] check [19:27:50] Yeah doesn't happen for me. [19:27:57] also works for me, cannot reproduce [19:27:59] dmehus: ^ can you? [19:28:04] going to check on incognite [19:28:05] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [19:28:14] ^ ah, that will explain it [19:28:16] Reception123: I can reproduce [19:28:18] a location issue [19:28:22] just came here to report this [19:28:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [19:28:26] looking [19:28:29] !sre 502 bad gateway on https://wintergatancommunity.miraheze.org/ [19:28:31] [ Wintergatan Community Wiki ] - wintergatancommunity.miraheze.org [19:28:35] it works on incognnito [19:28:47] thanks SPF :) [19:28:49] !log restart varnish on cp12 [19:28:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [19:29:12] MarioMario456, thanks, that's interesting, could be the way anonymous users are routed to our cache proxies [19:29:27] PROBLEM - allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.org could not be found [19:29:31] should work now, looks like varnish crashed? [19:29:34] PROBLEM - piwik.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for piwik.miraheze.org could not be found [19:29:34] PROBLEM - wiki.jumpbound.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jumpbound.com could not be found [19:29:35] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.net could not be found [19:29:35] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [19:29:35] PROBLEM - warmage.necrocity.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for warmage.necrocity.co.uk could not be found [19:29:46] SPF|Cloud: yep, works :) [19:29:49] ah.. OOM [19:29:53] oh [19:30:01] a lovely OOM :( [19:30:05] RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 4% [19:30:23] it got fixed [19:30:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:30:25] I was doing something on that server around the crash [19:30:30] oh [19:30:36] I'll stop... [19:30:41] resource intensive? [19:30:42] heh [19:30:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:33:05] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.11, 18.36, 20.23 [19:36:06] RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Sat 14 Aug 2021 18:06:42 GMT +0000. [19:36:15] RECOVERY - allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.org reverse DNS resolves to cp11.miraheze.org [19:36:27] RECOVERY - piwik.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - piwik.miraheze.org reverse DNS resolves to cp11.miraheze.org [19:36:27] RECOVERY - wiki.jumpbound.com - reverse DNS on sslhost is OK: rDNS OK - wiki.jumpbound.com reverse DNS resolves to cp10.miraheze.org [19:36:27] RECOVERY - wiki.seamly.net - reverse DNS on sslhost is OK: rDNS OK - wiki.seamly.net reverse DNS resolves to cp11.miraheze.org [19:36:29] RECOVERY - warmage.necrocity.co.uk - reverse DNS on sslhost is OK: rDNS OK - warmage.necrocity.co.uk reverse DNS resolves to cp11.miraheze.org [19:36:31] RECOVERY - vitriol.spoon.army - reverse DNS on sslhost is OK: rDNS OK - vitriol.spoon.army reverse DNS resolves to cp10.miraheze.org [19:37:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::1c30/cpweb [19:37:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.16, 23.28, 22.05 [19:37:18] SPF|Cloud: ns1? [19:37:18] RhinosF1: 2021-05-16 - 03:10:09BST tell RhinosF1 I email'd you something regarding trollpasta.com a few days ago [19:37:26] Whose that [19:37:48] Oh yes [19:38:27] SPF|Cloud: also https://phabricator.miraheze.org/T6952 [19:38:27] [ ⚓ T6952 cp12 OOM'd March 12 2021 16:36 ] - phabricator.miraheze.org [19:38:37] The out of space was seen then [19:39:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:39:18] hm.. [19:39:53] SPF|Cloud, I'm assuming downtime wasn't long enough to suggest we do an incident report, or do you still want to do an incident report for this one? [19:40:26] makes sense to have one [19:40:31] dmehus: given we haven't followed up on a 8 week old lessons learnt then I think we need one [19:40:32] yeah [19:40:40] RhinosF1, yeah I agree [19:40:44] Because there's aggregating factors [19:41:04] if nothing else, they're a good way to summarize the key causes for both incidents and propose solutions [19:41:08] that too [19:41:09] Should john be given phabricator admin? He's already Miraheze Staff [19:41:15] MarioMario456: no [19:41:20] He left SRE [19:41:20] John resigned, MarioMario456 [19:41:25] He is still a Steward though [19:41:26] oh [19:41:39] RhinosF1: it's not fair to say we haven't followed up [19:41:58] actions have been taken since then [19:42:27] SPF|Cloud: then they didn't work. There's definitely something to learn. [19:43:03] I read your comment as 'nothing has been done since' [19:43:06] well, in fairness, RhinosF1, SPF|Cloud didn't say there's not something we need to learn; he's just saying it's unfair to say we didn't look into it [19:44:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.01, 2.44, 3.82 [19:44:22] SPF|Cloud: yeah I assumed we'd simply missed it not that it hadn't worked. Still have the same issue seen twice though as a factor. [19:44:29] dmehus: I know [19:44:59] RhinosF1: ack [19:47:05] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.20, 17.87, 19.89 [19:48:06] Just seen reply on task [19:48:15] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.04, 2.11, 3.37 [19:48:26] https://www.freedesktop.org/software/systemd/man/systemd.service.html seems to say it should be [19:48:26] [ systemd.service ] - www.freedesktop.org [19:51:15] dmehus: are you going to write the IR? (for clarity) [19:53:10] SPF|Cloud: don't think Doug has edit-incidents [19:53:35] I can do it after my 'break', as long as we determine who is going to write the report I don't mind [19:53:48] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [19:53:52] I'd say you probably best [19:54:06] acknowledged [19:56:38] afk [20:07:21] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [20:16:03] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [20:32:14] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 6.49, 4.36, 2.71 [20:33:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.64, 22.70, 18.78 [20:35:07] SPF|Cloud: heh I could, but yeah what RhinosF1 said. Only SRE have `edit-incidents` [20:35:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.80, 19.59, 18.15 [20:36:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.35, 3.56, 2.87 [20:40:15] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.77, 2.78, 2.81 [20:48:16] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.97, 4.64, 3.56 [20:49:51] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp11.miraheze.org [20:50:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 37.89, 26.48, 20.89 [20:54:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.29, 23.81, 21.16 [20:55:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.99, 5.43, 4.38 [20:56:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.33, 3.86, 3.84 [20:58:14] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.51, 4.89, 4.26 [20:58:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.61, 20.21, 20.33 [20:59:36] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.91, 4.60, 4.31 [21:00:18] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.56, 3.67, 3.90 [21:01:45] PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitaldesignhq.com could not be found [21:06:18] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.32, 2.34, 3.19 [21:08:24] RECOVERY - wiki.digitaldesignhq.com - reverse DNS on sslhost is OK: rDNS OK - wiki.digitaldesignhq.com reverse DNS resolves to cp11.miraheze.org [21:12:03] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [21:18:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.52, 21.73, 19.69 [21:20:31] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 12.01, 6.36, 4.11 [21:25:01] PROBLEM - www.wikilukas.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikilukas.tk could not be found [21:25:01] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [21:25:04] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [21:25:04] PROBLEM - test1.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for test1.miraheze.org could not be found [21:26:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.09, 22.45, 21.49 [21:28:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.31, 23.39, 21.88 [21:30:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.44, 20.16, 20.91 [21:31:54] RECOVERY - www.wikilukas.tk - reverse DNS on sslhost is OK: rDNS OK - www.wikilukas.tk reverse DNS resolves to cp10.miraheze.org [21:31:54] RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: rDNS OK - n64brew.dev reverse DNS resolves to cp10.miraheze.org [21:32:00] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 3.32, 4.55, 2.73 [21:32:00] RECOVERY - test1.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - test1.miraheze.org reverse DNS resolves to cp11.miraheze.org [21:32:00] RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: rDNS OK - www.mh142.com reverse DNS resolves to cp10.miraheze.org [21:32:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.72, 18.82, 20.25 [21:33:55] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.65, 3.18, 2.44 [21:36:15] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.92, 2.52, 3.98 [21:38:15] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.69, 4.55, 4.53 [21:44:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.51, 2.60, 3.73 [21:45:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [21:46:15] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.33, 1.86, 3.32 [21:54:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.02, 17.07, 16.92 [21:54:37] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [21:56:15] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.39, 14.19, 15.87 [22:06:17] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.09, 4.63, 3.33 [22:08:08] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp10.miraheze.org [22:08:14] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.33, 3.43, 3.04 [22:10:14] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.54, 2.53, 2.76 [22:50:38] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [23:15:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.48, 6.00, 4.55 [23:18:58] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.97, 7.73, 5.43 [23:19:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.75, 6.91, 5.26 [23:20:03] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:04] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:17] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:22] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [23:20:26] PROBLEM - heavyironmodding.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for heavyironmodding.org could not be found [23:21:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.32, 8.33, 6.02 [23:21:58] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20889 bytes in 0.306 second response time [23:21:59]