[00:00:23] PROBLEM - cp11 HTTP 4xx/5xx ERROR Rate on cp11 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:01:51] PROBLEM - jobrunner2 Current Load on jobrunner2 is CRITICAL: CRITICAL - load average: 6.12, 5.94, 5.94 [00:02:23] PROBLEM - cp11 HTTP 4xx/5xx ERROR Rate on cp11 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:32] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 36.64, 25.65, 20.64 [00:03:23] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 4.75, 3.40, 2.90 [00:03:44] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 9.42, 4.80, 2.68 [00:04:49] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.50, 6.35, 4.82 [00:05:43] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 3.02, 3.90, 2.60 [00:07:19] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 1.92, 2.87, 2.82 [00:10:35] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 13.20, 21.65, 21.89 [00:10:48] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.88, 5.95, 5.14 [00:12:47] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 3.71, 5.04, 4.90 [00:16:32] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.66, 18.15, 20.18 [00:20:55] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.05, 20.21, 18.12 [00:22:55] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 20.35, 19.98, 18.28 [00:43:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtEXE [00:43:50] [02miraheze/mediawiki] 07Universal-Omega 0305f6952 - Update MatomoAnalytics [00:49:43] PROBLEM - jobrunner2 Current Load on jobrunner2 is WARNING: WARNING - load average: 4.86, 5.46, 5.94 [00:52:47] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:54:46] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:59:42] RECOVERY - jobrunner2 Current Load on jobrunner2 is OK: OK - load average: 3.35, 4.12, 5.04 [01:01:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtEXh [01:01:49] [02miraheze/mediawiki] 07Universal-Omega 03ae5cbd4 - Update DynamicPageList3 [01:05:42] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.34, 21.68, 19.05 [01:07:38] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.63, 21.55, 19.31 [01:09:34] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.55, 26.05, 21.09 [01:10:48] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:10:54] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:11:33] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:12:00] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:12:41] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:13:33] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:15:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.74, 22.21, 21.17 [01:19:17] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.39, 19.03, 20.09 [01:26:27] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.95, 3.25, 2.96 [01:28:28] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 3.12, 3.06, 2.92 [01:33:55] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [01:34:15] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:34:39] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [01:34:47] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:34:52] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:35:55] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:49:29] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 5.87, 3.94, 3.19 [01:49:41] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 39.26, 29.22, 22.33 [01:50:16] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.66, 5.41, 2.64 [01:51:28] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 4.00, 3.81, 3.23 [01:54:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [01:56:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:59:23] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 4.50, 4.00, 3.51 [02:01:24] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.49, 5.66, 3.71 [02:03:37] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.36, 4.85, 3.68 [02:05:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [02:05:46] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:00] oops [02:06:02] uh [02:06:23] PROBLEM - jobrunner2 Current Load on jobrunner2 is WARNING: WARNING - load average: 5.29, 5.12, 4.56 [02:06:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2607:5300:205:200::1c30/cpweb [02:07:20] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:07:59] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.91, 5.65, 4.27 [02:08:50] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lefrenchmelee.fr could not be found [02:10:05] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.93, 5.23, 4.28 [02:12:12] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 4.40, 5.06, 4.35 [02:12:17] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 2.63, 3.88, 3.97 [02:13:12] PROBLEM - cp12 Stunnel Http for mw6 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [02:13:50] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:59] PROBLEM - cp12 Stunnel Http for mw6 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:59] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:14:31] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.25, 17.96, 9.23 [02:15:07] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 20679 bytes in 6.087 second response time [02:15:09] RECOVERY - jobrunner2 Current Load on jobrunner2 is OK: OK - load average: 3.29, 4.49, 4.62 [02:15:43] RECOVERY - cp12 Stunnel Http for mw6 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15145 bytes in 0.342 second response time [02:16:15] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.004 second response time [02:16:15] RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: rDNS OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp7.miraheze.org [02:16:19] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 1.401 second response time [02:16:20] RECOVERY - cp12 Stunnel Http for mw6 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 1.361 second response time [02:16:38] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.30, 18.56, 8.77 [02:16:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.01, 20.26, 11.36 [02:17:19] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 5.91, 4.45, 4.13 [02:17:28] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw7 [02:17:39] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.55, 6.22, 5.06 [02:19:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.34, 22.41, 13.29 [02:20:10] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.74, 5.61, 5.04 [02:21:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.72, 21.26, 14.26 [02:22:08] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [02:22:26] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 11.69, 7.51, 5.78 [02:22:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:22:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.62, 20.49, 14.44 [02:22:52] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.80, 27.07, 15.89 [02:23:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.23, 21.90, 15.32 [02:25:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.80, 23.20, 16.40 [02:25:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.66, 22.42, 16.42 [02:27:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.69, 22.30, 16.89 [02:27:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:29:13] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.96, 5.68, 5.53 [02:31:48] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.21, 22.23, 18.18 [02:34:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.222.25.132/cpweb [02:34:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.77.107.210/cpweb, 51.89.160.142/cpweb [02:35:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.58, 24.73, 19.87 [02:35:14] PROBLEM - phab2 Current Load on phab2 is CRITICAL: CRITICAL - load average: 2.30, 1.43, 0.86 [02:35:30] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 2.48, 3.43, 3.89 [02:36:01] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 9.15, 7.19, 6.03 [02:36:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:37:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:37:39] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 4.54, 3.68, 3.90 [02:38:09] RECOVERY - phab2 Current Load on phab2 is OK: OK - load average: 1.26, 1.45, 0.97 [02:39:50] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.56, 3.72, 3.90 [02:40:23] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.09, 5.99, 5.87 [02:44:06] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 4.59, 3.45, 3.69 [02:44:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 4.80, 19.59, 21.27 [02:44:38] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.05, 5.82, 5.80 [02:44:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 3.39, 17.59, 20.53 [02:46:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 2.12, 13.58, 18.85 [02:46:22] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.70, 3.63, 3.73 [02:46:54] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.60, 5.11, 5.55 [02:47:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 2.38, 11.88, 18.00 [02:48:41] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 5.44, 4.05, 3.85 [02:48:58] PROBLEM - ping4 on phab2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 266.02 ms [02:49:25] PROBLEM - ping6 on phab2 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 848.05 ms [02:50:15] PROBLEM - wiki.meeusen.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.meeusen.net could not be found [02:50:39] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 2.65, 3.47, 3.67 [02:50:58] RECOVERY - ping4 on phab2 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms [02:51:27] RECOVERY - ping6 on phab2 is OK: PING OK - Packet loss = 0%, RTA = 0.35 ms [02:52:12] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 4.14, 17.41, 22.13 [02:52:14] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 3.80, 17.12, 22.01 [02:54:11] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 2.59, 12.30, 19.67 [02:54:12] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 2.59, 12.30, 19.67 [02:56:45] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 5.03, 3.85, 3.68 [02:57:04] RECOVERY - wiki.meeusen.net - reverse DNS on sslhost is OK: rDNS OK - wiki.meeusen.net reverse DNS resolves to cp6.miraheze.org [02:57:23] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.99, 4.54, 5.06 [02:58:43] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.48, 3.61, 3.61 [03:01:02] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 5.28, 4.37, 3.91 [03:02:08] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.44, 5.04, 5.14 [03:03:47] PROBLEM - nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonbinary.wiki could not be found [03:03:49] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minecraftathome.com could not be found [03:04:17] PROBLEM - hololive.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hololive.wiki could not be found [03:04:30] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 7.16, 6.09, 5.53 [03:06:32] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.12, 5.25, 5.29 [03:08:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JtEbJ [03:08:17] [02miraheze/mw-config] 07paladox 0315754c3 - Add site notice [03:08:18] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [03:08:20] [02mw-config] 07paladox opened pull request 03#3705: Add site notice - 13https://git.io/JtEbU [03:08:40] dat nagios? [03:08:56] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.98, 6.18, 5.64 [03:09:36] miraheze/mw-config - paladox the build passed. [03:10:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtEbm [03:10:08] [02miraheze/mw-config] 07paladox 0313a6070 - Switch dreamversewiki to use new infrastructure [03:10:45] RECOVERY - nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - nonbinary.wiki reverse DNS resolves to cp6.miraheze.org [03:10:49] catbeard Well not exactly, its icinga [03:10:53] RECOVERY - wiki.minecraftathome.com - reverse DNS on sslhost is OK: rDNS OK - wiki.minecraftathome.com reverse DNS resolves to cp6.miraheze.org [03:11:05] blegh [03:11:16] miraheze/mw-config - paladox the build passed. [03:11:18] RECOVERY - hololive.wiki - reverse DNS on sslhost is OK: rDNS OK - hololive.wiki reverse DNS resolves to cp6.miraheze.org [03:11:18] there's some things we have to use icingi for [03:11:32] but we're the recipient of the alert, not the originator [03:11:34] [02mw-config] 07paladox closed pull request 03#3705: Add site notice - 13https://git.io/JtEbU [03:11:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtEb3 [03:11:37] [02miraheze/mw-config] 07paladox 033219de3 - Add site notice (#3705) [03:11:54] you should check out nixstats [03:12:12] and write your own monitoring modules based on data you manage (: [03:12:36] miraheze/mw-config - paladox the build passed. [03:15:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::1c30/cpweb [03:15:52] PROBLEM - wiki.jafaron.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jafaron.net could not be found [03:15:52] PROBLEM - kunwok.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kunwok.org could not be found [03:15:54] PROBLEM - warmage.necrocity.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for warmage.necrocity.co.uk could not be found [03:15:56] PROBLEM - allthetropes.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.orain.org could not be found [03:15:57] PROBLEM - wiki.sdbx.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sdbx.dev could not be found [03:15:57] PROBLEM - wiki.yumeka.team - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yumeka.team could not be found [03:15:58] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [03:15:58] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vedopedia.witches-empire.com could not be found [03:16:12] PROBLEM - wiki.s23.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.s23.org could not be found [03:16:12] PROBLEM - wiki.raghuveer.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.raghuveer.net could not be found [03:16:15] PROBLEM - commonswiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commonswiki.miraheze.org could not be found [03:16:15] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.qadrishattari.xyz could not be found [03:17:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:21:52] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.97, 5.25, 5.93 [03:22:25] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 2.60, 3.67, 3.99 [03:22:34] RECOVERY - warmage.necrocity.co.uk - reverse DNS on sslhost is OK: rDNS OK - warmage.necrocity.co.uk reverse DNS resolves to cp6.miraheze.org [03:22:38] RECOVERY - kunwok.org - reverse DNS on sslhost is OK: rDNS OK - kunwok.org reverse DNS resolves to cp6.miraheze.org [03:22:38] RECOVERY - wiki.yumeka.team - reverse DNS on sslhost is OK: rDNS OK - wiki.yumeka.team reverse DNS resolves to cp6.miraheze.org [03:22:39] RECOVERY - wiki.jafaron.net - reverse DNS on sslhost is OK: rDNS OK - wiki.jafaron.net reverse DNS resolves to cp7.miraheze.org [03:22:43] RECOVERY - allthetropes.orain.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.orain.org reverse DNS resolves to cp7.miraheze.org [03:22:51] RECOVERY - wiki.sdbx.dev - reverse DNS on sslhost is OK: rDNS OK - wiki.sdbx.dev reverse DNS resolves to cp7.miraheze.org [03:22:53] RECOVERY - wiki.s23.org - reverse DNS on sslhost is OK: rDNS OK - wiki.s23.org reverse DNS resolves to cp7.miraheze.org [03:22:54] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - en.gyaanipedia.com reverse DNS resolves to cp6.miraheze.org [03:22:55] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: rDNS OK - vedopedia.witches-empire.com reverse DNS resolves to cp6.miraheze.org [03:23:05] RECOVERY - commonswiki.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - commonswiki.miraheze.org reverse DNS resolves to cp7.miraheze.org [03:23:07] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.qadrishattari.xyz reverse DNS resolves to cp7.miraheze.org [03:23:24] RECOVERY - wiki.raghuveer.net - reverse DNS on sslhost is OK: rDNS OK - wiki.raghuveer.net reverse DNS resolves to cp6.miraheze.org [03:24:43] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 5.71, 4.67, 4.32 [03:24:55] [02miraheze/mw-config] 07Universal-Omega deleted branch 03paladox-patch-3 [03:24:57] [02mw-config] 07Universal-Omega deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [03:27:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtEbX [03:27:11] [02miraheze/mw-config] 07Universal-Omega 03992992a - expand on sitenotice [03:27:46] PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found [03:27:47] PROBLEM - savagepedia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savagepedia.wiki could not be found [03:28:39] miraheze/mw-config - Universal-Omega the build passed. [03:32:45] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.68, 5.27, 5.36 [03:34:42] RECOVERY - savagepedia.wiki - reverse DNS on sslhost is OK: rDNS OK - savagepedia.wiki reverse DNS resolves to cp6.miraheze.org [03:34:49] RECOVERY - wiki.wysteria.ga - reverse DNS on sslhost is OK: rDNS OK - wiki.wysteria.ga reverse DNS resolves to cp6.miraheze.org [03:34:56] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.56, 5.01, 5.25 [03:39:08] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.94, 4.60, 5.04 [03:55:06] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minecraftathome.com could not be found [03:55:09] PROBLEM - wiki.globasa.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.globasa.net could not be found [03:59:37] PROBLEM - cloud2 APT on cloud2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:01:45] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 98 packages available for upgrade (0 critical updates). [04:01:59] RECOVERY - wiki.minecraftathome.com - reverse DNS on sslhost is OK: rDNS OK - wiki.minecraftathome.com reverse DNS resolves to cp6.miraheze.org [04:02:08] RECOVERY - wiki.globasa.net - reverse DNS on sslhost is OK: rDNS OK - wiki.globasa.net reverse DNS resolves to cp7.miraheze.org [04:04:49] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.56, 3.48, 3.98 [04:07:09] PROBLEM - mon1 Current Load on mon1 is CRITICAL: CRITICAL - load average: 5.26, 3.89, 4.04 [04:10:47] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 8.84, 8.14, 5.96 [04:12:58] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.92, 5.93, 5.42 [04:13:40] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 2.16, 3.50, 3.86 [04:15:03] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.61, 4.87, 5.10 [04:17:48] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 0.89, 2.33, 3.31 [04:28:20] PROBLEM - wiki.grottocenter.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grottocenter.org could not be found [04:28:24] PROBLEM - commonswiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commonswiki.miraheze.org could not be found [04:28:32] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.sdtef.org could not be found [04:28:33] PROBLEM - theatlas.pw - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for theatlas.pw could not be found [04:28:35] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hi.gyaanipedia.com could not be found [04:28:37] PROBLEM - meta.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.nocyclo.tk could not be found [04:28:38] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [04:28:38] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.qadrishattari.xyz could not be found [04:28:40] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [04:28:40] PROBLEM - froggy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for froggy.info could not be found [04:33:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [04:35:09] RECOVERY - wiki.grottocenter.org - reverse DNS on sslhost is OK: rDNS OK - wiki.grottocenter.org reverse DNS resolves to cp7.miraheze.org [04:35:15] RECOVERY - commonswiki.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - commonswiki.miraheze.org reverse DNS resolves to cp6.miraheze.org [04:35:15] RECOVERY - hi.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - hi.gyaanipedia.com reverse DNS resolves to cp7.miraheze.org [04:35:16] RECOVERY - meta.nocyclo.tk - reverse DNS on sslhost is OK: rDNS OK - meta.nocyclo.tk reverse DNS resolves to cp6.miraheze.org [04:35:21] RECOVERY - history.sdtef.org - reverse DNS on sslhost is OK: rDNS OK - history.sdtef.org reverse DNS resolves to cp6.miraheze.org [04:35:22] RECOVERY - theatlas.pw - reverse DNS on sslhost is OK: rDNS OK - theatlas.pw reverse DNS resolves to cp6.miraheze.org [04:35:22] RECOVERY - halloween.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - halloween.gyaanipedia.com reverse DNS resolves to cp6.miraheze.org [04:35:29] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - miraheze.wiki reverse DNS resolves to cp7.miraheze.org [04:35:29] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.qadrishattari.xyz reverse DNS resolves to cp6.miraheze.org [04:35:40] RECOVERY - froggy.info - reverse DNS on sslhost is OK: rDNS OK - froggy.info reverse DNS resolves to cp7.miraheze.org [04:36:54] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.02, 5.79, 5.29 [04:39:02] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.68, 5.03, 5.07 [04:39:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:58:51] PROBLEM - mediawiki.ecfin.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mediawiki.ecfin.eu could not be found [04:58:51] PROBLEM - dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dariawiki.org could not be found [05:03:39] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.67, 2.96, 2.49 [05:05:33] RECOVERY - dariawiki.org - reverse DNS on sslhost is OK: rDNS OK - dariawiki.org reverse DNS resolves to cp7.miraheze.org [05:05:37] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 1.87, 2.57, 2.41 [05:05:50] RECOVERY - mediawiki.ecfin.eu - reverse DNS on sslhost is OK: rDNS OK - mediawiki.ecfin.eu reverse DNS resolves to cp6.miraheze.org [05:09:49] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.59, 3.31, 2.74 [05:11:46] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 1.71, 2.61, 2.54 [05:12:30] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [05:17:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::1c30/cpweb [05:19:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:19:21] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - miraheze.wiki reverse DNS resolves to cp6.miraheze.org [05:36:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb [05:38:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:48:11] PROBLEM - private.revi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for private.revi.wiki could not be found [05:48:11] PROBLEM - dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dariawiki.org could not be found [05:48:15] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [05:51:07] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.09, 5.60, 4.61 [05:53:12] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.80, 4.82, 4.44 [05:54:54] RECOVERY - dariawiki.org - reverse DNS on sslhost is OK: rDNS OK - dariawiki.org reverse DNS resolves to cp7.miraheze.org [05:55:04] RECOVERY - private.revi.wiki - reverse DNS on sslhost is OK: rDNS OK - private.revi.wiki reverse DNS resolves to cp6.miraheze.org [05:55:07] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp6.miraheze.org [05:58:10] PROBLEM - holonet.pw - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for holonet.pw could not be found [05:58:11] PROBLEM - wiki.funkey-project.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.funkey-project.com could not be found [05:58:11] PROBLEM - wiki.ricochetuniverse.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ricochetuniverse.com could not be found [05:58:13] PROBLEM - wiki.slimelab.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimelab.net could not be found [06:04:53] RECOVERY - wiki.ricochetuniverse.com - reverse DNS on sslhost is OK: rDNS OK - wiki.ricochetuniverse.com reverse DNS resolves to cp6.miraheze.org [06:04:54] RECOVERY - wiki.slimelab.net - reverse DNS on sslhost is OK: rDNS OK - wiki.slimelab.net reverse DNS resolves to cp6.miraheze.org [06:04:58] RECOVERY - wiki.funkey-project.com - reverse DNS on sslhost is OK: rDNS OK - wiki.funkey-project.com reverse DNS resolves to cp6.miraheze.org [06:05:05] RECOVERY - holonet.pw - reverse DNS on sslhost is OK: rDNS OK - holonet.pw reverse DNS resolves to cp7.miraheze.org [06:06:26] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:09:29] PROBLEM - wiki.arnab.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arnab.at could not be found [06:09:29] PROBLEM - www.iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.iceria.org could not be found [06:10:20] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [06:10:21] PROBLEM - indiancannabis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for indiancannabis.wiki could not be found [06:10:21] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [06:13:19] PROBLEM - wiki.ct777.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ct777.cf could not be found [06:16:35] RECOVERY - wiki.arnab.at - reverse DNS on sslhost is OK: rDNS OK - wiki.arnab.at reverse DNS resolves to cp7.miraheze.org [06:16:36] RECOVERY - www.iceria.org - reverse DNS on sslhost is OK: rDNS OK - www.iceria.org reverse DNS resolves to cp6.miraheze.org [06:17:04] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 8.64, 6.86, 5.13 [06:17:12] RECOVERY - indiancannabis.wiki - reverse DNS on sslhost is OK: rDNS OK - indiancannabis.wiki reverse DNS resolves to cp6.miraheze.org [06:17:12] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - meta.gyaanipedia.com reverse DNS resolves to cp7.miraheze.org [06:17:13] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp6.miraheze.org [06:20:11] RECOVERY - wiki.ct777.cf - reverse DNS on sslhost is OK: rDNS OK - wiki.ct777.cf reverse DNS resolves to cp6.miraheze.org [06:21:30] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.87, 4.85, 4.76 [06:35:42] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:49:33] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [06:50:36] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:50:44] PROBLEM - cp12 Stunnel Http for mw6 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.245 second response time [06:50:47] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [06:50:56] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:51:11] PROBLEM - cp12 Stunnel Http for mw6 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:51:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb [06:51:38] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 21795 bytes in 0.232 second response time [06:52:02] PROBLEM - cloud2 Disk Space on cloud2 is WARNING: DISK WARNING - free space: / 190701 MB (10% inode=99%); [06:52:37] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.004 second response time [06:52:42] RECOVERY - cp12 Stunnel Http for mw6 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15139 bytes in 0.316 second response time [06:52:46] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.003 second response time [06:52:52] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15139 bytes in 0.006 second response time [06:53:06] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.40, 5.22, 4.95 [06:53:08] RECOVERY - cp12 Stunnel Http for mw6 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.345 second response time [06:53:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:53:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:55:13] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.65, 4.66, 4.78 [06:57:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.77.107.210/cpweb, 51.89.160.142/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [06:57:39] PROBLEM - kk.uncyc.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kk.uncyc.tk could not be found [06:57:58] PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found [06:58:08] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [06:58:14] back [06:58:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [06:58:36] Hello [06:58:39] Hi [06:58:45] just the bots lol [06:59:12] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:00:03] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 5.99, 5.80, 5.37 [07:00:22] PROBLEM - animefeet.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for animefeet.miraheze.org could not be found [07:00:23] PROBLEM - en.famepedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.famepedia.org could not be found [07:00:23] PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for translate.petrawiki.org could not be found [07:00:24] PROBLEM - intp.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for intp.miraheze.org could not be found [07:01:24] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 21795 bytes in 0.562 second response time [07:02:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:02:48] PROBLEM - www.highdeologywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.highdeologywiki.com could not be found [07:02:49] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thunis.eu could not be found [07:02:50] PROBLEM - www.burnout.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.burnout.wiki could not be found [07:02:50] PROBLEM - wiki.ct777.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ct777.cf could not be found [07:05:04] RECOVERY - kk.uncyc.tk - reverse DNS on sslhost is OK: rDNS OK - kk.uncyc.tk reverse DNS resolves to cp6.miraheze.org [07:05:18] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: rDNS OK - infectowiki.com reverse DNS resolves to cp6.miraheze.org [07:05:24] RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: rDNS OK - wiki.kourouklides.com reverse DNS resolves to cp6.miraheze.org [07:07:15] RECOVERY - translate.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - translate.petrawiki.org reverse DNS resolves to cp7.miraheze.org [07:07:20] RECOVERY - animefeet.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - animefeet.miraheze.org reverse DNS resolves to cp6.miraheze.org [07:07:21] RECOVERY - en.famepedia.org - reverse DNS on sslhost is OK: rDNS OK - en.famepedia.org reverse DNS resolves to cp7.miraheze.org [07:07:23] RECOVERY - intp.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - intp.miraheze.org reverse DNS resolves to cp6.miraheze.org [07:08:36] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 7.78, 6.78, 5.89 [07:09:30] RECOVERY - wiki.ct777.cf - reverse DNS on sslhost is OK: rDNS OK - wiki.ct777.cf reverse DNS resolves to cp6.miraheze.org [07:09:33] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.thunis.eu reverse DNS resolves to cp6.miraheze.org [07:09:34] RECOVERY - www.highdeologywiki.com - reverse DNS on sslhost is OK: rDNS OK - www.highdeologywiki.com reverse DNS resolves to cp6.miraheze.org [07:09:34] RECOVERY - www.burnout.wiki - reverse DNS on sslhost is OK: rDNS OK - www.burnout.wiki reverse DNS resolves to cp7.miraheze.org [07:10:00] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found [07:10:47] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.33, 5.39, 5.49 [07:11:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:13:10] shut up bots [07:14:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [07:14:34] stop disturbing me [07:14:45] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 8.61, 5.86, 5.58 [07:17:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [07:17:10] RECOVERY - wiki.ssij.se - reverse DNS on sslhost is OK: rDNS OK - wiki.ssij.se reverse DNS resolves to cp7.miraheze.org [07:18:58] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 5.41, 5.47, 5.46 [07:19:59] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [07:20:00] PROBLEM - oecumene.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for oecumene.org could not be found [07:20:01] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.permanentfuturelab.wiki could not be found [07:22:13] Is this wiki okay in terms of ToS? [07:22:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:23:08] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 11.87, 7.13, 6.00 [07:24:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:27:03] RECOVERY - oecumene.org - reverse DNS on sslhost is OK: rDNS OK - oecumene.org reverse DNS resolves to cp7.miraheze.org [07:27:03] RECOVERY - www.permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - www.permanentfuturelab.wiki reverse DNS resolves to cp6.miraheze.org [07:27:06] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - miraheze.wiki reverse DNS resolves to cp7.miraheze.org [07:27:13] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 5.78, 5.96, 5.73 [07:29:17] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 7.12, 6.08, 5.78 [07:29:58] PROBLEM - wiki.mikrodev.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mikrodev.com could not be found [07:31:38] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.27, 5.35, 5.55 [07:32:00] .deop [07:36:46] PROBLEM - cloud2 Disk Space on cloud2 is CRITICAL: DISK CRITICAL - free space: / 104109 MB (5% inode=99%); [07:36:51] RECOVERY - wiki.mikrodev.com - reverse DNS on sslhost is OK: rDNS OK - wiki.mikrodev.com reverse DNS resolves to cp6.miraheze.org [07:41:43] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.58, 4.29, 4.98 [07:41:46] Hello Firestar81! If you have any questions, feel free to ask and someone should answer soon. [07:41:56] bruh it's the same [07:48:09] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.57, 4.98, 5.13 [07:50:20] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.21, 4.42, 4.91 [07:59:32] back [08:07:47] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 2.54, 3.37, 5.70 [08:08:55] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 11.81, 13.38, 23.93 [08:11:44] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 2.24, 2.76, 4.92 [08:14:55] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 9.07, 11.88, 19.98 [09:04:34] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mobilityengineer.com could not be found [09:04:38] PROBLEM - aman.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for aman.awiki.org could not be found [09:04:38] PROBLEM - wiki.globasa.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.globasa.net could not be found [09:04:39] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fc.songcontests.eu could not be found [09:04:40] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found [09:04:42] PROBLEM - vmcodex.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vmcodex.net could not be found [09:04:44] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc-multiverse.dcwikis.com could not be found [09:04:45] PROBLEM - wiki.velaan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.velaan.org could not be found [09:04:50] PROBLEM - wiki.azimuth.navy - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.azimuth.navy could not be found [09:04:50] PROBLEM - wikipariksha.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikipariksha.com could not be found [09:05:38] RECOVERY - cloud2 Disk Space on cloud2 is OK: DISK OK - free space: / 805034 MB (45% inode=99%); [09:11:26] RECOVERY - wiki.mobilityengineer.com - reverse DNS on sslhost is OK: rDNS OK - wiki.mobilityengineer.com reverse DNS resolves to cp6.miraheze.org [09:11:26] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: rDNS OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp7.miraheze.org [09:11:28] RECOVERY - wiki.velaan.org - reverse DNS on sslhost is OK: rDNS OK - wiki.velaan.org reverse DNS resolves to cp6.miraheze.org [09:11:32] RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: rDNS OK - adadevelopersacademy.wiki reverse DNS resolves to cp7.miraheze.org [09:11:33] RECOVERY - wiki.azimuth.navy - reverse DNS on sslhost is OK: rDNS OK - wiki.azimuth.navy reverse DNS resolves to cp7.miraheze.org [09:11:33] RECOVERY - wikipariksha.com - reverse DNS on sslhost is OK: rDNS OK - wikipariksha.com reverse DNS resolves to cp7.miraheze.org [09:11:34] RECOVERY - wiki.globasa.net - reverse DNS on sslhost is OK: rDNS OK - wiki.globasa.net reverse DNS resolves to cp6.miraheze.org [09:11:36] RECOVERY - aman.awiki.org - reverse DNS on sslhost is OK: rDNS OK - aman.awiki.org reverse DNS resolves to cp6.miraheze.org [09:11:37] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: rDNS OK - fc.songcontests.eu reverse DNS resolves to cp6.miraheze.org [09:11:39] RECOVERY - vmcodex.net - reverse DNS on sslhost is OK: rDNS OK - vmcodex.net reverse DNS resolves to cp7.miraheze.org [09:14:18] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.04, 20.46, 15.91 [09:16:16] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.88, 18.32, 15.67 [09:38:13] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.97, 22.78, 18.85 [09:38:30] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 6.37, 6.60, 3.82 [09:40:31] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 1.28, 4.60, 3.42 [09:46:11] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 19.40, 20.34, 19.26 [09:49:44] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.31, 4.93, 4.35 [09:51:42] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 3.66, 4.58, 4.29 [09:52:49] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [09:54:08] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [09:54:11] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [09:54:37] PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [09:54:44] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:55:00] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [09:55:06] PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 100 packages available for upgrade (2 critical updates). [09:55:32] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:56:20] PROBLEM - gluster4 APT on gluster4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:56:35] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [09:57:19] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:58:20] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (3 critical updates). [09:58:28] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.86, 18.94, 9.16 [09:59:38] PROBLEM - glustermigrtemp1 APT on glustermigrtemp1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:59:38] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:59:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.46, 18.09, 9.16 [09:59:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.46, 18.09, 9.16 [09:59:47] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:00:20] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.70, 21.62, 11.26 [10:00:57] PROBLEM - rdb4 APT on rdb4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:02:28] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:02:32] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:02:54] PROBLEM - graylog1 APT on graylog1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:03:07] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:03:12] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:03:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.37, 23.48, 13.51 [10:03:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.79, 23.41, 13.60 [10:04:07] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:04:31] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:04:33] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:05:00] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:05:10] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:05:13] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:05:28] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:05:28] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:05:29] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:05:34] PROBLEM - phab2 Current Load on phab2 is CRITICAL: CRITICAL - load average: 2.85, 2.15, 1.11 [10:05:36] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [10:05:37] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [10:05:39] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 99 packages available for upgrade (2 critical updates). [10:05:47] PROBLEM - phab2 APT on phab2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:06:00] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:06:01] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:06:06] PROBLEM - rdb3 APT on rdb3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:06:11] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:06:51] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:07:35] PROBLEM - ping6 on phab2 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:07:56] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:08:00] PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [10:08:07] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:09:37] RECOVERY - ping6 on phab2 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms [10:11:38] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:11:44] PROBLEM - gluster2 APT on gluster2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:12:06] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:13:12] PROBLEM - puppet2 APT on puppet2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:13:34] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:15:39] PROBLEM - rdb2 APT on rdb2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:16:16] PROBLEM - phab2 Current Load on phab2 is WARNING: WARNING - load average: 0.48, 1.84, 1.93 [10:16:58] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:17:07] PROBLEM - ping4 on phab2 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 212.88 ms [10:17:21] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:18:36] PROBLEM - phab2 Current Load on phab2 is CRITICAL: CRITICAL - load average: 2.44, 2.04, 1.99 [10:19:53] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 81 packages available for upgrade (2 critical updates). [10:20:41] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (2 critical updates). [10:20:52] PROBLEM - phab2 Current Load on phab2 is WARNING: WARNING - load average: 0.98, 1.71, 1.87 [10:21:09] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 81 packages available for upgrade (2 critical updates). [10:21:10] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:21:11] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (2 critical updates). [10:21:21] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:21:25] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (2 critical updates). [10:21:38] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:22:02] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (2 critical updates). [10:22:08] PROBLEM - rdb1 APT on rdb1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [10:22:37] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [10:23:15] PROBLEM - phab2 Current Load on phab2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:23:20] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:23:24] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:26:19] RECOVERY - ping4 on phab2 is OK: PING OK - Packet loss = 0%, RTA = 90.06 ms [10:30:12] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.50, 23.08, 23.51 [10:32:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 3.49, 15.97, 20.85 [10:34:12] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 2.52, 11.59, 18.66 [10:34:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 2.44, 11.00, 18.31 [10:35:59] PROBLEM - phab2 phabricator.miraheze.org HTTPS on phab2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:56] PROBLEM - ping6 on phab2 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:37:55] RECOVERY - phab2 phabricator.miraheze.org HTTPS on phab2 is OK: HTTP OK: HTTP/1.1 200 OK - 18967 bytes in 0.170 second response time [10:38:58] RECOVERY - ping6 on phab2 is OK: PING OK - Packet loss = 0%, RTA = 0.39 ms [10:39:46] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 5.06, 17.35, 22.49 [10:40:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 2.74, 15.22, 21.55 [10:41:48] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 3.08, 12.48, 20.07 [10:42:26] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 3.32, 11.34, 19.36 [10:44:40] PROBLEM - phab2 Current Load on phab2 is WARNING: WARNING - load average: 0.02, 0.91, 1.90 [10:46:37] RECOVERY - phab2 Current Load on phab2 is OK: OK - load average: 0.13, 0.66, 1.68 [10:51:12] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 7.04, 5.54, 4.61 [10:53:11] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.58, 5.51, 4.71 [10:57:06] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.32, 5.81, 5.01 [10:59:08] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 4.66, 5.45, 4.97 [11:03:03] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.55, 5.77, 5.19 [11:05:05] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.66, 5.64, 5.21 [11:07:03] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.04, 5.64, 5.25 [11:07:42] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:09:00] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.13, 5.04, 5.07 [11:09:39] PROBLEM - cloud1 Puppet on cloud1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 day ago with 0 failures [11:12:57] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.56, 7.18, 5.92 [11:13:01] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 7.55, 5.33, 3.44 [11:15:01] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.50, 4.12, 3.22 [11:16:53] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.02, 5.80, 5.62 [11:34:36] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.07, 21.13, 18.67 [11:36:34] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.49, 4.77, 5.06 [11:38:34] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.80, 19.36, 18.59 [11:46:40] PROBLEM - bacula2 Bacula Static 1 on bacula2 is WARNING: WARNING: Full, 12804977 files, 596.3GB, 2021-01-19 11:44:00 (2.1 weeks ago) [11:49:22] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.54, 5.49, 5.13 [11:51:20] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.46, 5.45, 5.16 [12:03:13] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.87, 4.96, 5.07 [12:04:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.61, 21.91, 20.21 [12:07:14] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.77, 23.77, 19.29 [12:07:17] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.60, 5.74, 5.38 [12:08:20] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 18.26, 20.16, 19.97 [12:09:08] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.85, 23.33, 19.67 [12:11:11] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 2.93, 4.47, 4.95 [12:12:58] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.97, 18.77, 18.70 [12:31:47] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.79, 5.31, 4.68 [12:33:44] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.47, 5.26, 4.74 [12:35:42] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 7.22, 5.72, 4.95 [12:37:40] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 4.87, 5.43, 4.94 [12:39:38] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 3.89, 5.01, 4.85 [13:26:56] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:46:38] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.41, 4.71, 3.80 [13:50:34] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 7.06, 5.63, 4.35 [13:54:29] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 4.45, 5.42, 4.59 [13:56:28] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 2.77, 4.61, 4.41 [13:56:52] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:48:57] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JtuEv [14:48:59] [02miraheze/MirahezeMagic] 07paladox 03cbb317c - Fix domain for --server in generateSitemap [14:49:32] [02MirahezeMagic] 07paladox opened pull request 03#201: Fix domain for --server in generateSitemap - 13https://git.io/JtuEU [14:50:00] [02MirahezeMagic] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fQRGX [14:50:42] miraheze/MirahezeMagic - paladox the build passed. [14:51:15] [02miraheze/MirahezeMagic] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JtuEs [14:51:16] [02miraheze/MirahezeMagic] 07paladox 03469f949 - Merge pull request #201 from miraheze/paladox-patch-3 Fix domain for --server in generateSitemap [14:51:18] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 [14:52:11] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtuE4 [14:52:12] [02miraheze/mediawiki] 07paladox 03c16a5d6 - Update MirahezeMagic [14:52:21] miraheze/MirahezeMagic - paladox the build passed. [14:52:35] [02MirahezeMagic] 07paladox edited pull request 03#201: Fix domain for --server in generateSitemap - 13https://git.io/JtuEU [14:52:43] [02MirahezeMagic] 07paladox closed pull request 03#201: Fix domain for --server in generateSitemap - 13https://git.io/JtuEU [14:52:48] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fQRGX [15:02:26] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuuT [15:02:27] [02miraheze/MirahezeMagic] 07paladox 0393d01ff - Revert "Fix domain for --server in generateSitemap" This reverts commit cbb317c74db3dd342b36c9339e58e103c68a373b. [15:02:41] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jtuuk [15:02:42] [02miraheze/mediawiki] 07paladox 03f9f0ccc - Revert "Update MirahezeMagic" This reverts commit c16a5d69ef29c1d6cfc193e22479289cbd67c492. [15:03:31] miraheze/MirahezeMagic - paladox the build passed. [15:21:11] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.45, 3.01, 2.41 [15:23:10] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 1.88, 2.70, 2.38 [15:40:56] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:47:59] [02mediawiki] 07Universal-Omega closed pull request 03#1102: Bump extensions/SocialProfile from `f9d0ce5` to `7eb28e7` - 13https://git.io/JtRUL [15:49:21] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r "[[phab:T6806|Requested]]" /home/reception/metadel.txt [15:49:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:53:31] [02mediawiki] 07Universal-Omega closed pull request 03#1100: Bump extensions/ProtectSite from `00804de` to `5411442` - 13https://git.io/JtRUJ [15:53:34] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/ProtectSite-5411442 [15:53:35] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/ProtectSite-5411442 - 13https://git.io/vbL5b [15:54:30] [02mediawiki] 07Universal-Omega closed pull request 03#1136: Bump extensions/BlogPage from `58f8c4b` to `7142d2f` - 13https://git.io/JtRTZ [15:55:20] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-7eb28e7 [15:55:21] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-7eb28e7 - 13https://git.io/vbL5b [15:55:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jtu2l [15:55:33] [02miraheze/mediawiki] 07dependabot[bot] 03be58586 - Bump extensions/SocialProfile from `f9d0ce5` to `7eb28e7` (#1102) Bumps [extensions/SocialProfile](https://github.com/wikimedia/mediawiki-extensions-SocialProfile) from `f9d0ce5` to `7eb28e7`. - [Release notes](https://github.com/wikimedia/mediawiki-extensions-SocialProfile/releases) - [Commits](https://github.com/wikimedia/mediawiki-extensions- [15:55:33] SocialProfile/compare/f9d0ce520ec0d615d8f378f68296daf136c3248b...7eb28e7734c791c11d6af14d20ed82ef5d16ab7b) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [15:55:34] [ GitHub - wikimedia/mediawiki-extensions-SocialProfile: Github mirror of MediaWiki extension SocialProfile - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_a ] - github.com [15:55:34] [ Releases · wikimedia/mediawiki-extensions-SocialProfile · GitHub ] - github.com [15:55:34] [ Page not found · GitHub · GitHub ] - github.com [15:57:29] [02mediawiki] 07Universal-Omega closed pull request 03#1132: Bump extensions/VoteNY from `1accba0` to `f86db23` - 13https://git.io/JtRTL [15:57:30] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jtu2u [15:57:32] [02miraheze/mediawiki] 07dependabot[bot] 035dbfa44 - Bump extensions/VoteNY from `1accba0` to `f86db23` (#1132) Bumps [extensions/VoteNY](https://github.com/wikimedia/mediawiki-extensions-VoteNY) from `1accba0` to `f86db23`. - [Release notes](https://github.com/wikimedia/mediawiki-extensions-VoteNY/releases) - [Commits](https://github.com/wikimedia/mediawiki-extensions- [15:57:32] VoteNY/compare/1accba0ee312fb337ebc89383647814724082583...f86db230661493e63d939aa8f3a9ab8fc72ffa5b) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [15:57:32] [ GitHub - wikimedia/mediawiki-extensions-VoteNY: Github mirror of MediaWiki extension VoteNY - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for cont ] - github.com [15:57:33] [ Releases · wikimedia/mediawiki-extensions-VoteNY · GitHub ] - github.com [15:57:34] [ Page not found · GitHub · GitHub ] - github.com [16:00:29] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jtu21 [16:00:30] [02miraheze/mediawiki] 07dependabot[bot] 03fec01f4 - Bump extensions/ProtectSite from `00804de` to `5411442` (#1100) Bumps [extensions/ProtectSite](https://github.com/wikimedia/mediawiki-extensions-ProtectSite) from `00804de` to `5411442`. - [Release notes](https://github.com/wikimedia/mediawiki-extensions-ProtectSite/releases) - [Commits](https://github.com/wikimedia/mediawiki-extensions- [16:00:30] ProtectSite/compare/00804decbad71ab8284fc9c55905fa2277c326f9...54114423d524a8381aa6a7cebe9f4a9654425e8b) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [16:00:31] [ GitHub - wikimedia/mediawiki-extensions-ProtectSite: Github mirror of MediaWiki extension ProtectSite - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_acces ] - github.com [16:00:31] [ Releases · wikimedia/mediawiki-extensions-ProtectSite · GitHub ] - github.com [16:00:32] [ Page not found · GitHub · GitHub ] - github.com [16:00:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jtu2M [16:00:39] [02miraheze/mediawiki] 07dependabot[bot] 032917c72 - Bump extensions/BlogPage from `58f8c4b` to `7142d2f` (#1136) Bumps [extensions/BlogPage](https://github.com/wikimedia/mediawiki-extensions-BlogPage) from `58f8c4b` to `7142d2f`. - [Release notes](https://github.com/wikimedia/mediawiki-extensions-BlogPage/releases) - [Commits](https://github.com/wikimedia/mediawiki-extensions- [16:00:39] BlogPage/compare/58f8c4bf10d621036caba6cb8a9784ff87b1b957...7142d2f4fa3370e7aaddb6f1fd1194104e0d6fe0) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [16:00:40] [ GitHub - wikimedia/mediawiki-extensions-BlogPage: Github mirror of MediaWiki extension BlogPage - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for ] - github.com [16:00:40] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/BlogPage-7142d2f [16:00:41] [ Releases · wikimedia/mediawiki-extensions-BlogPage · GitHub ] - github.com [16:00:41] [ Page not found · GitHub · GitHub ] - github.com [16:00:42] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/BlogPage-7142d2f - 13https://git.io/vbL5b [16:03:58] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/VoteNY-f86db23 [16:03:59] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/VoteNY-f86db23 - 13https://git.io/vbL5b [16:08:06] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:08:36] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:08:44] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:08:53] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:08:57] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.66, 21.79, 18.49 [16:09:26] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:09:32] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:10:36] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 33.60, 27.82, 21.90 [16:10:57] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.71, 21.08, 18.66 [16:11:06] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:11:11] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:11:21] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:11:32] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:12:41] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:12:48] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:14:32] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 17.94, 23.11, 21.39 [16:16:55] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.26, 19.50, 18.80 [16:20:29] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 20.24, 19.70, 20.11 [16:26:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtuVY [16:26:30] [02miraheze/puppet] 07paladox 03368b153 - mediawiki: Redirect /sitemaps/ to static.miraheze.org [16:26:32] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:26:33] [02puppet] 07paladox opened pull request 03#1623: mediawiki: Redirect /sitemaps/ to static.miraheze.org - 13https://git.io/JtuVO [16:29:14] [02puppet] 07paladox closed pull request 03#1623: mediawiki: Redirect /sitemaps/ to static.miraheze.org - 13https://git.io/JtuVO [16:29:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuV8 [16:29:17] [02miraheze/puppet] 07paladox 0371205b2 - mediawiki: Redirect /sitemaps/ to static.miraheze.org (#1623) [16:29:19] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:29:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [16:32:36] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JtuVV [16:32:38] [02miraheze/MirahezeMagic] 07paladox 03e3d79d4 - GenerateMirahezeSitemap: Fix wgServer and also fix [16:32:39] [02MirahezeMagic] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fQRGX [16:32:41] [02MirahezeMagic] 07paladox opened pull request 03#202: GenerateMirahezeSitemap: Fix wgServer and also fix - 13https://git.io/JtuVw [16:33:19] [02MirahezeMagic] 07paladox edited pull request 03#202: GenerateMirahezeSitemap: Fix wgServer and also fix url path - 13https://git.io/JtuVw [16:33:24] [02MirahezeMagic] 07paladox closed pull request 03#202: GenerateMirahezeSitemap: Fix wgServer and also fix url path - 13https://git.io/JtuVw [16:34:17] miraheze/MirahezeMagic - paladox the build passed. [16:34:45] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±5] 13https://git.io/JtuVM [16:34:47] [02miraheze/mediawiki] 07paladox 0373c4cfe - Update MirahezeMagic [16:34:48] [02miraheze/mediawiki] 07paladox 03efff689 - Merge branch 'REL1_35' of github.com:miraheze/mediawiki into REL1_35 [16:35:18] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.31, 19.79, 19.26 [16:36:15] [02miraheze/MirahezeMagic] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JtuV9 [16:36:17] [02miraheze/MirahezeMagic] 07paladox 03e3d79d4 - GenerateMirahezeSitemap: Fix wgServer and also fix [16:36:18] [02miraheze/MirahezeMagic] 07paladox 03744ef90 - Merge pull request #202 from miraheze/paladox-patch-3 GenerateMirahezeSitemap: Fix wgServer and also fix url path [16:36:20] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 [16:36:21] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fQRGX [16:37:17] miraheze/MirahezeMagic - paladox the build passed. [16:42:17] PROBLEM - data.maantietaja.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.maantietaja.org reverse DNS resolves to eu10.tmd.cloud [16:43:10] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.70, 22.13, 20.39 [16:43:42] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:44:39] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:44:45] !log /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/generateMirahezeSitemap.php - jobrunner1 [16:44:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:08] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.92, 22.13, 20.58 [16:47:19] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:52:34] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.69, 21.99, 19.50 [16:53:02] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:53:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.46, 19.38, 20.35 [16:54:31] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.41, 21.07, 19.49 [16:57:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuwN [16:57:15] [02miraheze/puppet] 07paladox 033bfe27d - varnish: Don't cache sitemap [16:57:52] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 9.28, 9.00, 7.80 [17:00:13] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.49, 22.90, 21.40 [17:01:17] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [17:01:40] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 8.62, 8.50, 7.59 [17:01:46] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 7.01, 8.35, 7.80 [17:02:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.61, 23.40, 21.78 [17:02:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.15, 19.47, 19.51 [17:03:14] !log reboot cp6 as varnish was failing. Logs showed out of space? [17:03:50] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [17:04:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:16] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 30.25, 25.14, 22.56 [17:04:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 2607:5300:205:200::1c30/cpweb [17:04:40] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.45, 9.27, 7.49 [17:04:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 51.222.25.132/cpweb [17:05:37] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:05:44] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 7.09, 8.32, 7.80 [17:05:59] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:05:59] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:06:05] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:06:33] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:06:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:06:38] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 8.04, 8.77, 7.52 [17:06:40] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:07:40] PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hr.petrawiki.org could not be found [17:07:40] PROBLEM - wiki.fourta.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fourta.org could not be found [17:07:41] PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.com could not be found [17:07:45] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc-multiverse.dcwikis.com could not be found [17:07:46] PROBLEM - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teessidehackspace.org.uk could not be found [17:07:46] PROBLEM - wiki.velaan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.velaan.org could not be found [17:07:48] PROBLEM - wiki.azimuth.navy - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.azimuth.navy could not be found [17:07:48] PROBLEM - wikipariksha.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikipariksha.com could not be found [17:07:48] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:07:50] PROBLEM - rangpurpedia.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rangpurpedia.xyz could not be found [17:08:15] PROBLEM - allthetropes.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.orain.org could not be found [17:08:23] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.06, 23.64, 22.62 [17:08:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:08:45] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.65, 9.01, 7.77 [17:08:48] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 6.77, 8.57, 8.18 [17:08:49] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [17:09:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jturz [17:09:39] [02miraheze/puppet] 07paladox 036673604 - Fix [17:10:20] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:10:46] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.41, 7.72, 7.93 [17:12:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:1056::2/cpweb [17:12:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::1c30/cpweb [17:12:44] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.07, 9.94, 8.44 [17:12:46] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.84, 8.27, 7.77 [17:13:45] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [17:14:40] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: rDNS OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp6.miraheze.org [17:14:42] RECOVERY - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is OK: rDNS OK - wiki.teessidehackspace.org.uk reverse DNS resolves to cp6.miraheze.org [17:14:42] RECOVERY - wiki.velaan.org - reverse DNS on sslhost is OK: rDNS OK - wiki.velaan.org reverse DNS resolves to cp6.miraheze.org [17:14:44] RECOVERY - wiki.azimuth.navy - reverse DNS on sslhost is OK: rDNS OK - wiki.azimuth.navy reverse DNS resolves to cp6.miraheze.org [17:14:45] RECOVERY - wikipariksha.com - reverse DNS on sslhost is OK: rDNS OK - wikipariksha.com reverse DNS resolves to cp12.miraheze.org [17:14:46] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 8.61, 8.82, 8.26 [17:14:47] RECOVERY - rangpurpedia.xyz - reverse DNS on sslhost is OK: rDNS OK - rangpurpedia.xyz reverse DNS resolves to cp12.miraheze.org [17:14:48] RECOVERY - hr.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - hr.petrawiki.org reverse DNS resolves to cp6.miraheze.org [17:14:48] RECOVERY - miraheze.com - reverse DNS on sslhost is OK: rDNS OK - miraheze.com reverse DNS resolves to cp6.miraheze.org [17:14:48] RECOVERY - wiki.fourta.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fourta.org reverse DNS resolves to cp6.miraheze.org [17:14:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.58, 9.15, 8.33 [17:15:08] RECOVERY - allthetropes.orain.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.orain.org reverse DNS resolves to cp7.miraheze.org [17:15:14] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [17:16:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jturh [17:16:11] [02miraheze/puppet] 07paladox 03a2dce36 - varnish: Do not cache sitemaps [17:16:23] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.78, 5.75, 5.02 [17:16:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:16:46] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 11.36, 9.71, 8.65 [17:16:51] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 9.10, 9.49, 8.71 [17:17:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuoU [17:17:16] [02miraheze/puppet] 07paladox 031541109 - Fix [17:18:35] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.85, 23.25, 22.64 [17:18:44] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.27, 9.37, 8.64 [17:18:49] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.61, 9.62, 8.69 [17:18:58] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 11.92, 9.60, 8.44 [17:19:23] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:19:48] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:19:52] PROBLEM - wikibeats.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikibeats.net could not be found [17:19:57] PROBLEM - wiki.park-city.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.park-city.club could not be found [17:20:04] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [17:20:05] PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found [17:20:39] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.32, 22.26, 22.36 [17:20:47] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 13.58, 10.62, 9.25 [17:20:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:20:58] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 8.61, 9.33, 8.50 [17:22:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuoW [17:22:37] [02miraheze/puppet] 07paladox 03e15d2ab - Update default.vcl [17:22:46] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.58, 9.32, 8.80 [17:24:18] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [17:24:28] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.06, 5.07, 4.96 [17:24:42] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.75, 9.63, 8.96 [17:26:40] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.80, 9.87, 9.43 [17:26:42] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.94, 9.35, 8.94 [17:26:42] RECOVERY - wikibeats.net - reverse DNS on sslhost is OK: rDNS OK - wikibeats.net reverse DNS resolves to cp7.miraheze.org [17:26:49] RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: rDNS OK - monarchists.wiki reverse DNS resolves to cp7.miraheze.org [17:26:51] RECOVERY - wiki.park-city.club - reverse DNS on sslhost is OK: rDNS OK - wiki.park-city.club reverse DNS resolves to cp7.miraheze.org [17:27:17] PROBLEM - www.baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.baharna.org could not be found [17:27:18] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [17:27:18] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.nonbinary.wiki could not be found [17:27:21] PROBLEM - marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marinebiodiversitymatrix.org could not be found [17:27:22] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mineland.eu could not be found [17:27:23] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [17:27:25] PROBLEM - en.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.nocyclo.tk could not be found [17:28:10] PROBLEM - wiki.teamrelectric.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teamrelectric.ca could not be found [17:28:46] PROBLEM - www.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pornwiki.org could not be found [17:28:46] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [17:28:49] PROBLEM - iesd.mobagenie.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iesd.mobagenie.my.id could not be found [17:28:50] PROBLEM - arquivo.ucmg.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arquivo.ucmg.ml could not be found [17:28:53] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [17:28:57] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [17:28:58] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [17:28:59] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 7.09, 8.48, 8.48 [17:28:59] PROBLEM - techwiki.techboyg5blog.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for techwiki.techboyg5blog.com could not be found [17:29:02] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [17:29:03] PROBLEM - wiki.candela.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.candela.digital could not be found [17:29:04] PROBLEM - programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for programming.red could not be found [17:29:05] PROBLEM - vortekspedia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vortekspedia.tk could not be found [17:29:47] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:30:27] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.64, 8.13, 8.43 [17:30:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jtuob [17:30:29] [02miraheze/mw-config] 07paladox 03031a8c6 - Switch over wikis beginning with c to new file storage [17:34:00] RECOVERY - www.baharna.org - reverse DNS on sslhost is OK: rDNS OK - www.baharna.org reverse DNS resolves to cp7.miraheze.org [17:34:02] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: rDNS OK - infectowiki.com reverse DNS resolves to cp7.miraheze.org [17:34:03] RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - data.nonbinary.wiki reverse DNS resolves to cp7.miraheze.org [17:34:09] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.mineland.eu reverse DNS resolves to cp7.miraheze.org [17:34:11] RECOVERY - marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - marinebiodiversitymatrix.org reverse DNS resolves to cp6.miraheze.org [17:34:14] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - fr.gyaanipedia.com reverse DNS resolves to cp6.miraheze.org [17:34:17] RECOVERY - en.nocyclo.tk - reverse DNS on sslhost is OK: rDNS OK - en.nocyclo.tk reverse DNS resolves to cp6.miraheze.org [17:34:24] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.31, 8.57, 8.52 [17:34:28] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 11.63, 9.90, 9.44 [17:34:37] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.89, 22.35, 21.69 [17:34:56] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 13.29, 9.73, 8.83 [17:35:08] RECOVERY - wiki.teamrelectric.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.teamrelectric.ca reverse DNS resolves to cp6.miraheze.org [17:35:35] RECOVERY - www.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.pornwiki.org reverse DNS resolves to cp7.miraheze.org [17:35:35] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: rDNS OK - www.zenbuddhism.info reverse DNS resolves to cp7.miraheze.org [17:35:39] RECOVERY - iesd.mobagenie.my.id - reverse DNS on sslhost is OK: rDNS OK - iesd.mobagenie.my.id reverse DNS resolves to cp7.miraheze.org [17:35:41] RECOVERY - arquivo.ucmg.ml - reverse DNS on sslhost is OK: rDNS OK - arquivo.ucmg.ml reverse DNS resolves to cp7.miraheze.org [17:35:42] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp7.miraheze.org [17:35:45] RECOVERY - wiki.candela.digital - reverse DNS on sslhost is OK: rDNS OK - wiki.candela.digital reverse DNS resolves to cp7.miraheze.org [17:35:46] RECOVERY - programming.red - reverse DNS on sslhost is OK: rDNS OK - programming.red reverse DNS resolves to cp7.miraheze.org [17:35:47] RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: rDNS OK - n64brew.dev reverse DNS resolves to cp7.miraheze.org [17:35:49] RECOVERY - vortekspedia.tk - reverse DNS on sslhost is OK: rDNS OK - vortekspedia.tk reverse DNS resolves to cp6.miraheze.org [17:35:52] RECOVERY - www.mcpk.wiki - reverse DNS on sslhost is OK: rDNS OK - www.mcpk.wiki reverse DNS resolves to cp7.miraheze.org [17:35:52] RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: rDNS OK - www.mh142.com reverse DNS resolves to cp7.miraheze.org [17:35:56] RECOVERY - techwiki.techboyg5blog.com - reverse DNS on sslhost is OK: rDNS OK - techwiki.techboyg5blog.com reverse DNS resolves to cp7.miraheze.org [17:36:32] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.02, 9.84, 9.24 [17:36:34] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.99, 21.76, 21.53 [17:36:53] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 8.65, 9.49, 8.87 [17:37:33] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JtuKW [17:37:34] [02miraheze/MirahezeMagic] 07paladox 03b95c4ad - Use UploadDirectory within generateMirahezeSitemap [17:37:36] [02MirahezeMagic] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fQRGX [17:37:37] [02MirahezeMagic] 07paladox opened pull request 03#203: Use UploadDirectory within generateMirahezeSitemap - 13https://git.io/JtuKl [17:38:34] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.85, 9.45, 9.18 [17:38:47] miraheze/MirahezeMagic - paladox the build passed. [17:39:31] [02MirahezeMagic] 07paladox synchronize pull request 03#203: Use UploadDirectory within generateMirahezeSitemap - 13https://git.io/JtuKl [17:39:32] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JtuKz [17:39:34] [02miraheze/MirahezeMagic] 07paladox 03096c648 - Update generateMirahezeSitemap.php [17:39:56] [02MirahezeMagic] 07paladox closed pull request 03#203: Use UploadDirectory within generateMirahezeSitemap - 13https://git.io/JtuKl [17:39:58] [02miraheze/MirahezeMagic] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/JtuK2 [17:39:59] [02miraheze/MirahezeMagic] 07paladox 0350f5be9 - Merge pull request #203 from miraheze/paladox-patch-3 Use UploadDirectory within generateMirahezeSitemap [17:40:01] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fQRGX [17:40:02] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 [17:40:14] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.30, 9.01, 9.27 [17:40:19] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 7.16, 8.10, 8.37 [17:40:32] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtuKw [17:40:33] [02miraheze/mediawiki] 07paladox 0350a7228 - Update MirahezeMagic [17:40:42] miraheze/MirahezeMagic - paladox the build passed. [17:40:57] miraheze/MirahezeMagic - paladox the build passed. [17:42:09] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 11.48, 9.83, 9.53 [17:42:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuKo [17:42:50] [02miraheze/mw-config] 07paladox 032420ac6 - Switch over wikis beginning with f to new file storage [17:44:11] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 8.80, 9.47, 9.44 [17:44:18] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.50, 4.71, 4.43 [17:44:25] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.70, 9.02, 8.68 [17:44:36] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.25, 8.81, 8.90 [17:44:48] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 7.00, 7.78, 8.36 [17:45:51] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.43, 5.11, 4.60 [17:46:20] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 7.52, 8.44, 8.50 [17:46:29] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 18.48, 19.35, 20.34 [17:46:34] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.04, 7.80, 8.53 [17:47:50] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.67, 5.30, 4.71 [17:48:32] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 8.08, 7.88, 8.46 [17:49:57] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.87, 5.62, 4.92 [17:50:17] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.53, 5.71, 4.96 [17:50:29] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.94, 21.03, 20.88 [17:52:16] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.65, 5.71, 5.05 [17:53:03] I just got a 502 Bad Gateway Error when attempting to move a file [17:53:06] !sre [17:53:56] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 7.67, 6.16, 5.25 [17:54:25] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.30, 22.45, 21.42 [17:55:32] blackwidowmovie0: hi. When you report bugs/errors please make sure to provide a link or tell us the location of where it happened [17:55:45] otherwise we can't investigate and/or reproduce a vague error [17:55:51] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 6.62, 7.56, 8.42 [17:56:16] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.39, 5.93, 5.28 [17:56:24] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.97, 23.26, 21.89 [17:57:52] Reception123: Sorry. Tried to move `File:Darcy Lewis - WandaVision.jpg` to `File:Darcy Lewis (Portal).jpg` on `wandavisionwiki` and got a `502 Bad Gateway` error [17:58:12] blackwidowmovie0: okay, could you please try again and see if it works this time? [17:58:13] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.32, 5.63, 5.24 [17:58:17] it could just have been a temporary timeout [17:58:21] Reception123: Trying [17:58:22] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.91, 23.67, 22.18 [17:58:33] Reception123: Done [17:58:38] it worked [17:58:47] ok, great [17:58:57] if you get errors like that it helps to just wait a bit and try again :) [17:59:22] Ok, thanks! [18:00:25] Reception123: Has the wiki deletion script been run? [18:00:43] if so, can `mcuwiki` be re-created [18:00:54] it has not been run yet [18:02:20] very well [18:06:08] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 3.92, 4.80, 5.02 [18:06:56] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:12:50] PROBLEM - permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for permanentfuturelab.wiki could not be found [18:12:55] PROBLEM - celeste.ink - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for celeste.ink could not be found [18:12:55] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [18:12:56] PROBLEM - intp.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for intp.miraheze.org could not be found [18:12:57] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [18:12:59] PROBLEM - wiki.funkey-project.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.funkey-project.com could not be found [18:13:18] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found [18:13:19] PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tulpa.info could not be found [18:13:20] PROBLEM - piwik.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for piwik.miraheze.org could not be found [18:14:18] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.46, 8.47, 8.05 [18:14:38] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 9.87, 7.99, 7.83 [18:14:43] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:16:15] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 7.39, 8.00, 7.94 [18:16:36] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.01, 7.29, 7.60 [18:19:33] RECOVERY - permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - permanentfuturelab.wiki reverse DNS resolves to cp7.miraheze.org [18:19:48] RECOVERY - celeste.ink - reverse DNS on sslhost is OK: rDNS OK - celeste.ink reverse DNS resolves to cp7.miraheze.org [18:19:49] RECOVERY - wiki.funkey-project.com - reverse DNS on sslhost is OK: rDNS OK - wiki.funkey-project.com reverse DNS resolves to cp7.miraheze.org [18:19:49] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - miraheze.wiki reverse DNS resolves to cp6.miraheze.org [18:19:59] RECOVERY - intp.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - intp.miraheze.org reverse DNS resolves to cp6.miraheze.org [18:20:01] RECOVERY - commons.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - commons.gyaanipedia.com reverse DNS resolves to cp7.miraheze.org [18:20:18] RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: rDNS OK - wiki.tulpa.info reverse DNS resolves to cp7.miraheze.org [18:20:18] RECOVERY - piwik.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - piwik.miraheze.org reverse DNS resolves to cp7.miraheze.org [18:20:21] RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - pt.graalmilitary.com reverse DNS resolves to cp6.miraheze.org [18:22:18] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:26:35] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.91, 8.27, 7.63 [18:27:06] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.34, 8.57, 8.28 [18:28:30] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.54, 7.50, 7.42 [18:31:00] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 7.31, 8.08, 8.16 [18:32:59] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 6.42, 5.61, 5.04 [18:35:01] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 11.03, 9.59, 8.76 [18:35:01] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.21, 5.22, 4.95 [18:35:29] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 9.29, 8.70, 8.04 [18:36:57] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.63, 9.13, 8.68 [18:36:59] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.73, 5.07, 4.93 [18:37:28] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.70, 7.35, 7.62 [18:38:55] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 7.35, 8.41, 8.47 [18:50:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.16, 22.07, 23.70 [18:50:39] .deop [18:52:12] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.81, 23.43, 24.01 [18:53:59] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 3.98, 5.24, 5.83 [18:56:11] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.93, 22.01, 23.40 [18:59:59] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 7.00, 6.04, 5.96 [19:02:01] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 4.22, 5.41, 5.75 [19:04:10] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.70, 22.91, 23.24 [19:05:06] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 9.09, 7.86, 7.14 [19:05:13] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 9.40, 8.50, 7.81 [19:07:03] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 8.08, 7.63, 7.13 [19:07:09] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 8.00, 8.30, 7.83 [19:08:15] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 17.51, 21.90, 22.96 [19:10:06] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.77, 5.82, 5.71 [19:12:05] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.24, 5.38, 5.55 [19:16:05] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.76, 5.61, 5.55 [19:18:04] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 4.16, 5.26, 5.45 [19:20:05] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.02, 5.58, 5.55 [19:20:14] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.23, 23.08, 22.64 [19:22:04] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 3.26, 4.87, 5.30 [19:22:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.36, 23.60, 22.92 [19:24:03] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.19, 5.41, 5.45 [19:24:56] * hispano76 greet [19:28:04] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.21, 5.71, 5.59 [19:32:08] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.08, 5.71, 5.59 [19:32:16] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.64, 20.98, 21.41 [19:34:22] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.23, 8.35, 7.32 [19:36:07] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.59, 5.69, 5.62 [19:36:12] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.79, 22.31, 21.94 [19:36:20] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.34, 7.64, 7.19 [19:38:16] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.04, 22.62, 22.07 [19:44:15] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 19.61, 22.62, 22.45 [19:46:17] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.53, 5.52, 5.49 [19:48:12] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.36, 24.40, 23.08 [19:48:13] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 7.23, 4.92, 4.04 [19:48:20] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.95, 5.40, 5.44 [19:50:10] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.10, 23.95, 23.09 [19:50:11] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.55, 5.29, 4.29 [19:51:11] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.52, 2.93, 2.57 [19:52:08] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.69, 5.10, 4.35 [19:53:14] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 2.37, 2.72, 2.54 [19:58:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.97, 17.99, 20.36 [19:59:33] * blackwidowmovie0 wants to ignore icinga [20:00:21] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.20, 5.34, 5.28 [20:02:11] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.85, 20.01, 20.76 [20:02:26] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.17, 5.33, 5.30 [20:03:45] @blackwidowmovie0 You can. Just click on @icinga-miraheze and click "Ignore..." [20:06:10] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.42, 19.18, 20.40 [20:10:11] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.49, 20.97, 20.91 [20:14:11] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.76, 18.65, 19.99 [20:20:33] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 4.26, 4.57, 4.99 [20:23:46] dmehus: Ah, thanks [20:34:10] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.52, 18.70, 17.80 [20:36:12] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.19, 17.85, 17.61 [20:50:14] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.81, 3.27, 2.65 [20:50:24] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 11.16, 8.60, 6.57 [20:50:36] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.51, 6.93, 5.24 [20:52:12] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 3.03, 3.07, 2.65 [20:52:34] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.58, 5.86, 5.05 [20:53:28] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 12.84, 9.77, 6.91 [20:54:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::1c30/cpweb [20:54:43] PROBLEM - baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for baharna.org could not be found [20:54:44] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [20:54:45] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [20:54:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [20:55:05] @blackwidowmovie0 ok, np [20:56:08] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [20:56:09] PROBLEM - wiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.miraheze.org could not be found [20:56:19] PROBLEM - es.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.nocyclo.tk could not be found [20:56:56] PROBLEM - wiki.park-city.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.park-city.club could not be found [20:57:01] PROBLEM - opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for opendatascot.org could not be found [20:57:04] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.cineastas.pt could not be found [20:57:05] PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found [20:57:13] PROBLEM - www.allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.allthetropes.org could not be found [20:57:22] PROBLEM - www.baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.baharna.org could not be found [20:57:25] PROBLEM - marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marinebiodiversitymatrix.org could not be found [20:57:28] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [20:57:43] PROBLEM - dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dariawiki.org could not be found [20:57:43] PROBLEM - debilepedie.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for debilepedie.gq could not be found [20:57:44] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.autocountsoft.com could not be found [20:57:45] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found [20:57:45] PROBLEM - wiki.make717.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.make717.org could not be found [20:57:48] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [20:58:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:58:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:58:55] PROBLEM - wiki.fourta.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fourta.org could not be found [20:58:56] PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.com could not be found [20:58:58] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [20:58:59] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [20:59:02] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [20:59:03] PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found [20:59:03] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [20:59:03] PROBLEM - wiki.candela.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.candela.digital could not be found [20:59:04] PROBLEM - programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for programming.red could not be found [20:59:06] PROBLEM - vortekspedia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vortekspedia.tk could not be found [21:00:22] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.22, 9.82, 8.54 [21:01:25] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp7.miraheze.org [21:01:26] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp7.miraheze.org [21:01:44] RECOVERY - baharna.org - reverse DNS on sslhost is OK: rDNS OK - baharna.org reverse DNS resolves to cp7.miraheze.org [21:02:20] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.46, 8.48, 8.21 [21:02:48] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: rDNS OK - wiki.nowchess.org reverse DNS resolves to cp7.miraheze.org [21:02:55] RECOVERY - wiki.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - wiki.miraheze.org reverse DNS resolves to cp6.miraheze.org [21:03:09] RECOVERY - es.nocyclo.tk - reverse DNS on sslhost is OK: rDNS OK - es.nocyclo.tk reverse DNS resolves to cp6.miraheze.org [21:03:20] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.61, 8.41, 7.97 [21:03:45] RECOVERY - opendatascot.org - reverse DNS on sslhost is OK: rDNS OK - opendatascot.org reverse DNS resolves to cp6.miraheze.org [21:03:48] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: rDNS OK - guia.cineastas.pt reverse DNS resolves to cp7.miraheze.org [21:03:49] RECOVERY - mrlove.wiki - reverse DNS on sslhost is OK: rDNS OK - mrlove.wiki reverse DNS resolves to cp7.miraheze.org [21:03:50] RECOVERY - wiki.park-city.club - reverse DNS on sslhost is OK: rDNS OK - wiki.park-city.club reverse DNS resolves to cp7.miraheze.org [21:04:05] RECOVERY - www.baharna.org - reverse DNS on sslhost is OK: rDNS OK - www.baharna.org reverse DNS resolves to cp7.miraheze.org [21:04:06] RECOVERY - www.allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - www.allthetropes.org reverse DNS resolves to cp6.miraheze.org [21:04:13] RECOVERY - marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - marinebiodiversitymatrix.org reverse DNS resolves to cp6.miraheze.org [21:04:17] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - fr.gyaanipedia.com reverse DNS resolves to cp6.miraheze.org [21:04:23] RECOVERY - dariawiki.org - reverse DNS on sslhost is OK: rDNS OK - dariawiki.org reverse DNS resolves to cp6.miraheze.org [21:04:23] RECOVERY - debilepedie.gq - reverse DNS on sslhost is OK: rDNS OK - debilepedie.gq reverse DNS resolves to cp6.miraheze.org [21:04:28] RECOVERY - wiki.make717.org - reverse DNS on sslhost is OK: rDNS OK - wiki.make717.org reverse DNS resolves to cp7.miraheze.org [21:04:29] RECOVERY - wiki.autocountsoft.com - reverse DNS on sslhost is OK: rDNS OK - wiki.autocountsoft.com reverse DNS resolves to cp7.miraheze.org [21:04:31] RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp6.miraheze.org [21:04:34] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp6.miraheze.org [21:05:42] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp6.miraheze.org [21:05:42] RECOVERY - miraheze.tk - reverse DNS on sslhost is OK: rDNS OK - miraheze.tk reverse DNS resolves to cp6.miraheze.org [21:05:43] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp7.miraheze.org [21:05:44] RECOVERY - wiki.candela.digital - reverse DNS on sslhost is OK: rDNS OK - wiki.candela.digital reverse DNS resolves to cp7.miraheze.org [21:05:45] RECOVERY - programming.red - reverse DNS on sslhost is OK: rDNS OK - programming.red reverse DNS resolves to cp6.miraheze.org [21:05:49] RECOVERY - vortekspedia.tk - reverse DNS on sslhost is OK: rDNS OK - vortekspedia.tk reverse DNS resolves to cp6.miraheze.org [21:05:50] RECOVERY - wiki.fourta.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fourta.org reverse DNS resolves to cp7.miraheze.org [21:05:53] RECOVERY - miraheze.com - reverse DNS on sslhost is OK: rDNS OK - miraheze.com reverse DNS resolves to cp7.miraheze.org [21:05:55] RECOVERY - www.mcpk.wiki - reverse DNS on sslhost is OK: rDNS OK - www.mcpk.wiki reverse DNS resolves to cp7.miraheze.org [21:05:55] RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: rDNS OK - www.mh142.com reverse DNS resolves to cp7.miraheze.org [21:17:31] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 12.42, 8.71, 6.84 [21:18:37] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 12.11, 10.01, 8.32 [21:19:27] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.62, 7.62, 6.68 [21:19:37] PROBLEM - rupowerrangersfanon.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rupowerrangersfanon.tk could not be found [21:19:42] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found [21:20:35] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.56, 8.30, 7.89 [21:22:33] PROBLEM - gluster1 APT on gluster1 is UNKNOWN: APT UNKNOWN: 0 packages available for upgrade (0 critical updates). errors detected. [21:24:32] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [21:26:30] RECOVERY - rupowerrangersfanon.tk - reverse DNS on sslhost is OK: rDNS OK - rupowerrangersfanon.tk reverse DNS resolves to cp6.miraheze.org [21:26:36] RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: rDNS OK - adadevelopersacademy.wiki reverse DNS resolves to cp6.miraheze.org [21:26:52] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:40:17] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.77, 5.06, 3.45 [21:41:43] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.75, 20.18, 18.70 [21:41:49] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.02, 5.06, 4.34 [21:42:16] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.34, 5.49, 3.81 [21:43:41] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 18.24, 19.67, 18.70 [21:43:52] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 4.24, 4.67, 4.28 [21:48:27] PROBLEM - gluster1 APT on gluster1 is UNKNOWN: APT UNKNOWN: 0 packages available for upgrade (0 critical updates). errors detected. [21:51:32] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.02, 20.18, 19.07 [21:53:21] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 12.81, 8.58, 6.44 [21:53:33] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 18.77, 19.73, 19.05 [21:54:20] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 12.98, 8.51, 6.32 [21:56:32] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [21:57:19] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 6.47, 9.11, 7.29 [21:58:17] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.27, 5.55, 5.27 [21:58:21] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 7.83, 8.47, 6.87 [21:58:52] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.12, 4.94, 4.49 [22:00:39] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 11.08, 8.48, 6.90 [22:00:58] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.38, 5.45, 4.73 [22:01:09] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 12.56, 9.42, 7.26 [22:02:31] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [22:02:36] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 4.61, 5.00, 5.10 [22:03:08] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:03:11] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:12] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 5.44, 5.68, 4.93 [22:03:26] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:03:34] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 10.30, 9.79, 8.15 [22:03:47] PROBLEM - cp12 Stunnel Http for mw4 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:03:50] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:03:55] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 28.08, 23.38, 20.73 [22:03:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb [22:04:03] PROBLEM - jobrunner2 Current Load on jobrunner2 is WARNING: WARNING - load average: 5.52, 4.94, 3.84 [22:04:16] PROBLEM - cp12 Stunnel Http for mw4 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:04:50] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.33, 9.90, 8.08 [22:05:03] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.867 second response time [22:05:07] PROBLEM - files.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.petrawiki.org could not be found [22:05:09] PROBLEM - wiki.mastodon.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mastodon.kr could not be found [22:05:10] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 21795 bytes in 0.876 second response time [22:05:10] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [22:05:11] PROBLEM - pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pornwiki.org could not be found [22:05:12] PROBLEM - www.rothwell-leeds.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.rothwell-leeds.co.uk could not be found [22:05:15] PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found [22:05:23] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15139 bytes in 0.007 second response time [22:05:30] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 9.70, 9.69, 8.30 [22:05:33] PROBLEM - reviwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviwiki.info could not be found [22:05:43] RECOVERY - cp12 Stunnel Http for mw4 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 0.381 second response time [22:05:47] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15131 bytes in 1.048 second response time [22:05:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:06:13] RECOVERY - cp12 Stunnel Http for mw4 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15139 bytes in 0.331 second response time [22:06:15] RhinosF1|NotHere: Do you have the list of resources, or are you busy right now? If the latter, totally fine [22:06:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:06:42] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.97, 9.74, 8.12 [22:06:44] blackwidowmovie0: very busy, I came on because I got messages saying stuff we're on fire. [22:06:56] RhinosF1|NotHere: We're no fire? [22:07:11] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 3.30, 4.47, 4.61 [22:07:19] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 5.26, 9.39, 8.23 [22:07:25] blackwidowmovie0: miraheze bots had some urgent issues that needed analysis. [22:07:28] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 12.76, 10.83, 8.88 [22:07:37] RhinosF1|NotHere: Seems interesting [22:07:43] if you need help, just DM [22:07:55] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 19.06, 22.22, 20.99 [22:08:19] blackwidowmovie0: no [22:08:22] ok [22:08:40] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.30, 7.74, 7.57 [22:09:16] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.12, 7.76, 7.76 [22:09:50] RECOVERY - jobrunner2 Current Load on jobrunner2 is OK: OK - load average: 4.20, 4.91, 4.22 [22:09:52] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.86, 20.16, 20.37 [22:10:43] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.95, 7.68, 7.77 [22:11:54] RECOVERY - files.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - files.petrawiki.org reverse DNS resolves to cp6.miraheze.org [22:11:56] RECOVERY - wiki.mastodon.kr - reverse DNS on sslhost is OK: rDNS OK - wiki.mastodon.kr reverse DNS resolves to cp7.miraheze.org [22:11:58] RECOVERY - pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - pornwiki.org reverse DNS resolves to cp7.miraheze.org [22:12:02] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikiru.wildterra2.com reverse DNS resolves to cp7.miraheze.org [22:12:05] RECOVERY - www.rothwell-leeds.co.uk - reverse DNS on sslhost is OK: rDNS OK - www.rothwell-leeds.co.uk reverse DNS resolves to cp7.miraheze.org [22:12:11] RECOVERY - orain.org - reverse DNS on sslhost is OK: rDNS OK - orain.org reverse DNS resolves to cp6.miraheze.org [22:12:28] RECOVERY - reviwiki.info - reverse DNS on sslhost is OK: rDNS OK - reviwiki.info reverse DNS resolves to cp6.miraheze.org [22:15:19] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.32, 9.16, 9.18 [22:19:19] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.48, 6.85, 8.21 [22:21:15] PROBLEM - gluster2 Current Load on gluster2 is CRITICAL: CRITICAL - load average: 6.56, 5.70, 4.91 [22:21:45] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.89, 23.05, 20.87 [22:22:41] PROBLEM - gluster1 APT on gluster1 is UNKNOWN: APT UNKNOWN: 0 packages available for upgrade (0 critical updates). errors detected. [22:23:15] PROBLEM - gluster2 Current Load on gluster2 is WARNING: WARNING - load average: 4.64, 5.28, 4.85 [22:23:42] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 19.74, 22.09, 20.80 [22:25:14] RECOVERY - gluster2 Current Load on gluster2 is OK: OK - load average: 3.80, 4.75, 4.71 [22:26:37] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [22:27:40] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 31.86, 23.67, 21.53 [22:29:37] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.54, 23.01, 21.57 [22:33:35] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 32.70, 25.59, 22.77 [22:34:46] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.37, 8.10, 6.91 [22:36:44] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.36, 6.98, 6.64 [22:37:36] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.05, 21.84, 21.98 [22:39:35] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.86, 24.43, 22.92 [22:41:33] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.35, 22.80, 22.55 [22:43:31] Hello cmetz! If you have any questions, feel free to ask and someone should answer soon. [22:44:23] hi cmetz [22:45:00] When I'm logged in and make a change to my Wiki, I can see the change immediately. But if I log out, or someone else who isn't logged in views the page, they see the old, pre-change, page. Changes eventually become visible, so it appears like this is a caching or propagation issue. Is there anything I can do to make changes visible to the general [22:45:00] public sooner? [22:47:33] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.05, 16.75, 19.92 [22:48:04] cmetz: which wiki? [22:48:11] loudounseac [22:48:16] https://loudounseac.miraheze.org/wiki/February_3,_2021 [22:48:17] [ February 3, 2021 - lcpsseac ] - loudounseac.miraheze.org [22:48:20] here's an example page [22:48:27] PROBLEM - gluster1 APT on gluster1 is UNKNOWN: APT UNKNOWN: 0 packages available for upgrade (0 critical updates). errors detected. [22:48:47] If you go to View History, you can see the edit, but the page that renders when you go there as a non-logged-in user does not show the change. [22:49:21] yep, can reproduce [22:50:22] looking [22:51:01] Thanks. It's been doing this for a while now. [22:54:29] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [22:56:52] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:58:39] uhm, no varnish purges anymore? what is this [23:01:28] purges no working SPF|Cloud ? [23:01:37] nope [23:01:40] i mean users reported seeing that on file pages. [23:01:50] but haven't reported it for pages until today [23:03:05] It happened a couple months ago also but I can't remember how it was resolved but the issue sounds exactly like then. [23:04:52] sounds like this problem has been happening a while [23:05:08] is this tracked in phabricator? [23:06:53] SPF|Cloud: Nope. There's been phabricator tasks before but the specific wikis end up resolved so they get closed. [23:07:16] oh boy, this is fun [23:07:42] ufw is denying all traffic to port 81 on cache proxies? [23:09:00] Feb 3 23:06:23 cp6 kernel: [21788.022155] [UFW BLOCK] IN=ens18 (...) SRC=51.77.107.211 DST=51.77.107.210 (...) PROTO=TCP SPT=20232 DPT=81 (...) [23:09:04] really? [23:09:39] https://github.com/miraheze/puppet/blob/master/modules/role/manifests/varnish.pp [23:09:39] [ puppet/varnish.pp at master · miraheze/puppet · GitHub ] - github.com [23:09:40] hmm [23:10:01] even more firewall rules that need to be automated? auch [23:10:16] PROBLEM - test2 Current Load on test2 is CRITICAL: CRITICAL - load average: 7.94, 4.43, 1.95 [23:10:22] SPF|Cloud test2 isn't in there [23:10:26] let me automate this... [23:11:21] right, so only test2 is denied [23:11:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtuAE [23:11:52] [02miraheze/puppet] 07paladox 03477b733 - varnish: Automate firewall rule for port 81 [23:11:54] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:11:55] [02puppet] 07paladox opened pull request 03#1624: varnish: Automate firewall rule for port 81 - 13https://git.io/JtuAu [23:12:13] SPF|Cloud ^ [23:12:14] PROBLEM - test2 Current Load on test2 is WARNING: WARNING - load average: 2.06, 3.62, 1.97 [23:12:20] however, I see no traffic to port 81 whatsoever [23:12:37] that doesn't seem correct, since we have enough edits per second [23:12:39] [02puppet] 07paladox synchronize pull request 03#1624: varnish: Automate firewall rule for port 81 - 13https://git.io/JtuAu [23:12:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtuA2 [23:12:42] [02miraheze/puppet] 07paladox 031b11c12 - Update varnish.pp [23:12:51] +1 [23:12:54] [02puppet] 07paladox closed pull request 03#1624: varnish: Automate firewall rule for port 81 - 13https://git.io/JtuAu [23:12:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuAV [23:12:57] [02miraheze/puppet] 07paladox 03b164075 - varnish: Automate firewall rule for port 81 (#1624) * varnish: Automate firewall rule for port 81 * Update varnish.pp [23:12:59] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:13:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:14:16] RECOVERY - test2 Current Load on test2 is OK: OK - load average: 0.53, 2.50, 1.76 [23:14:31] puppet ran [23:14:39] should be able to access the port now [23:14:53] right [23:15:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuAX [23:15:36] [02miraheze/puppet] 07paladox 03733b6b6 - cp3: Switch to puppet3 [23:15:36] cp6 isn't receiving varnish purges [23:16:16] yep, I can even reproduce on https://meta.miraheze.org/wiki/User:Southparkfan/Test [23:16:17] [ User:Southparkfan/Test - Miraheze Meta ] - meta.miraheze.org [23:16:43] looks like there are zero purges [23:17:11] isn't there a job run on the jobrunner for purges? [23:18:13] could be... htmlCacheUpdate [23:18:35] [02dns] 07RhinosF1 opened pull request 03#186: miraheze.wiki: failover dns for [bots|tools|db]1 - 13https://git.io/JtuAF [23:18:44] paladox: can you ^ [23:19:07] [02dns] 07paladox closed pull request 03#186: miraheze.wiki: failover dns for [bots|tools|db]1 - 13https://git.io/JtuAF [23:19:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuAN [23:19:10] [02miraheze/dns] 07RhinosF1 038f9e193 - miraheze.wiki: failover dns for [bots|tools|db]1 (#186) [23:19:49] Notice: /Stage[main]/Dns/Exec[gdnsd-syntax]/returns: # error: CNAME 'tools1.miraheze.wiki.' points to known same-zone NXDOMAIN 'tools1.mirahezebots.org.miraheze.wiki.' [23:20:07] RhinosF1|NotHere ^ [23:20:10] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuAh [23:20:11] [02miraheze/dns] 07paladox 03a507673 - Revert "miraheze.wiki: failover dns for [bots|tools|db]1 (#186)" This reverts commit 8f9e193cbeb537da3d477e58f31953ddd0db2638. [23:20:32] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [23:20:58] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 255.99 ms [23:21:03] htmlCacheUpdate: 2 queued; 0 claimed (0 active, 0 abandoned); 0 delayed [23:21:09] an edit on meta inserts the job [23:21:47] paladox: tools1.mirahezebots.org.miraheze.wiki, why the miraheze.wiki on end? [23:22:03] that was your commit... [23:22:25] you did it in zones/miraheze.wiki [23:22:28] paladox: yeah I only want tools1.mirahezebots.org though [23:22:57] you forgot the . at the end i guess? [23:23:48] [02dns] 07RhinosF1 opened pull request 03#187: Patch 17 - 13https://git.io/JtuxO [23:23:58] paladox: ^ [23:24:10] better commit msg... [23:24:29] PROBLEM - gluster1 APT on gluster1 is UNKNOWN: APT UNKNOWN: 0 packages available for upgrade (0 critical updates). errors detected. [23:24:46] [02dns] 07RhinosF1 edited pull request 03#187: miraheze.wiki: failover dns for [bots|tools|db]1 - 13https://git.io/JtuxO [23:24:51] paladox: ^ [23:25:04] [02dns] 07paladox closed pull request 03#187: miraheze.wiki: failover dns for [bots|tools|db]1 - 13https://git.io/JtuxO [23:25:06] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuxZ [23:25:07] [02miraheze/dns] 07RhinosF1 039a5d6f0 - miraheze.wiki: failover dns for [bots|tools|db]1 (#187) * miraheze.wiki: failover dns for [bots|tools|db]1 * Update miraheze.wiki [23:25:46] deployed [23:26:29] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [23:27:00] paladox: ty [23:31:54] what I got so far: https://github.com/miraheze/mediawiki/blob/REL1_35/includes/jobqueue/jobs/HTMLCacheUpdateJob.php#L71 is called, but all cache updates go to https://github.com/miraheze/mediawiki/blob/REL1_35/includes/jobqueue/jobs/HTMLCacheUpdateJob.php#L79 [23:31:55] [ mediawiki/HTMLCacheUpdateJob.php at REL1_35 · miraheze/mediawiki · GitHub ] - github.com [23:31:55] [ mediawiki/HTMLCacheUpdateJob.php at REL1_35 · miraheze/mediawiki · GitHub ] - github.com [23:32:27] whereas 'invalidateTitles', which purges varnish, is only called in the elseif or else statements, but not in the if statement [23:33:03] I presume partitionBacklinkJob creates purge jobs for cdn purge, but I couldn't verify that [23:40:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtuxA [23:40:09] [02miraheze/mw-config] 07paladox 030a9ee8e - Switch over wikis beginning with h to new file storage [23:41:11] miraheze/mw-config - paladox the build passed. [23:41:28] https://github.com/wikimedia/mediawiki/blob/ffbe2b8043957d11595210761b4407cc9e77cde8/includes/page/WikiPage.php#L3696 nevermind, the purging should happen here.. [23:41:28] [ mediawiki/WikiPage.php at ffbe2b8043957d11595210761b4407cc9e77cde8 · wikimedia/mediawiki · GitHub ] - github.com [23:41:39] let's see [23:50:45] after an edit to https://meta.miraheze.org/wiki/User:Southparkfan/Test, https://github.com/wikimedia/mediawiki/blob/467a4f32b0475fd2840264ada94e3ce87518d22a/includes/cache/HtmlCacheUpdater.php#L122 is equal to [ 'https://meta.miraheze.org/wiki/User:Southparkfan/Test', 'https://meta.miraheze.org/w/index.php?title=User:Southparkfan/Test&action=history' ] [23:50:46] [ User:Southparkfan/Test - Miraheze Meta ] - meta.miraheze.org [23:50:48] [ mediawiki/HtmlCacheUpdater.php at 467a4f32b0475fd2840264ada94e3ce87518d22a · wikimedia/mediawiki · GitHub ] - github.com [23:50:48] seems right [23:50:49] [ Revision history of "User:Southparkfan/Test" - Miraheze Meta ] - meta.miraheze.org