[00:03:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.41, 23.87, 18.49 [00:03:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIszc [00:03:50] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 33.13, 23.44, 16.32 [00:03:52] [02miraheze/mw-config] 07Universal-Omega 03e8e3f99 - Complete re-organization of ManageWikiSettings [00:03:53] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [00:03:57] [02mw-config] 07Universal-Omega opened pull request 03#3523: Complete re-organization of ManageWikiSettings - 13https://git.io/JIszC [00:04:16] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.61, 2.52, 1.64 [00:05:50] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 15.69, 20.54, 16.15 [00:06:15] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 2.90, 2.68, 1.81 [00:06:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIszD [00:06:35] [02miraheze/mw-config] 07Universal-Omega 03461cf4a - edit tab rename to editing [00:06:36] [02mw-config] 07Universal-Omega synchronize pull request 03#3523: Complete re-organization of ManageWikiSettings - 13https://git.io/JIszC [00:07:51] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.53, 18.16, 15.79 [00:13:12] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.60, 19.75, 18.88 [00:13:57] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 35.41, 26.79, 20.08 [00:13:57] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.19, 6.20, 4.96 [00:15:55] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.27, 5.54, 4.87 [00:17:50] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 14.60, 20.79, 19.23 [00:19:51] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.13, 19.86, 19.10 [00:23:56] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.55, 23.18, 20.79 [00:25:03] PROBLEM - puppet2 puppetdb on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8081: Connection refused [00:25:13] PROBLEM - puppet2 puppetserver on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8140: Connection refused [00:25:15] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [00:25:17] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:25:30] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown) [00:25:35] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown) [00:25:38] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown) [00:25:39] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 2 minutes ago with 26 failures. Failed resources (up to 3 shown) [00:25:40] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 476 failures. Last run 2 minutes ago with 476 failures. Failed resources (up to 3 shown) [00:25:50] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:26:04] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 494 failures. Last run 2 minutes ago with 494 failures. Failed resources (up to 3 shown) [00:26:07] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown) [00:26:10] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [00:26:13] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 494 failures. Last run 2 minutes ago with 494 failures. Failed resources (up to 3 shown) [00:26:26] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [00:26:28] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 475 failures. Last run 2 minutes ago with 475 failures. Failed resources (up to 3 shown) [00:26:35] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown) [00:26:39] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [00:26:41] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [00:26:41] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 3 minutes ago with 20 failures. Failed resources (up to 3 shown) [00:26:58] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 492 failures. Last run 2 minutes ago with 492 failures. Failed resources (up to 3 shown) [00:27:03] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [00:27:05] RECOVERY - puppet2 puppetdb on puppet2 is OK: TCP OK - 0.004 second response time on 51.89.160.129 port 8081 [00:27:13] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 488 failures. Last run 3 minutes ago with 488 failures. Failed resources (up to 3 shown) [00:27:15] RECOVERY - puppet2 puppetserver on puppet2 is OK: TCP OK - 0.001 second response time on 51.89.160.129 port 8140 [00:27:33] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 479 failures. Last run 2 minutes ago with 479 failures. Failed resources (up to 3 shown) [00:27:33] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 494 failures. Last run 3 minutes ago with 494 failures. Failed resources (up to 3 shown) [00:27:35] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 491 failures. Last run 3 minutes ago with 491 failures. Failed resources (up to 3 shown) [00:29:52] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/proxmox.gpg],File[/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg],File[/etc/network/interfaces.d/50-cloud-init.cfg],File[/etc/logrotate.d/puppet] [00:30:40] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 319 failures. Last run 2 minutes ago with 319 failures. Failed resources (up to 3 shown): File[/etc/nginx/fastcgi_params],File[/usr/local/bin/nginx-prometheus-exporter],File[antiguabarbudacalypso.com],File[antiguabarbudacalypso.com_private] [00:31:59] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 4.83, 3.84, 3.04 [00:34:40] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.91, 7.03, 6.03 [00:35:04] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.04, 7.00, 5.82 [00:35:35] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 2 minutes ago with 18 failures. Failed resources (up to 3 shown) [00:35:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 9 seconds ago with 17 failures. Failed resources (up to 3 shown) [00:35:52] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.84, 24.28, 22.18 [00:35:55] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.24, 3.63, 3.13 [00:35:57] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:24] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:39] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 35 seconds ago with 18 failures. Failed resources (up to 3 shown) [00:36:54] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 3 minutes ago with 18 failures. Failed resources (up to 3 shown) [00:37:02] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.71, 6.41, 5.76 [00:37:50] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 19.89, 22.27, 21.68 [00:37:52] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 3.12, 3.38, 3.09 [00:38:06] PROBLEM - puppet2 puppetserver on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8140: Connection refused [00:38:22] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIsVU [00:38:23] [02miraheze/MirahezeMagic] 07Universal-Omega 035cb5ad1 - Add new MWS tab messages [00:38:25] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [00:38:32] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.15, 6.70, 6.16 [00:39:06] [02MirahezeMagic] 07Universal-Omega opened pull request 03#164: Add new MWS tab messages - 13https://git.io/JIsVt [00:39:56] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [00:40:00] RECOVERY - puppet2 puppetserver on puppet2 is OK: TCP OK - 0.000 second response time on 51.89.160.129 port 8140 [00:41:50] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.64, 22.47, 21.85 [00:42:23] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.17, 7.19, 6.48 [00:43:02] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.04, 7.33, 6.27 [00:43:50] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.37, 22.85, 22.13 [00:43:52] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:44:18] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.09, 7.24, 6.57 [00:47:02] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.79, 6.60, 6.27 [00:47:43] PROBLEM - puppet2 puppetserver on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8140: Connection refused [00:48:09] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.64, 6.77, 6.58 [00:49:39] RECOVERY - puppet2 puppetserver on puppet2 is OK: TCP OK - 0.000 second response time on 51.89.160.129 port 8140 [00:50:45] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:50:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIswu [00:50:50] [02miraheze/mw-config] 07Universal-Omega 034b381b2 - Update ManageWikiSettings.php [00:50:51] [02mw-config] 07Universal-Omega synchronize pull request 03#3523: Complete re-organization of ManageWikiSettings - 13https://git.io/JIszC [00:53:32] !log depool; repool; mw[45] [00:53:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:49] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:53:50] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.14, 23.99, 22.70 [00:53:51] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:54:09] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:54:13] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:54:23] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:54:25] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [00:54:37] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [00:54:38] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:54:41] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:54:53] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:54:59] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:15] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:17] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:29] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:33] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:34] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:55:35] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:39] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:39] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:40] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:46] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:54] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:56:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIsrC [00:56:19] [02miraheze/mw-config] 07Universal-Omega 033d9b4f3 - fix tabs [00:56:20] [02mw-config] 07Universal-Omega synchronize pull request 03#3523: Complete re-organization of ManageWikiSettings - 13https://git.io/JIszC [00:56:20] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.12, 5.96, 4.76 [00:56:26] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:56:42] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:58:18] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.24, 5.81, 4.86 [01:00:40] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:00:41] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:01:12] PROBLEM - puppet2 Current Load on puppet2 is CRITICAL: connect to address 51.89.160.129 port 5666: Connection refusedconnect to host 51.89.160.129 port 5666: Connection refused [01:01:51] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 12.11, 20.36, 22.28 [01:03:07] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 2.48, 0.76, 0.27 [01:03:13] PROBLEM - puppet2 puppetserver on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8140: Connection refused [01:03:16] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [01:04:09] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [01:04:28] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 253 failures. Last run 2 minutes ago with 253 failures. Failed resources (up to 3 shown) [01:04:41] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown) [01:05:13] RECOVERY - puppet2 puppetserver on puppet2 is OK: TCP OK - 0.002 second response time on 51.89.160.129 port 8140 [01:05:35] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 285 failures. Last run 3 minutes ago with 285 failures. Failed resources (up to 3 shown) [01:05:40] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 250 failures. Last run 3 minutes ago with 250 failures. Failed resources (up to 3 shown) [01:06:40] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: connect to address 51.89.160.129 port 5666: Connection refusedconnect to host 51.89.160.129 port 5666: Connection refused [01:07:51] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.03, 15.56, 19.53 [01:09:33] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 19 failures. Last run 2 minutes ago with 19 failures. Failed resources (up to 3 shown) [01:09:37] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 2 minutes ago with 20 failures. Failed resources (up to 3 shown) [01:09:39] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:09:40] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 19 failures. Last run 2 minutes ago with 19 failures. Failed resources (up to 3 shown) [01:09:52] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 2 minutes ago with 18 failures. Failed resources (up to 3 shown) [01:09:52] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:09:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 2 minutes ago with 16 failures. Failed resources (up to 3 shown) [01:10:05] ^ me [01:10:23] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:10:26] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:10:28] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 3 minutes ago with 20 failures. Failed resources (up to 3 shown) [01:10:37] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [01:10:37] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown) [01:10:41] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 3 minutes ago with 23 failures. Failed resources (up to 3 shown) [01:11:00] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 3 minutes ago with 16 failures. Failed resources (up to 3 shown) [01:11:28] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [01:11:30] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [01:11:39] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw4 [01:12:23] PROBLEM - mw4 Check Gluster Clients on mw4 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [01:13:36] ? [01:13:51] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [01:13:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:14:11] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.37, 6.48, 5.17 [01:14:12] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [01:14:24] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [01:14:24] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [01:14:25] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [01:14:37] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [01:14:38] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [01:14:40] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:14:40] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [01:14:59] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:14] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:29] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [01:15:30] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:30] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [01:15:30] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [01:15:34] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:36] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [01:15:37] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:38] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [01:15:38] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:39] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:40] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [01:15:54] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:08] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 6.72, 6.47, 5.32 [01:16:14] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:26] RECOVERY - mw4 Check Gluster Clients on mw4 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:16:28] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:33] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [01:16:55] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:12] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:18:10] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [01:18:11] PROBLEM - mw5 Check Gluster Clients on mw5 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [01:19:28] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw5 [01:19:29] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [01:19:38] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw5 [01:22:10] RECOVERY - mw5 Check Gluster Clients on mw5 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:22:13] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [01:23:29] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [01:23:29] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [01:23:38] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [01:24:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIsig [01:24:25] [02miraheze/mw-config] 07Universal-Omega 03690607d - revert rename edit tab to editing [01:24:27] [02mw-config] 07Universal-Omega synchronize pull request 03#3523: Complete re-organization of ManageWikiSettings - 13https://git.io/JIszC [01:25:50] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIsiS [01:25:51] [02miraheze/MirahezeMagic] 07Universal-Omega 0348815d9 - Update en.json [01:25:53] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#164: Add new MWS tab messages - 13https://git.io/JIsVt [01:33:59] !log depool; repool; mw[67] [01:34:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:01] PROBLEM - puppet2 Current Load on puppet2 is CRITICAL: CRITICAL - load average: 4.55, 2.90, 1.72 [01:35:48] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIsXv [01:35:49] [02miraheze/MirahezeMagic] 07Universal-Omega 03d3a2048 - Update en.json [01:35:51] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#164: Add new MWS tab messages - 13https://git.io/JIsVt [01:36:08] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [01:37:35] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [01:37:38] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [01:37:44] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw6 [01:39:05] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 2.03, 0.82, 0.30 [01:40:16] PROBLEM - mw6 Check Gluster Clients on mw6 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [01:40:48] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown) [01:42:21] RECOVERY - mw6 Check Gluster Clients on mw6 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:43:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [01:44:14] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.95, 8.17, 5.40 [01:44:36] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:45:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:45:07] PROBLEM - mw7 Check Gluster Clients on mw7 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [01:46:09] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 3.79, 6.41, 5.08 [01:46:30] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:46:58] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:54:15] !log set mw[4567] cores at 5; set puppet2 cores to 2 [01:54:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:50] Congrats Doug [02:04:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIsMi [02:04:49] [02miraheze/puppet] 07paladox 030e2d41e - puppet: Tweak configuration * Up the heap for puppetserver and puppetdb * Decrease puppetdb command processing to 1 [02:04:50] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:04:52] [02puppet] 07paladox opened pull request 03#1540: puppet: Tweak configuration - 13https://git.io/JIsMP [02:05:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JIsMS [02:05:29] [02miraheze/puppet] 07paladox 03d489e33 - Update init.pp [02:05:31] [02puppet] 07paladox synchronize pull request 03#1540: puppet: Tweak configuration - 13https://git.io/JIsMP [02:05:46] [02puppet] 07paladox closed pull request 03#1540: puppet: Tweak configuration - 13https://git.io/JIsMP [02:05:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JIsMQ [02:05:49] [02miraheze/puppet] 07paladox 0356211a7 - puppet: Tweak configuration (#1540) * puppet: Tweak configuration * Up the heap for puppetserver and puppetdb * Decrease puppetdb command processing to 1 * Update init.pp [02:05:50] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:05:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:37:16] miraheze/mw-config/Universal-Omega-patch-3/690607d - Universal Omega The build passed. https://travis-ci.org/miraheze/mw-config/builds/747749525 [02:51:10] ty @Sario :) [03:33:48] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 1.25, 2.17, 1.13 [03:34:01] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (2 critical updates). [03:35:01] miraheze/MirahezeMagic/Universal-Omega-patch-1/d3a2048 - Universal Omega The build passed. https://travis-ci.org/miraheze/MirahezeMagic/builds/747750307 [03:35:47] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.03, 1.01, 0.89 [04:31:18] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (2 critical updates). [04:35:23] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (2 critical updates). [05:20:06] PROBLEM - cubashpedia.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cubashpedia.cf' expires in 15 day(s) (Mon 21 Dec 2020 05:17:43 GMT +0000). [05:23:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIGLe [05:23:42] [02miraheze/ssl] 07MirahezeSSLBot 031c1b179 - Bot: Update SSL cert for cubashpedia.cf [05:33:41] RECOVERY - cubashpedia.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'cubashpedia.cf' will expire on Fri 05 Mar 2021 04:23:33 GMT +0000. [06:19:20] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:21:18] RECOVERY - mw7 APT on mw7 is OK: APT OK: 12 packages available for upgrade (0 critical updates). [06:33:20] RECOVERY - mw6 APT on mw6 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:39:23] RECOVERY - services1 APT on services1 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [07:00:23] RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 12 packages available for upgrade (0 critical updates). [07:16:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIGlu [07:16:25] [02miraheze/mw-config] 07Universal-Omega 0338cd73a - Update ManageWikiSettings.php [07:16:27] [02mw-config] 07Universal-Omega synchronize pull request 03#3523: Complete re-organization of ManageWikiSettings - 13https://git.io/JIszC [08:43:04] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JIG2w [08:43:06] [02miraheze/ssl] 07Reception123 034fb7801 - add johanloopmans.nl cert [09:04:40] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (2 critical updates). [09:30:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIGiZ [09:30:21] [02miraheze/services] 07MirahezeSSLBot 0364121a6 - BOT: Updating services config for wikis [09:57:23] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JIGMw [09:57:24] [02miraheze/dns] 07Reception123 03ca3feff - add rothwell-leeds.co.uk zone [10:03:21] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JIGDK [10:03:22] [02miraheze/ssl] 07Reception123 03e564e99 - add www.rothwell-leeds.co.uk cert [14:32:44] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 9.36, 6.64, 5.67 [14:34:39] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.18, 6.19, 5.63 [14:45:12] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 4.49, 3.23, 2.24 [14:47:12] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 3.26, 3.22, 2.36 [15:21:12] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 6.87, 3.77, 2.64 [15:22:05] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.13, 7.89, 6.83 [15:22:26] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.41, 8.03, 6.74 [15:22:39] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.12, 19.58, 17.45 [15:23:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:105a::10/cpweb, 2607:5300:205:200::2ac4/cpweb [15:24:39] PROBLEM - wikibeats.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikibeats.net could not be found [15:24:39] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 19.42, 19.53, 17.70 [15:25:13] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.90, 3.84, 2.94 [15:25:15] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc-multiverse.dcwikis.com could not be found [15:25:16] PROBLEM - tensegritywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tensegritywiki.com could not be found [15:25:16] PROBLEM - electowiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for electowiki.org could not be found [15:25:17] PROBLEM - www.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pornwiki.org could not be found [15:25:18] PROBLEM - wiki.helioss.co - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.helioss.co could not be found [15:25:18] PROBLEM - www.baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.baharna.org could not be found [15:25:19] PROBLEM - zw.fontainebleau-avon.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zw.fontainebleau-avon.fr could not be found [15:25:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:25:59] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 9.22, 9.83, 7.95 [15:26:11] PROBLEM - pwiki.arkcls.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.arkcls.com could not be found [15:26:13] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thelonsdalebattalion.co.uk could not be found [15:26:13] PROBLEM - savagepedia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savagepedia.wiki could not be found [15:26:13] PROBLEM - wiki.mcpirevival.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mcpirevival.tk could not be found [15:26:14] PROBLEM - opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for opendatascot.org could not be found [15:26:25] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 9.27, 9.28, 7.62 [15:28:23] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 7.18, 8.24, 7.42 [15:29:13] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 2.35, 3.26, 2.93 [15:29:56] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 3.60, 6.91, 7.23 [15:31:30] RECOVERY - wikibeats.net - reverse DNS on sslhost is OK: rDNS OK - wikibeats.net reverse DNS resolves to cp7.miraheze.org [15:32:04] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: rDNS OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp6.miraheze.org [15:32:07] RECOVERY - electowiki.org - reverse DNS on sslhost is OK: rDNS OK - electowiki.org reverse DNS resolves to cp7.miraheze.org [15:32:07] RECOVERY - tensegritywiki.com - reverse DNS on sslhost is OK: rDNS OK - tensegritywiki.com reverse DNS resolves to cp7.miraheze.org [15:32:09] RECOVERY - wiki.helioss.co - reverse DNS on sslhost is OK: rDNS OK - wiki.helioss.co reverse DNS resolves to cp6.miraheze.org [15:32:09] RECOVERY - www.baharna.org - reverse DNS on sslhost is OK: rDNS OK - www.baharna.org reverse DNS resolves to cp6.miraheze.org [15:32:10] RECOVERY - zw.fontainebleau-avon.fr - reverse DNS on sslhost is OK: rDNS OK - zw.fontainebleau-avon.fr reverse DNS resolves to cp7.miraheze.org [15:32:15] RECOVERY - www.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.pornwiki.org reverse DNS resolves to cp7.miraheze.org [15:32:52] RECOVERY - pwiki.arkcls.com - reverse DNS on sslhost is OK: rDNS OK - pwiki.arkcls.com reverse DNS resolves to cp7.miraheze.org [15:32:55] RECOVERY - thelonsdalebattalion.co.uk - reverse DNS on sslhost is OK: rDNS OK - thelonsdalebattalion.co.uk reverse DNS resolves to cp6.miraheze.org [15:32:56] RECOVERY - wiki.mcpirevival.tk - reverse DNS on sslhost is OK: rDNS OK - wiki.mcpirevival.tk reverse DNS resolves to cp6.miraheze.org [15:32:56] RECOVERY - savagepedia.wiki - reverse DNS on sslhost is OK: rDNS OK - savagepedia.wiki reverse DNS resolves to cp7.miraheze.org [15:33:11] RECOVERY - opendatascot.org - reverse DNS on sslhost is OK: rDNS OK - opendatascot.org reverse DNS resolves to cp7.miraheze.org [15:34:34] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (2 critical updates). [16:04:22] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 11.96, 8.31, 6.70 [16:05:46] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.conworlds.org' expires in 15 day(s) (Mon 21 Dec 2020 16:00:55 GMT +0000). [16:06:17] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.43, 7.47, 6.59 [16:08:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIZHK [16:08:15] [02miraheze/ssl] 07MirahezeSSLBot 037c26094 - Bot: Update SSL cert for wiki.conworlds.org [16:17:55] @dmehus Congratulations [16:18:06] The Newest Steward of Miraheze [16:18:24] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (2 critical updates). [16:19:15] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Fri 05 Mar 2021 15:08:06 GMT +0000. [16:32:58] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (2 critical updates). [16:33:56] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 9.30, 7.43, 6.56 [16:35:56] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 7.22, 7.41, 6.67 [17:04:10] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.50, 8.46, 7.30 [17:04:24] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 11.66, 9.24, 7.72 [17:06:05] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.93, 7.63, 7.15 [17:06:23] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.59, 8.61, 7.69 [17:08:22] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 6.54, 7.88, 7.53 [17:13:47] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.nonbinary.wiki could not be found [17:13:48] PROBLEM - public.revi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for public.revi.wiki could not be found [17:13:48] PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found [17:13:50] PROBLEM - en.wikireserve.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.wikireserve.com could not be found [17:13:52] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [17:13:54] PROBLEM - poserdazfreebies.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for poserdazfreebies.orain.org could not be found [17:13:54] PROBLEM - warmage.necrocity.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for warmage.necrocity.co.uk could not be found [17:13:54] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found [17:13:54] PROBLEM - wiki.meeusen.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.meeusen.net could not be found [17:13:55] PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found [17:13:56] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 11.03, 9.00, 7.83 [17:13:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [17:13:59] PROBLEM - kk.uncyc.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kk.uncyc.tk could not be found [17:13:59] PROBLEM - www.programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.programming.red could not be found [17:14:06] PROBLEM - marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marinebiodiversitymatrix.org could not be found [17:14:08] PROBLEM - wiki.grottocenter.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grottocenter.org could not be found [17:14:18] PROBLEM - animefeet.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for animefeet.miraheze.org could not be found [17:14:22] PROBLEM - towarzystwo.prawa.kredytowego.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for towarzystwo.prawa.kredytowego.pl could not be found [17:14:24] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.45, 9.33, 8.23 [17:14:30] PROBLEM - beaconspace.unrestrictedlorefare.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for beaconspace.unrestrictedlorefare.com could not be found [17:14:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb [17:14:32] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [17:14:41] PROBLEM - www.johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.johanloopmans.nl could not be found [17:14:45] PROBLEM - wiki.whentheycry.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.whentheycry.org could not be found [17:15:56] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.72, 8.34, 7.76 [17:15:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:16:25] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 7.09, 8.49, 8.06 [17:16:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:20:34] @dmehus what do you think of techaccesswiki [17:20:38] RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - data.nonbinary.wiki reverse DNS resolves to cp6.miraheze.org [17:20:39] RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: rDNS OK - monarchists.wiki reverse DNS resolves to cp7.miraheze.org [17:20:39] RECOVERY - kk.uncyc.tk - reverse DNS on sslhost is OK: rDNS OK - kk.uncyc.tk reverse DNS resolves to cp7.miraheze.org [17:20:40] RECOVERY - www.programming.red - reverse DNS on sslhost is OK: rDNS OK - www.programming.red reverse DNS resolves to cp6.miraheze.org [17:20:41] RECOVERY - en.wikireserve.com - reverse DNS on sslhost is OK: rDNS OK - en.wikireserve.com reverse DNS resolves to cp6.miraheze.org [17:20:47] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: rDNS OK - podpedia.org reverse DNS resolves to cp6.miraheze.org [17:20:47] RECOVERY - public.revi.wiki - reverse DNS on sslhost is OK: rDNS OK - public.revi.wiki reverse DNS resolves to cp6.miraheze.org [17:20:48] RECOVERY - warmage.necrocity.co.uk - reverse DNS on sslhost is OK: rDNS OK - warmage.necrocity.co.uk reverse DNS resolves to cp6.miraheze.org [17:20:49] RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: rDNS OK - adadevelopersacademy.wiki reverse DNS resolves to cp6.miraheze.org [17:20:50] RECOVERY - wiki.meeusen.net - reverse DNS on sslhost is OK: rDNS OK - wiki.meeusen.net reverse DNS resolves to cp7.miraheze.org [17:20:52] RECOVERY - orain.org - reverse DNS on sslhost is OK: rDNS OK - orain.org reverse DNS resolves to cp6.miraheze.org [17:20:53] RECOVERY - poserdazfreebies.orain.org - reverse DNS on sslhost is OK: rDNS OK - poserdazfreebies.orain.org reverse DNS resolves to cp7.miraheze.org [17:20:53] RECOVERY - marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - marinebiodiversitymatrix.org reverse DNS resolves to cp6.miraheze.org [17:20:58] RECOVERY - wiki.grottocenter.org - reverse DNS on sslhost is OK: rDNS OK - wiki.grottocenter.org reverse DNS resolves to cp7.miraheze.org [17:20:59] RECOVERY - animefeet.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - animefeet.miraheze.org reverse DNS resolves to cp6.miraheze.org [17:21:05] RECOVERY - towarzystwo.prawa.kredytowego.pl - reverse DNS on sslhost is OK: rDNS OK - towarzystwo.prawa.kredytowego.pl reverse DNS resolves to cp6.miraheze.org [17:21:21] RECOVERY - beaconspace.unrestrictedlorefare.com - reverse DNS on sslhost is OK: rDNS OK - beaconspace.unrestrictedlorefare.com reverse DNS resolves to cp6.miraheze.org [17:21:24] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: rDNS OK - wiki.nowchess.org reverse DNS resolves to cp7.miraheze.org [17:21:28] RECOVERY - wiki.whentheycry.org - reverse DNS on sslhost is OK: rDNS OK - wiki.whentheycry.org reverse DNS resolves to cp7.miraheze.org [17:21:37] RECOVERY - www.johanloopmans.nl - reverse DNS on sslhost is OK: rDNS OK - www.johanloopmans.nl reverse DNS resolves to cp7.miraheze.org [17:23:56] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 12.00, 9.60, 8.29 [17:24:23] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 11.40, 10.44, 8.92 [17:25:56] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.70, 8.57, 8.08 [17:26:23] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.17, 9.43, 8.75 [17:27:56] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.63, 7.31, 7.69 [17:28:23] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.83, 8.26, 8.41 [17:28:35] blackwidowmovie0: I think you are talking about something like [17:28:41] .mh mh:dev [17:28:41] https://meta.miraheze.org/wiki/mh:dev [17:28:58] .mh mh:dev: [17:28:58] https://meta.miraheze.org/wiki/mh:dev: [17:42:24] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 9.42, 7.96, 7.81 [17:42:47] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 12.73, 8.56, 7.73 [17:44:43] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.97, 8.09, 7.68 [17:46:23] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 6.55, 7.82, 7.85 [17:52:23] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 12.60, 8.96, 8.09 [17:54:48] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [17:56:22] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.77, 8.76, 8.31 [17:58:21] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.43, 7.30, 7.83 [18:04:07] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 5.31, 3.82, 2.56 [18:04:23] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 13.65, 10.17, 8.76 [18:05:06] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 8.63, 8.78, 7.86 [18:06:03]