[00:02:56] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:02] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.16, 6.96, 6.46 [00:03:03] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:04] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:07] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:12] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [00:03:15] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:15] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:16] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 2 minutes ago with 17 failures. Failed resources (up to 3 shown) [00:03:23] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:28] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 2 minutes ago with 15 failures. Failed resources (up to 3 shown) [00:03:38] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:44] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:11] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:16] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:25] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:26] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:30] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:05:02] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.42, 7.06, 6.55 [00:05:31] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 204 failures. Last run 4 minutes ago with 204 failures. Failed resources (up to 3 shown) [00:07:01] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.52, 7.30, 6.68 [00:09:01] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.15, 7.18, 6.71 [00:13:00] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.10, 7.68, 7.01 [00:13:26] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:13:28] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:14:16] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:26] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:14:30] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [00:15:03] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:04] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:08] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:12] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:15] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:15] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:16] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:32] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [00:15:38] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:44] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:16:11] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:16:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.49, 7.50, 7.10 [00:33:58] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:46] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.015 second response time [00:40:20] RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 5 processes with command name 'php-fpm7.3' [00:40:54] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.22, 7.12, 7.05 [00:42:54] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.93, 7.37, 7.14 [01:02:49] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.10, 7.36, 7.05 [01:04:48] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.44, 7.21, 7.02 [01:10:47] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.58, 6.64, 6.79 [05:47:23] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.010 second response time [05:47:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [05:48:12] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:48:15] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:48:20] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:48:44] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:45] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [05:49:12] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [05:49:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [05:51:03] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.47, 1.07, 1.89 [05:52:45] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 21% [05:53:04] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.15, 0.76, 1.67 [05:53:12] PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:45] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [06:01:51] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 51% [06:03:25] !log purge binary logs before '2019-07-13 06:00:00'; [06:05:12] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 2% [06:05:12] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5805 bytes in 0.036 second response time [06:05:17] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41778 bytes in 0.223 second response time [06:05:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:05:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:05:48] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 2% [06:05:49] RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 347 bytes in 0.015 second response time [06:06:12] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [06:06:15] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [06:06:20] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [06:06:45] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 7% [06:06:48] !log purge binary logs before '2019-07-13 06:00:00'; [06:06:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:07:03] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 4.89, 2.03, 1.43 [06:26:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.24, 7.42, 6.17 [06:30:32] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.09, 7.49, 6.50 [06:31:31] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 48% [06:33:29] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 0% [06:36:30] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.85, 6.52, 6.36 [06:43:03] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.74, 1.33, 1.98 [06:47:04] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.03, 1.46, 1.87 [06:49:04] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.76, 1.47, 1.82 [06:50:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXLK [06:50:12] [02miraheze/services] 07MirahezeSSLBot 038025c8c - BOT: Updating services config for wikis [06:55:04] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.18, 1.57, 1.72 [06:55:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXL6 [06:55:11] [02miraheze/services] 07MirahezeSSLBot 03bee6766 - BOT: Updating services config for wikis [06:57:04] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.85, 1.68, 1.74 [06:59:04] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.04, 1.78, 1.76 [07:01:04] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.83, 1.70, 1.72 [07:03:03] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.29, 1.59, 1.68 [07:07:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXLX [07:07:04] [02miraheze/puppet] 07Reception123 0300e262e - fix typo [09:14:57] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.52, 6.80, 6.14 [09:18:56] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.59, 6.74, 6.26 [09:22:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.75, 6.99, 6.49 [09:24:55] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.29, 6.79, 6.47 [10:53:25] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.39, 1.66, 1.14 [10:55:22] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.50, 1.66, 1.20 [11:32:21] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.98, 1.71, 1.50 [11:34:19] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.52, 1.61, 1.48 [11:50:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXmQ [11:50:13] [02miraheze/services] 07MirahezeSSLBot 03a00e29b - BOT: Updating services config for wikis [11:55:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXmA [11:55:11] [02miraheze/services] 07MirahezeSSLBot 0322c6fa9 - BOT: Updating services config for wikis [12:00:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXYf [12:00:14] [02miraheze/services] 07MirahezeSSLBot 0347e901b - BOT: Updating services config for wikis [12:09:25] !log renamed medlitreviewswiki to theliteratureprojectwiki. [12:09:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:10:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXYk [12:10:13] [02miraheze/services] 07MirahezeSSLBot 03558a099 - BOT: Updating services config for wikis [14:33:40] PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:57] RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 5.092 second response time [14:52:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.44, 6.53, 5.83 [14:54:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.56, 6.22, 5.80 [15:12:23] PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:15] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.82, 1.63, 1.00 [15:26:39] RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.009 second response time [15:38:56] PROBLEM - misc3 Lizardfs Master Port 1 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9419: Connection refused [15:39:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:39:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:39:48] PROBLEM - misc3 Lizardfs Master Port 2 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9420: Connection refused [15:39:51] PROBLEM - misc3 Lizardfs Master Port 3 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9421: Connection refused [15:40:12] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:40:15] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:40:20] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:41:12] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [15:41:39] !log restarted lizardfs-master on misc3 (appears to have stopped it's self) [15:42:56] RECOVERY - misc3 Lizardfs Master Port 1 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9419 [15:43:12] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 12% [15:43:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:43:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:43:48] RECOVERY - misc3 Lizardfs Master Port 2 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9420 [15:43:51] RECOVERY - misc3 Lizardfs Master Port 3 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9421 [15:44:12] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:44:15] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:44:20] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:51:03] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.42, 1.39, 1.82 [15:57:04] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.19, 1.41, 1.69 [15:57:31] !log [16:41:38] <+paladox> !log restarted lizardfs-master on misc3 (appears to have stopped it's self) [15:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:01:04] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.94, 2.06, 1.88 [16:07:03] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.77, 1.67, 1.79 [16:11:54] !log root@db4:/bacula/restore# rm -rf * [16:11:57] Reception123 ^ [16:11:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:12:12] :) [16:17:03] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.47, 1.25, 1.57 [16:23:40] PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 703 bytes in 0.024 second response time [16:25:39] RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.017 second response time [16:53:24] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.95, 1.57, 1.28 [16:55:21] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.03, 1.35, 1.23 [17:40:03] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.94, 6.64, 6.17 [17:42:02] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.96, 6.41, 6.14 [18:08:56] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.14, 7.15, 6.56 [18:10:55] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.52, 7.67, 6.81 [18:12:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.62, 7.53, 6.86 [18:30:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.66, 6.44, 6.72 [18:36:49] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.97, 6.81, 6.79 [18:38:48] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.77, 6.75, 6.77 [19:30:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXZV [19:30:14] [02miraheze/services] 07MirahezeSSLBot 03e1e41fc - BOT: Updating services config for wikis [19:32:36] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.15, 7.44, 6.55 [19:41:32] !log reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki Reception123 --delete [19:41:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:42:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.16, 7.33, 7.17 [19:46:32] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_33 [+0/-0/±2] 13https://git.io/fjXZX [19:46:34] [02miraheze/mediawiki] 07JohnFLewis 037776f16 - update MW/CW [19:48:33] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.42, 6.20, 6.71 [19:49:49] [02mw-config] 07JohnFLewis closed pull request 03#2694: Revert "Add missing sql to WikiForum" - 13https://git.io/fjinI [19:49:51] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXZD [19:49:52] [02miraheze/mw-config] 07paladox 035c2516d - Revert "Add missing sql to WikiForum (#2693)" (#2694) This reverts commit 359c10c09bf5c6a6f4651e09265a8e1462c901f0. [20:20:31] [02mw-config] 07paladox deleted branch 03revert-2693-paladox-patch-2 - 13https://git.io/vbvb3 [20:20:33] [02miraheze/mw-config] 07paladox deleted branch 03revert-2693-paladox-patch-2 [21:09:31] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wintermoor.net_private] [21:15:31] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:57:04] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.12, 1.57, 0.72 [22:03:04] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.54, 1.90, 1.16 [22:05:04] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.30, 2.12, 1.33 [22:16:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.28, 6.73, 6.25 [22:20:07] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.41, 6.52, 6.27 [22:29:04] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.41, 1.42, 1.81 [22:31:03] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.67, 1.19, 1.67 [22:44:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.89, 7.08, 6.49 [22:46:06] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.81, 7.45, 6.68 [22:48:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.65, 6.57, 6.44 [22:54:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.80, 7.14, 6.70 [22:54:35] PROBLEM - munwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'munwiki.info' expires in 15 day(s) (Mon 29 Jul 2019 10:51:10 PM GMT +0000). [22:54:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXcf [22:54:58] [02miraheze/ssl] 07MirahezeSSLBot 03e5f74dd - Bot: Update SSL cert for munwiki.info [23:04:06] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.34, 7.30, 6.88 [23:04:35] RECOVERY - munwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'munwiki.info' will expire on Fri 11 Oct 2019 09:54:51 PM GMT +0000. [23:06:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.38, 7.26, 6.91 [23:07:04] someone want to put eyes on mw3? [23:08:50] PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:12:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.47, 6.51, 6.74 [23:17:12] RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 6.756 second response time [23:44:01] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.54, 6.83, 6.51 [23:46:01] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.04, 6.52, 6.43