[00:00:47] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.019 second response time [00:03:47] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:56] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.004 second response time [00:05:43] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 1.072 second response time [00:06:26] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.218 second response time [00:07:12] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.021 second response time [00:08:02] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.005 second response time [00:08:16] Today certainly has been a busy day [00:08:46] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.027 second response time [00:08:57] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 7.496 second response time [00:09:53] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 5.894 second response time [00:11:16] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 6.89, 3.82, 3.10 [00:11:48] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.694 second response time [00:12:18] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.693 second response time [00:12:44] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.51, 3.26, 2.82 [00:14:42] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.29, 3.64, 3.01 [00:15:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.81, 7.45, 6.81 [00:16:37] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.77, 3.32, 2.97 [00:16:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:17:27] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.71, 7.68, 6.96 [00:17:37] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:20:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:20:37] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.33, 5.67, 4.14 [00:20:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [00:22:08] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:22:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:25:57] hi [00:26:25] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [00:26:26] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [00:26:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:27:30] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 1.64, 3.69, 3.86 [00:27:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.06, 7.98, 7.55 [00:28:21] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:28:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:28:23] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:29:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:29:27] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 7.62, 8.09, 7.65 [00:31:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.79, 7.81, 7.59 [00:31:32] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.97, 4.17, 3.96 [00:33:34] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.63, 8.33, 7.79 [00:34:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [00:35:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:35:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.03, 7.91, 7.71 [00:35:52] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.30, 6.47, 5.75 [00:37:41] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.19, 8.33, 7.87 [00:37:56] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.59, 7.12, 6.06 [00:39:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:39:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.79, 7.71, 7.70 [00:39:56] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.19, 7.02, 6.15 [00:40:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:41:39] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.32, 8.51, 7.98 [00:42:05] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.00, 7.45, 6.40 [00:43:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [00:43:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:43:57] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:13] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.66, 7.25, 6.44 [00:44:26] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:44:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:44:44] Its still bloody 503'ing on me [00:45:08] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:46:00] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 8.628 second response time [00:46:09] Are you any closer to fixing it? [00:46:14] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.39, 7.14, 6.46 [00:46:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:47:35] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.81, 7.17, 7.55 [00:47:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:48:13] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.52, 6.17, 6.19 [00:48:26] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:51:34] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.64, 7.92, 7.71 [00:52:35] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [00:52:49] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [00:53:11] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.292 second response time [00:53:28] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [00:53:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 7.51, 7.58 [00:53:34] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:53:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:55:09] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.390 second response time [00:55:34] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 2.18, 2.76, 3.97 [00:56:38] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:57:25] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:57:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.07, 7.49, 7.52 [00:57:32] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:57:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:59:33] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 8.75, 5.00, 4.49 [01:00:43] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:01:38] Sorry that I keep bothering [01:01:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.86, 7.69, 7.60 [01:01:50] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [01:01:55] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [01:01:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:06:37] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:08:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:08:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:08:16] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:08:41] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:13:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEHR [01:13:43] [02miraheze/puppet] 07paladox 033a519a0 - Update mfschunkserver.cfg.erb [01:14:12] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.84, 7.27, 7.16 [01:14:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEH0 [01:14:39] [02miraheze/puppet] 07paladox 0310661d1 - Update mfschunkserver.cfg.erb [01:16:42] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:16:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:16:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:16:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:16:56] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:18:13] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:28] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:44] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:18:46] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.95, 7.28, 7.20 [01:18:54] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 1.22, 2.77, 3.98 [01:19:06] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:19:44] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:55] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:33] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:44] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:51] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:06] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.004 second response time [01:21:11] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.00, 5.68, 6.59 [01:21:22] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.673 second response time [01:21:30] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:39] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:03] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 4.177 second response time [01:22:27] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.017 second response time [01:22:29] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.006 second response time [01:23:27] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.01, 1.71, 3.21 [01:24:59] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [01:25:07] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.006 second response time [01:25:58] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:35] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:36] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 8.026 second response time [01:27:00] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:40] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:28:39] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 1.07, 1.55, 3.64 [01:29:06] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.020 second response time [01:29:31] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:45] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:06] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:36] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.40, 1.49, 3.36 [01:31:29] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.013 second response time [01:31:40] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [01:32:38] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.693 second response time [01:32:52] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.395 second response time [01:36:16] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:36:51] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.008 second response time [01:37:15] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:37:19] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.683 second response time [01:37:42] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.391 second response time [01:38:00] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.004 second response time [01:38:18] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 6.834 second response time [01:38:28] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.003 second response time [01:38:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:38:47] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.632 second response time [01:39:13] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.633 second response time [01:39:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:39:54] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:40:16] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:40:45] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:43:15] PROBLEM - mw3 Disk Space on mw3 is WARNING: DISK WARNING - free space: / 8416 MB (10% inode=98%); [01:48:01] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.73, 7.00, 5.99 [01:48:45] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [01:49:18] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [01:50:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [01:52:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:52:47] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:52:56] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.10, 7.38, 5.86 [01:53:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:53:52] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.74, 7.88, 6.82 [01:54:55] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.40, 6.23, 5.62 [02:05:50] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.54, 7.94, 7.19 [02:06:04] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.49, 7.30, 6.17 [02:10:08] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.95, 7.31, 6.41 [02:12:24] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.99, 6.46, 6.25 [02:12:33] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.16, 3.42, 2.99 [02:12:34] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:14:29] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.21, 3.15, 2.95 [02:14:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:18:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.62, 7.34, 7.45 [02:23:16] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.86, 7.97, 6.92 [02:23:25] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 10397 MB (13% inode=98%); [02:23:31] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.86, 6.72, 6.24 [02:24:03] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.77, 8.06, 7.62 [02:25:15] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.57, 7.39, 6.84 [02:25:30] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.52, 6.31, 6.15 [02:31:11] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.37, 6.39, 6.65 [02:35:07] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.85, 7.00, 6.87 [02:37:05] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.86, 6.31, 6.64 [02:42:48] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [02:42:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:43:49] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:43:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [02:44:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [02:44:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:45:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:45:52] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.04, 7.40, 7.96 [02:45:58] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:48:03] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.66, 8.25, 7.32 [02:50:01] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.98, 7.47, 7.15 [02:50:55] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [02:50:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:51:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:51:50] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.83, 7.68, 7.86 [02:51:51] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [02:52:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:52:54] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:53:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:53:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:53:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.78, 7.15, 7.63 [02:57:54] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.28, 8.23, 7.93 [02:59:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.86, 7.24, 7.60 [03:00:05] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.59, 5.99, 6.68 [03:11:54] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.85, 5.76, 6.65 [03:38:47] PROBLEM - mw1 Disk Space on mw1 is CRITICAL: DISK CRITICAL - free space: / 4600 MB (5% inode=98%); [03:43:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:43:59] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [03:45:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:45:53] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [03:55:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [03:55:28] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.24, 7.15, 6.68 [03:59:04] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:59:25] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.42, 6.43, 6.50 [04:07:32] PROBLEM - mw1 Disk Space on mw1 is WARNING: DISK WARNING - free space: / 6528 MB (8% inode=98%); [04:08:03] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.91, 5.95, 5.43 [04:09:34] RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 11259 MB (14% inode=98%); [04:12:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.25, 6.68, 5.87 [04:35:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEQD [04:35:10] [02miraheze/services] 07MirahezeSSLBot 03baf21d1 - BOT: Updating services config for wikis [05:13:52] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.94, 7.17, 6.29 [05:17:54] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.73, 6.46, 6.19 [05:52:19] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.46, 2.41, 1.71 [05:53:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.77, 6.44, 5.79 [05:54:14] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.59, 2.33, 1.76 [05:55:54] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.85, 6.12, 5.75 [06:22:54] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.99, 7.04, 6.58 [06:26:16] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2867 MB (11% inode=94%); [06:32:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.82, 7.57, 6.96 [06:33:03] Reception123: we're up from what I can see, topic change? [06:33:42] yup [06:34:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.71, 7.54, 7.02 [06:35:31] RhinosF1: is it me or is everything pretty slow? [06:36:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.78, 8.52, 7.42 [06:37:52] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.85, 6.49, 5.25 [06:39:52] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.92, 5.77, 5.13 [06:43:13] Reception123: slow again [06:45:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:45:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:45:53] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [06:46:28] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.42, 7.96, 7.77 [06:47:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:47:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:47:49] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [06:51:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [06:53:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:54:27] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.58, 7.73, 7.69 [06:55:34] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:56:22] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [06:56:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:56:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.60, 7.35, 7.56 [06:56:31] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.29, 3.46, 2.63 [06:58:29] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.46, 3.13, 2.61 [07:00:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:04:18] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [07:04:28] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [07:04:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:06:01] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:06:35] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 12.25, 8.62, 7.85 [07:14:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.84, 7.94, 7.86 [07:16:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 7.56, 8.01, 7.90 [07:18:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.23, 7.50, 7.73 [07:20:34] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.78, 6.89, 6.15 [07:22:31] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.34, 6.44, 6.08 [07:34:17] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.15, 5.92, 6.80 [07:43:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.13, 6.72, 6.79 [07:47:25] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.51, 6.49, 6.72 [07:53:28] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeE5a [07:53:30] [02miraheze/ssl] 07Reception123 03dd11c2d - add tep.wiki cert [07:56:11] PROBLEM - wiki.isina.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:57:59] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:58:05] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:58:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:58:10] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [07:58:39] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [08:02:14] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 9.969 second response time [08:04:17] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.812 second response time [08:07:31] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [08:07:44] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.49, 7.95, 6.99 [08:07:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:08:07] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [08:08:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [08:11:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.59, 7.38, 6.98 [08:25:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.32, 6.37, 6.74 [08:41:35] hi [08:42:26] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.90, 6.52, 6.39 [08:46:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.23, 6.66, 6.48 [08:53:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.06, 7.42, 6.85 [08:55:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.50, 6.97, 6.74 [08:57:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.49, 6.32, 6.52 [09:06:55] RECOVERY - wiki.isina.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.isina.ir' will expire on Fri 17 Jan 2020 11:24:18 AM GMT +0000. [09:18:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.86, 7.48, 6.57 [09:19:07] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.21, 6.95, 6.44 [09:21:07] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.59, 6.51, 6.34 [09:22:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.93, 6.60, 6.44 [09:22:57] anyone? [09:23:17] irc is so empty nowadays [09:33:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.83, 6.85, 6.53 [09:35:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.04, 6.09, 6.28 [10:53:01] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%); [11:45:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.97, 6.58, 5.99 [11:47:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.72, 6.31, 5.97 [12:35:09] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.82, 7.37, 6.20 [12:35:22] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.35, 6.49 [12:39:07] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.91, 7.54, 6.57 [12:41:05] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.52, 6.77, 6.51 [12:44:56] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.08, 6.81, 6.62 [12:45:07] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.90, 6.17, 6.28 [12:46:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.03, 6.58, 6.55 [12:52:41] Are you any closer to fixing the 503's [12:59:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:59:22] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [13:00:22] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.42, 6.94, 6.34 [13:00:26] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [13:00:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [13:00:51] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [13:02:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.10, 6.88, 6.64 [13:02:17] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.72, 7.00, 6.41 [13:04:21] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.79, 6.53, 6.33 [13:05:08] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [13:05:12] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.10, 3.79, 2.57 [13:06:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [13:06:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:10:18] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.70, 8.06, 7.23 [13:10:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [13:11:34] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.02, 3.48, 2.89 [13:12:40] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [13:12:41] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [13:13:00] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [13:13:02] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.67, 7.37, 6.77 [13:13:30] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.50, 3.10, 2.82 [13:13:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:14:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.24, 7.47, 7.18 [13:15:00] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.33, 6.99, 6.73 [13:15:26] BurningPrincess: apologies for the issues [13:15:55] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.44, 3.55, 2.62 [13:16:42] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [13:16:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:16:57] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.12, 6.25, 6.50 [13:17:50] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 3.13, 3.29, 2.63 [13:18:15] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.09, 7.85, 7.34 [13:18:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [13:20:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.83, 7.68, 7.33 [13:22:03] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.37, 8.04, 7.51 [13:23:57] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.86, 7.65, 7.43 [13:25:51] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.39, 7.95, 7.55 [13:34:32] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.53, 7.89, 7.03 [13:39:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.84, 7.40, 7.75 [13:41:05] RhinosF1, you are so nice and the staff here are so great [13:42:12] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.86, 7.41, 7.07 [13:47:59] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.77, 6.38, 6.75 [13:59:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.63, 5.89, 6.58 [14:04:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEAH [14:04:08] [02miraheze/mw-config] 07paladox 030bd06dd - Move staffwiki to private directory [14:08:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEAF [14:08:31] [02miraheze/mw-config] 07paladox 03eb6ee91 - Move rhinosf1wiki to private directory [14:13:12] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.20, 7.22, 6.86 [14:15:07] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.33, 7.31, 6.93 [14:17:07] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.66, 7.08, 6.52 [14:18:58] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.02, 6.69, 6.76 [14:21:08] !log depool and repool mw1 [14:21:09] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.88, 6.31, 6.36 [14:21:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:24:38] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [14:32:10] !log depool and repool mw2 [14:32:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:33:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 6.75, 6.67 [14:33:53] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [14:34:58] !log depool and repool mw3 [14:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:35:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.14, 7.13, 6.81 [14:37:08] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.78, 6.57, 6.64 [14:38:30] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [14:42:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.11, 6.75, 6.65 [14:42:58] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.33, 7.62, 6.67 [14:44:50] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.19, 6.19, 6.46 [14:44:54] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.03, 6.64, 6.42 [14:58:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.94, 6.97, 6.42 [15:00:29] !log depool mw2 [15:00:34] !log upgrade php on mw2 [15:00:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:01:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:01:47] !log upgrade openssl on mw2 [15:02:03] !log repool mw2 [15:02:09] !log depool mw3 [15:02:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:02:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:02:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:03:05] !log upgrade php and openssl on mw3 [15:03:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:04:00] !log repool mw3 [15:04:04] !log depool mw1 [15:04:05] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.19, 6.72, 6.64 [15:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:04:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:04:37] !log upgrade openssl on mw1 [15:04:47] !log upgraded nodejs on mw[123] [15:04:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:05:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:05:02] !log also upgraded php7.3 on mw[123] [15:05:14] !log repool mw1 [15:05:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:05:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:06:56] !log upgrade php, nodejs and openssl on test1 [15:07:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:09:08] !log upgrade puppet-agent on test1 [15:09:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:13:22] PROBLEM - mw1 Disk Space on mw1 is WARNING: DISK WARNING - free space: / 8214 MB (10% inode=98%); [15:17:50] !log upgrade php, nodejs and openssl on misc[124] [15:17:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:19:27] * paladox changes ^ [15:19:33] !log upgrade nodejs on misc4 [15:19:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:20:46] !log upgrade puppet-agent on misc[123] [15:20:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:20:55] !log upgrade puppet-agent on misc4 [15:21:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:23:37] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:23:37] !log upgrade systemd-sysv tzdata udev unzip libssl-dev libssl1.1 libsystemd0 on misc1 [15:23:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:24:15] !log upgrade libldap-common libsodium23 libxml2 libxslt1.1 php7.2 php7.3 autopoint base-files debian-archive-keyring gettext gettext-base on misc1 [15:24:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:24:49] !log upgrade libcupsimage2 libglib2.0-0 libglib2.0-data libicu57 libidn2-0 libldap-2.4-2 libcups2 on misc1 [15:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:25:37] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:49] !log upgrade icinga2 on misc1 [15:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:06:41] RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 32038 MB (41% inode=98%); [16:28:54] !log running ./wikibackups.sh /home/reception/allpublic.dblist in Reception123 homedir on mw1 [16:29:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:32:34] .mhca Examknow [16:32:34] https://meta.miraheze.org/wiki/Special:CentralAuth/Examknow [16:32:59] .mhca apap04 [16:32:59] https://meta.miraheze.org/wiki/Special:CentralAuth/apap04 [18:00:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeuef [18:00:14] [02miraheze/services] 07MirahezeSSLBot 0330e74ce - BOT: Updating services config for wikis [18:05:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeueT [18:05:10] [02miraheze/services] 07MirahezeSSLBot 03d1f3ad8 - BOT: Updating services config for wikis [18:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeueI [18:10:09] [02miraheze/services] 07MirahezeSSLBot 03a7f02f0 - BOT: Updating services config for wikis [18:11:14] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [18:13:14] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:15:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeue6 [18:15:10] [02miraheze/services] 07MirahezeSSLBot 033278928 - BOT: Updating services config for wikis [18:37:13] hello again! [18:37:46] I have an empty wiki but i realised i asked for the wrong name. scan it be renamed or is it better to just start with a new one? [18:47:16] No pages created? ATY [18:47:25] correct [18:50:42] Um, you can re-apply for a wiki with the correct name. And the previous one, request the deletion to a Steward. ATY [18:51:12] "request the deletion to a Steward" on phabricator? [18:52:38] on https://meta.miraheze.org/wiki/Stewards%27_noticeboard [18:52:40] [ Stewards' noticeboard - Miraheze Meta ] - meta.miraheze.org [18:52:54] will do [18:52:56] thank you! [18:53:44] :) [19:16:42] !log upgraded mariadb to 10.2 on misc1 [19:16:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:18:20] revi: https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=private.revi.wiki%20-%20Comodo [19:18:21] [ Icinga Web 2 Login ] - icinga.miraheze.org [19:18:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeuvn [19:18:30] Guest/Guest [19:18:31] [02miraheze/puppet] 07paladox 03605c9d8 - monitoring: Install mariadb-client-10.2 [19:25:02] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.88, 6.50, 5.82 [19:27:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.98, 6.98, 6.41 [19:30:57] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.76, 6.34, 5.98 [19:33:05] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.63, 6.25, 6.32 [20:38:13] !log depool and repool mw1 [20:38:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:41:01] paladox: should status be up now? [20:41:10] yup [20:42:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.47, 6.92, 6.37 [20:43:13] paladox: never all caps [20:43:22] ? [20:44:47] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.62, 6.71, 6.35 [20:46:24] paladox: topic [20:46:36] RhinosF1 why not? [20:46:48] paladox: doesn’t look nice [20:47:01] Dosen't it? Looked fine to me [20:48:26] paladox: stands out too much [20:48:37] Also we always format that way [20:48:44] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.27, 7.60, 6.74 [20:49:26] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:49:55] all over capitalization? sheesh, you guys :P [20:52:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.81, 7.78, 7.02 [20:52:44] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [20:52:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeufS [20:52:47] [02miraheze/puppet] 07paladox 03b133ae8 - Update php_fpm.pp [20:52:49] [02puppet] 07paladox opened pull request 03#1123: Update php_fpm.pp - 13https://git.io/Jeuf9 [20:53:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeufH [20:53:23] [02miraheze/puppet] 07paladox 03cdc5230 - Update php.pp [20:53:24] [02puppet] 07paladox synchronize pull request 03#1123: Update php_fpm.pp - 13https://git.io/Jeuf9 [20:53:41] [02puppet] 07paladox edited pull request 03#1123: Allow setting min value for emergency_restart_threshold - 13https://git.io/Jeuf9 [20:53:44] [02puppet] 07paladox closed pull request 03#1123: Allow setting min value for emergency_restart_threshold - 13https://git.io/Jeuf9 [20:53:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JeufQ [20:53:47] [02miraheze/puppet] 07paladox 03dda749e - Allow setting min value for emergency_restart_threshold (#1123) * Update php_fpm.pp * Update php.pp [20:54:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [20:54:44] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.11, 8.17, 7.24 [20:54:44] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [20:56:22] !log depool and repool mw[123] [20:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:57:06] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:58:40] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 13.03, 8.82, 6.47 [20:58:57] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [21:00:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.30, 7.82, 7.42 [21:10:24] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.52, 7.21, 7.12 [21:16:11] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.33, 5.75, 6.52 [21:24:10] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 11.18, 8.08, 7.12 [21:26:09] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.63, 7.76, 7.12 [21:28:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.08, 6.32, 6.78 [21:32:09] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.23, 7.73, 7.31 [21:34:08] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.38, 6.82, 7.03 [21:38:08] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.32, 5.66, 6.52 [21:42:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.68, 6.59, 6.42 [21:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuJg [21:45:13] [02miraheze/services] 07MirahezeSSLBot 031d8e1dc - BOT: Updating services config for wikis [21:48:08] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.23, 6.96, 6.74 [21:48:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.58, 6.70, 6.62 [21:50:10] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.23, 6.60, 6.63 [21:54:08] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.20, 7.06, 6.80 [21:56:10] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.57, 6.60, 6.67 [22:03:05] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.59, 7.70, 7.05 [22:06:59] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.12, 7.82, 7.27 [22:16:48] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.92, 6.23, 6.79 [22:26:35] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.25, 7.14, 6.69 [22:28:31] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.28, 7.13, 6.74 [22:32:26] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.47, 6.78, 6.68 [22:36:25] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.99, 6.89, 6.79 [22:38:21] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.36, 6.62, 6.69 [22:48:08] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.95, 7.81, 7.08 [22:50:09] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.86, 7.26, 6.96 [22:52:21] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:25] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 1 minute ago with 17 failures. Failed resources (up to 3 shown) [22:52:28] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:37] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:48] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:53] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:54] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:56] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:57] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:59] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:06] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:07] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:17] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:20] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:47] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:55] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:54:07] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:54:16] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:54:22] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:58:12] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.18, 7.70, 7.21 [23:03:45] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:03:55] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:04:08] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:04:17] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:04:22] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:04:25] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:04:26] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:04:42] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:04:47] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:04:54] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:04:59] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:05:03] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:05:04] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:05:07] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:05:10] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:05:10] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:05:16] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:06:30] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:07:15] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:10:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.44, 6.88, 6.56 [23:12:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.74, 6.55, 6.48 [23:20:36] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:22:35] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4217 files, 8.761MB, 2019-10-15 18:59:00 (1.7 weeks ago) [23:37:17] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [23:37:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeuTo [23:37:20] [02miraheze/puppet] 07paladox 0317a0b5e - Send thumbnail generation requests to mw2 (also send thumbernails to mw2) [23:37:22] [02puppet] 07paladox opened pull request 03#1124: Send thumbnail generation requests to mw2 (also send thumbernails to … - 13https://git.io/JeuTK [23:40:46] [02puppet] 07paladox edited pull request 03#1124: Send thumbnail generation requests to mw2 (also send thumbernails to mw2) - 13https://git.io/JeuTK [23:40:50] [02puppet] 07paladox edited pull request 03#1124: Send thumbnail generation requests to mw2 (also send thumbernails to mw2) - 13https://git.io/JeuTK [23:42:50] [02puppet] 07paladox synchronize pull request 03#1124: Send thumbnail generation requests to mw2 (also send thumbernails to mw2) - 13https://git.io/JeuTK [23:42:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeuTS [23:42:53] [02miraheze/puppet] 07paladox 03a2428b4 - Update default.vcl [23:43:22] [02puppet] 07paladox synchronize pull request 03#1124: Send thumbnail generation requests to mw2 (also send thumbernails to mw2) - 13https://git.io/JeuTK [23:43:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeuT9 [23:43:25] [02miraheze/puppet] 07paladox 03f3702c6 - Update default.vcl [23:52:34] [02puppet] 07paladox closed pull request 03#1124: Send thumbnail generation requests to mw2 (also send thumbernails to mw2) - 13https://git.io/JeuTK [23:52:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuTd [23:52:37] [02miraheze/puppet] 07paladox 03659a0ee - Send thumbnail generation requests to mw2 (also send thumbernails to mw2) (#1124) * Send thumbnail generation requests to mw2 (also send thumbernails to mw2) * Update default.vcl * Update default.vcl [23:52:39] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [23:52:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [23:53:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.00, 6.69, 6.14 [23:55:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.23, 6.50, 6.13 [23:57:17] !log depool and repool mw1 [23:57:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master