[05:14:18] Hello sparr_! If you have any questions, feel free to ask and someone should answer soon. [06:26:46] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2965 MB (12% inode=94%); [09:11:35] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [09:13:12] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [09:13:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [09:13:35] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18698 bytes in 0.264 second response time [09:15:05] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [09:15:11] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [09:15:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:58:59] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [11:43:45] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CRITICAL - load average: 2.66, 1.83, 0.95 [11:45:44] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 0.63, 1.68, 1.14 [13:44:26] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CRITICAL - load average: 3.05, 1.93, 1.06 [13:46:26] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 0.51, 1.03, 0.88 [14:50:52] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:49] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:06:59] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:07:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:07:22] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [16:08:02] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:08:25] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [16:08:41] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [16:11:50] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1352 bytes in 0.121 second response time [16:12:07] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15295 bytes in 0.005 second response time [16:12:30] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [16:12:45] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [16:13:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:13:33] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [16:13:50] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18697 bytes in 0.788 second response time [16:14:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:27:26] * hispano76 greetings [19:32:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:34:34] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:41:45] !log unlocked phab auth config (temp) [20:41:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:47:28] !log locked phab auth config [20:47:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:51:05] Hey SPF|Cloud [20:53:47] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.95, 3.27, 1.73 [20:55:45] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.34, 1.61, 1.39 [23:14:51] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-redis] [23:23:10] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures