[00:06:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [00:09:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:06:59] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [01:07:03] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [01:09:39] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:09:42] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:14:53] !log restart lizardfs-master [01:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:27:33] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3014 MB (12% inode=94%); [10:44:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [10:47:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:16:18] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.81, 7.81, 5.13 [13:24:04] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.77, 6.40, 5.73 [13:24:34] [ANNOUNCEMENT] Just a reminder, Channel Founders and/or operators ZppixBot's channelmgnt module (Channel Management) is only usable by people listed as a chanop in ZppixBot's code, please see https://git.io/JecEc or if you need assistance with adding current/new channel operators please contact Zppix or join #ZppixBot. Thank you! [14:56:12] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2641 MB (10% inode=94%); [15:17:54] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 5 minutes ago with 8 failures. Failed resources (up to 3 shown) [15:18:24] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 5 minutes ago with 8 failures. Failed resources (up to 3 shown) [15:18:39] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 161 failures. Last run 5 minutes ago with 161 failures. Failed resources (up to 3 shown): File[wiki.svenskabriardklubben.se_private],File[www.schulwiki.de],File[www.schulwiki.de_private],File[holonet.pw] [15:18:53] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 5 minutes ago with 8 failures. Failed resources (up to 3 shown) [15:19:12] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 6 minutes ago with 8 failures. Failed resources (up to 3 shown) [15:19:57] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 166 failures. Last run 6 minutes ago with 166 failures. Failed resources (up to 3 shown) [15:24:24] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:24:46] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:25:00] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:47] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:26:33] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:27:13] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:48:17] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [15:51:05] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:54:05] pages loading really slow [19:00:24] should recover now.