[00:11:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [00:23:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server] [00:41:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:53:48] <𝐂.𝐒𝐲𝐝𝐞> ← CSyde65 has left (Quit: I guess I'll leave since there hasn't been any discussion in the last couple of hours.) [01:03:44] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:05:50] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 3.094 second response time [01:15:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server] [01:52:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:54:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.08, 7.11, 6.59 [01:56:34] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.76, 6.76, 6.54 [02:06:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.43, 7.11, 6.77 [02:06:51] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:08:52] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.499 second response time [02:10:27] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.67, 6.66, 6.61 [02:26:01] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.54, 7.88, 7.31 [02:26:37] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:27:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [02:27:35] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:18] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:30:52] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 4.100 second response time [02:31:55] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 6.485 second response time [02:31:56] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.58, 7.92, 7.60 [02:32:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [02:33:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:34:26] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [02:34:52] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.516 second response time [02:35:49] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:36:04] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.79, 7.19, 7.52 [02:36:48] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:37:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:37:57] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.901 second response time [02:38:34] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [02:38:56] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 1.292 second response time [02:39:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:39:50] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:43:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [02:48:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:50:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.64, 6.65, 6.76 [03:03:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.73, 6.94, 6.76 [03:05:10] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.95, 6.30, 6.53 [03:12:55] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:34:15] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter] [03:51:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:13:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [06:21:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:27:39] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2659 MB (11% inode=94%); [06:33:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [06:39:39] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [06:54:07] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.59, 6.82, 6.04 [06:56:07] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.50, 6.50, 6.08 [07:12:20] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:24:14] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [07:41:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [08:05:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server] [08:21:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:33:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 2 minutes ago with 16 failures. Failed resources (up to 3 shown): Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client] [09:01:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [09:14:00] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [09:41:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:53:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [10:03:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:14:00] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[exim4] [10:23:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:03:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Service[ssh],Service[postfix],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [11:11:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [11:27:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [11:29:19] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:43:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [12:51:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [12:55:51] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:57:48] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2358 MB (9% inode=94%); [13:03:59] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [13:21:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:03:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [15:11:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:26:40] revi: are you around to handle a few wiki requests (if i dont hear from you in about half hour ill just google translate it) [15:26:46] a few = 2 [15:27:33] Zppix: are you around to help helper with preloader on phab [15:27:36] or LakesideMiners [15:27:43] RhinosF1: I can whats up [15:27:57] How can I help? [15:28:11] RhinosF1: what do i need to do? [15:28:19] Zppix, LakesideMiners: https://phabricator.miraheze.org/T4791#93808 [15:28:20] [ ⚓ T4791 Extension review: BoilerRoom ] - phabricator.miraheze.org [15:28:29] I got it i think LakesideMiners [15:29:53] Ah, yah. I don’t have a lot of experience with extensions. Might try to set up a mediawiki install on my laptop to test things. [15:30:04] LakesideMiners: I got it this is a very simple question imo [15:33:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100],Exec[git_pull_dns] [15:34:21] RhinosF1: did my reply to him make sense? [15:35:55] RhinosF1: also keep in mind Preloader requires enabling by an sysadmin [15:36:14] Zppix: ok [15:36:20] why? [15:37:39] RhinosF1: because it requires config in the GH repo unless they dont mind using what we have set as default [15:38:02] RhinosF1: paladox told me to keep it as managewiki-restricted [15:38:39] Zppix: ok [15:41:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:43:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.19, 6.77, 6.15 [15:45:34] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.87, 6.59, 6.23 [15:49:26] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.95, 6.98, 6.59 [15:53:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [16:02:54] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.01, 7.28, 7.05 [16:04:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.16, 7.27, 7.10 [16:21:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:28:07] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.03, 6.44, 6.72 [17:21:21] * hispano76 greetings [17:22:42] hispano76: Hello [17:33:59] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [17:43:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:54:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [18:01:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [18:08:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.60, 7.17, 6.69 [18:10:27] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.17, 6.79, 6.67 [18:12:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [18:12:46] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:14:45] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.391 second response time [18:23:02] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.45, 7.78, 7.17 [18:23:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [18:32:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.41, 7.52, 7.16 [18:34:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.78, 6.94, 7.04 [18:41:42] Examknow: fixed next time just ping me rather than go through on wiki, it's politer when I'm online [18:41:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:48:12] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.54, 7.82, 7.36 [18:53:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [18:54:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [18:55:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:56:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.91, 7.49, 7.77 [18:56:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:57:01] RhinosF1: Okay sorry [19:04:24] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [19:04:29] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.70, 5.77, 6.60 [19:05:14] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:33] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:51] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:01] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [19:06:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [19:06:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [19:07:11] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Sat 18 Jan 2020 05:39:44 AM GMT +0000. [19:07:28] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [19:07:46] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 0.25, 0.20, 0.20 [19:07:57] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [19:08:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:08:01] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:11:21] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[monarchists.wiki_private] [19:12:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:13:09] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.08, 7.62, 7.03 [19:15:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.57, 7.61, 7.17 [19:15:15] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:22:50] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.43, 7.69, 7.37 [19:24:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.20, 7.65, 7.44 [19:30:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:32:48] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.61, 6.24, 6.68 [19:33:16] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:33:18] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:34:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:35:15] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.391 second response time [19:35:17] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 1.052 second response time [19:35:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:43:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [19:51:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:14:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [20:31:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:43:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [21:01:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [21:43:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [21:53:20] .status mhmeta Off [21:53:25] RhinosF1: Updating User:RhinosF1/Status to Off! [21:53:33] RhinosF1: Updated! [22:00:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeXLT [22:00:10] [02miraheze/services] 07MirahezeSSLBot 0394d9d27 - BOT: Updating services config for wikis [22:01:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:23:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [22:31:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:04:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [23:05:47] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [23:12:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:13:58] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40287 MB (10% inode=96%); [23:24:05] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures