[00:05:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [00:23:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:05:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [01:13:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:22:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.06, 6.63, 6.37 [02:24:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.41, 7.57, 6.81 [02:30:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.13, 7.77, 7.39 [02:42:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.14, 7.17, 6.99 [02:44:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.38, 7.15, 7.02 [02:58:30] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.09, 6.03, 6.61 [03:15:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [03:43:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:05:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [04:13:13] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:05:11] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [05:23:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:05:11] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [06:07:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.24, 6.52, 5.80 [06:09:45] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.73, 6.20, 5.84 [06:13:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:16:01] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/Je1YG [06:16:02] [02miraheze/ssl] 07Reception123 03ada4bd5 - add www.portalsofphereon.com cert [06:16:04] [02miraheze/ssl] 07Reception123 03e612966 - add wiki.kujum.org cert [06:44:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [06:51:49] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:03:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.92, 6.97, 6.50 [07:07:44] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.52, 6.65, 6.55 [07:20:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1Oq [07:20:10] [02miraheze/services] 07MirahezeSSLBot 03ba01a07 - BOT: Updating services config for wikis [08:15:26] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [08:23:27] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:24:03] PROBLEM - tep.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:01] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Sat 25 Jan 2020 06:51:18 AM GMT +0000. [08:35:21] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-udp-from-any-to-any-port-53] [08:41:02] PROBLEM - tep.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:01] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Sat 25 Jan 2020 06:51:18 AM GMT +0000. [08:43:20] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:59:58] PROBLEM - tep.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Je1st [09:04:58] [02miraheze/puppet] 07paladox 0368fe567 - Varnish: Fix regex [09:05:00] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [09:05:01] [02puppet] 07paladox opened pull request 03#1155: Varnish: Fix regex - 13https://git.io/Je1sq [09:10:43] [02puppet] 07paladox closed pull request 03#1155: Varnish: Fix regex - 13https://git.io/Je1sq [09:10:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1ss [09:10:46] [02miraheze/puppet] 07paladox 03d1e447b - Varnish: Fix regex (#1155) [09:10:48] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [09:10:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [09:33:28] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 1 minute ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [09:49:00] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Sat 25 Jan 2020 06:51:18 AM GMT +0000. [09:53:32] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:53:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.22, 6.85, 6.35 [09:55:45] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.81, 6.38, 6.28 [10:11:46] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.01, 7.26, 6.49 [10:13:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.33, 7.35, 6.70 [10:15:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.89, 6.74, 6.61 [10:35:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1GQ [10:35:09] [02miraheze/services] 07MirahezeSSLBot 0368650b7 - BOT: Updating services config for wikis [10:35:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [10:38:51] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:41:07] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.57, 0.90, 0.81 [10:43:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:45:23] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [11:53:22] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:35:20] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [12:43:20] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:55:20] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [13:03:20] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:03:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.24, 7.06, 6.27 [14:05:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.59, 6.48, 6.23 [14:45:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.06, 6.84, 6.49 [14:47:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.57, 6.30, 6.35 [15:03:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.29, 6.85, 6.43 [15:05:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.16, 6.24, 6.29 [15:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je18G [15:10:09] [02miraheze/services] 07MirahezeSSLBot 03b3c12fa - BOT: Updating services config for wikis [15:15:33] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 4 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [15:43:21] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.98, 6.96, 6.49 [15:54:14] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 16 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d] [15:54:58] Hello Heca! If you have any questions, feel free to ask and someone should answer soon. [15:55:37] Hi, I have been aways for a while and I cannot access my https://programming.red/ wiki. It seems it is not here anymore ! [15:56:00] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.23, 6.07, 6.31 [15:59:12] Could you please help ? [15:59:18] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:01:24] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.35, 0.31, 0.27 [16:02:15] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:05:44] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:44] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.06, 0.20, 0.24 [16:31:32] PROBLEM - wiki.tallguysfree.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tallguysfree.com' expires in 15 day(s) (Sun 15 Dec 2019 04:28:05 PM GMT +0000). [16:31:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.17, 6.46, 6.08 [16:31:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je14D [16:31:46] [02miraheze/ssl] 07MirahezeSSLBot 0369bd572 - Bot: Update SSL cert for wiki.tallguysfree.com [16:35:23] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [16:35:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.29, 6.24, 6.17 [16:43:22] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:30] RECOVERY - wiki.tallguysfree.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tallguysfree.com' will expire on Thu 27 Feb 2020 03:31:39 PM GMT +0000. [16:55:12] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.74, 6.90, 6.70 [17:02:49] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.02, 7.52, 7.13 [17:04:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.18, 7.16, 7.09 [17:08:33] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.04, 6.43, 6.78 [17:20:05] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.16, 6.76, 6.57 [17:25:20] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [17:25:54] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.38, 6.61, 6.69 [17:33:24] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:03:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.10, 6.94, 6.53 [18:05:22] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [18:07:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.42, 7.64, 6.99 [18:13:32] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:19:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.63, 7.57, 7.79 [18:21:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.13, 7.88, 7.86 [18:23:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.96, 7.58, 7.76 [18:27:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.05, 7.77, 7.76 [18:29:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.59, 7.70, 7.74 [18:31:48] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.20, 8.03, 7.87 [18:32:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [18:33:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.43, 7.66, 7.79 [18:34:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:37:45] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.39, 7.86, 7.76 [18:45:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.47, 7.58, 7.78 [18:47:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.40, 7.89, 7.84 [18:49:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.11, 7.57, 7.73 [18:51:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.72, 8.12, 7.90 [18:53:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [18:55:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [18:55:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.66, 7.92, 7.93 [18:57:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:57:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:01:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.21, 8.10, 7.97 [19:03:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.92, 7.57, 7.79 [19:07:47] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.49, 8.06, 7.90 [19:09:47] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.02, 7.66, 7.79 [19:11:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.35, 8.48, 8.08 [19:21:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.45, 7.58, 7.93 [19:22:29] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [19:22:51] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:32] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 4.314 second response time [19:24:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:24:49] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.712 second response time [19:31:48] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.93, 7.84, 7.68 [19:33:48] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.77, 7.80, 7.69 [19:38:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.53, 7.98, 7.76 [19:41:15] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:41:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [19:41:59] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:15] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 2.036 second response time [19:43:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:43:57] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.534 second response time [19:44:22] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:44:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.29, 7.84, 7.88 [19:48:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.36, 7.61, 7.69 [19:50:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.34, 7.17, 7.48 [19:52:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:52:11] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.50, 7.63, 7.58 [19:59:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.95, 7.85, 7.79 [20:01:53] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.50, 8.07, 7.88 [20:02:29] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:02:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:02:48] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:03:02] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:04:07] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [20:04:27] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.633 second response time [20:04:34] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1ui [20:04:35] [02miraheze/ssl] 07Reception123 037596715 - www.portalsofphereon.com redirect [20:04:44] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:04:48] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.768 second response time [20:04:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [20:05:07] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.857 second response time [20:05:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.23, 7.52, 7.78 [20:06:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:11:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:15:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [20:15:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:17:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:17:50] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.14, 6.09, 6.68 [20:19:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:23:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1zJ [20:23:37] [02miraheze/puppet] 07paladox 0359aee3e - matomo: Update to 3.13.0 [20:23:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:24:50] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.05, 7.17, 6.97 [20:27:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:28:53] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.391 second response time [20:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1zG [20:35:09] [02miraheze/services] 07MirahezeSSLBot 032a7689f - BOT: Updating services config for wikis [20:35:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.65, 6.54, 6.75 [20:35:25] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_matomo] [20:41:42] [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_33 [+0/-0/±3] 13https://git.io/Je1zR [20:41:44] [02miraheze/mediawiki] 07Pix1234 03488cee7 - Update various ext from upstream [20:45:04] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.03, 6.83, 6.36 [20:47:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:48:58] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.17, 6.30, 6.40 [20:49:12] !log rebuild LC on mw* [20:49:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:51:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:52:28] Zppix: LFS6? [20:53:55] !log rebuild lc on lizardfs6 [20:54:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:55:49] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:02:09] !log root@misc2:/srv/matomo# sudo service php7.3-fpm restart [21:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:08:42] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:08:46] PROBLEM - misc3 SSH on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:56] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:10:38] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.18, 0.28, 0.27 [21:10:42] RECOVERY - misc3 SSH on misc3 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u1 (protocol 2.0) [21:10:51] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [21:25:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [21:33:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:33:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 8.00, 7.66, 6.94 [21:34:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [21:34:07] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:31] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.13, 6.43, 6.61 [21:36:07] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.503 second response time [21:38:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:41:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.84, 6.80, 6.68 [21:43:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.31, 7.59, 7.02 [21:45:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.28, 7.34, 7.04 [21:49:00] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.40, 6.30, 6.67 [21:52:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.48, 7.24, 6.93 [22:01:09] !log depool mw2 [22:01:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:01:41] !log upgrade php on mw2 to 7.3.12 [22:01:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:02:34] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.21, 6.60, 6.78 [22:02:41] !log repool mw2 [22:02:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:13:45] Reception123, Zppix: replied [22:14:55] RhinosF1: hello, how are you? [22:15:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-udp-from-any-to-any-port-53] [22:15:37] hispano76: not too good tbh, see community Noticeboard [22:16:48] RhinosF1: sorry :( [22:17:36] Thanks [22:17:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je12R [22:17:39] [02miraheze/puppet] 07paladox 03705d8bd - php: Only install php-mail-mime on php 7.2 [22:17:44] * RhinosF1 is off to sleep [22:18:44] All right [22:21:21] Um, is that last one understood or is that a wrong phrase? "All right" equals "okay."? [22:21:45] or in spanish "Esta bien" [22:22:16] hispano76: due to the context, I got it :) [22:22:31] ok [22:22:43] thanks SPF|Cloud :) [22:22:54] I guess you meant to say 'alright', I would read 'all right' as 'all is fine' [22:22:55] no problem ;) [22:23:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:25:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.68, 7.34, 6.89 [22:35:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [22:46:26] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.42, 6.60, 6.76 [22:52:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.18, 7.28, 6.88 [22:53:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:54:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.64, 7.08, 6.91 [22:59:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:00:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.46, 7.58, 7.24 [23:01:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:04:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.06, 6.83, 7.17 [23:05:11] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [23:06:30] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:30] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.64, 7.77, 7.47 [23:08:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.03, 7.28, 7.33 [23:10:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 7.48, 7.39 [23:14:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.51, 7.72, 7.57 [23:23:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:24:10] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:24:44] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:25:08] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:12] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.343 second response time [23:26:52] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 1.598 second response time [23:29:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:32:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.06, 7.27, 7.23 [23:34:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.81, 6.67, 7.01 [23:38:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.93, 6.13, 6.61 [23:44:55] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [23:52:46] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures