[00:12:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.73, 6.82, 6.37 [00:13:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [00:14:29] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.35, 5.93, 6.14 [00:15:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [00:21:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [00:33:41] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:45:03] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [00:45:35] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [00:53:04] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:53:33] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:15:30] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-443] [01:33:37] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:56:42] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.51, 6.97, 6.10 [02:28:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.81, 6.92, 6.86 [02:35:23] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:36:08] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.48, 6.34, 6.80 [02:47:48] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:15:36] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [03:23:42] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:23:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [03:24:13] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:33:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:35:37] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [03:36:35] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:43:40] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:53:41] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [04:15:57] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:43:44] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:56:09] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:15:59] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [05:22:07] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:25:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-redis] [05:34:42] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:47:05] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:04:08] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:16:49] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:45:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.87, 6.92, 6.44 [06:47:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.72, 6.56, 6.39 [06:54:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [07:02:00] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:02:17] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.75, 7.07, 6.54 [07:04:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.49, 8.22, 7.11 [07:06:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.30, 7.74, 7.17 [07:10:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.18, 6.15, 6.70 [07:12:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:24:58] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [07:27:19] OOM [07:53:37] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [08:04:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.95, 7.04, 6.19 [08:08:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.18, 6.37, 6.25 [08:23:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [08:26:03] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:31:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [08:43:07] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [08:57:35] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 4 minutes ago with 12 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9253] [09:03:38] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [09:13:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [09:21:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [09:35:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [10:24:00] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [10:31:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [10:34:32] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:43:59] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [10:47:00] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:52:00] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [11:02:48] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.30, 6.57, 5.87 [11:04:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.10, 6.31, 5.97 [11:12:42] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [11:13:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [11:21:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [11:23:21] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [11:25:04] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:44:00] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [11:51:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [11:53:05] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [12:03:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [12:11:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [12:12:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.11, 7.09, 6.21 [12:14:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.58, 7.08, 6.42 [12:16:34] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.15, 6.41, 6.32 [12:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeX8M [12:55:09] [02miraheze/services] 07MirahezeSSLBot 03f3c4ff3 - BOT: Updating services config for wikis [13:34:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.00, 6.62, 6.15 [13:36:54] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.24, 6.04, 6.03 [13:54:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [14:11:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:32:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.84, 6.87, 6.51 [14:33:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [14:34:08] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.89, 6.72, 6.56 [14:51:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [14:54:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.55, 6.92, 6.44 [14:58:08] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.30, 6.65, 6.51 [15:54:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [16:01:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:23:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [16:24:07] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 6.88, 6.42 [16:26:07] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.27, 6.20, 6.27 [16:31:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:44:58] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.61, 6.86, 6.58 [16:46:55] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.27, 6.61, 6.56 [16:52:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.82, 6.92, 6.52 [16:56:32] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.96, 6.38, 6.48 [17:02:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.68, 7.32, 6.81 [17:24:08] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.45, 6.53, 6.75 [17:45:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.88, 7.19, 6.82 [17:52:48] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.19, 7.52, 7.09 [17:54:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.28, 6.81, 6.89 [18:10:10] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.03, 7.48, 7.15 [18:18:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.16, 7.70, 7.68 [18:22:10] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.82, 8.05, 7.80 [18:24:07] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.48, 7.91, 7.82 [18:26:07] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 7.79, 7.77 [18:28:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.52, 7.49, 7.65 [18:30:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.57, 8.04, 7.84 [18:32:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:32:43] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:33:19] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:19] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [18:34:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:34:42] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.481 second response time [18:35:22] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.959 second response time [18:36:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.77, 7.93, 7.94 [18:40:07] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.70, 8.11, 8.02 [18:42:05] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:42:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.77, 8.00, 8.00 [18:44:08] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 7.89, 8.13, 8.07 [18:46:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.69, 6.86, 7.56 [18:54:08] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.34, 8.05, 7.68 [18:56:07] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.37, 7.83, 7.68 [19:00:08] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.64, 7.79, 7.65 [19:03:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:06:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.06, 7.84, 7.87 [19:08:07] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.27, 8.15, 7.99 [19:09:28] !log restart php7.3-fpm on mw3 [19:09:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeXa6 [19:10:09] [02miraheze/services] 07MirahezeSSLBot 0381b2887 - BOT: Updating services config for wikis [19:11:31] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [19:11:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:12:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.14, 7.13, 7.55 [19:14:08] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.48, 7.34, 7.49 [19:21:34] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeXap [19:21:35] [02miraheze/ssl] 07Reception123 03dc6f4b2 - add wiki.worldsofweary.com cert [19:33:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [19:41:38] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:44:32] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.28, 6.26, 7.65 [19:48:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.18, 5.25, 6.67 [19:52:46] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.31, 7.58, 7.12 [19:55:56] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_checkout_MediaWiki core],Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80] [20:05:32] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [20:13:28] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:17:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.71, 7.22, 7.93 [20:23:26] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:32] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53] [20:23:33] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.02, 8.10, 8.07 [20:27:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:29:41] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:33:26] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeXwf [20:35:09] [02miraheze/services] 07MirahezeSSLBot 03eb1c5ab - BOT: Updating services config for wikis [20:42:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [20:42:38] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:44:36] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 4.129 second response time [20:45:20] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [20:47:02] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [20:50:07] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.08, 6.61, 5.78 [20:50:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:52:06] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.22, 6.49, 5.92 [20:52:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:53:20] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:12] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [21:00:42] PROBLEM - lizardfs6 Puppet on lizardfs6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [21:05:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [21:13:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:53:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [22:03:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:15:56] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.34, 7.38, 8.00 [22:25:13] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [22:25:50] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.92, 8.29, 7.92 [22:27:49] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.77, 7.59, 7.78 [22:29:48] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.55, 8.20, 7.95 [22:31:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:35:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.55, 7.70, 7.95 [22:41:41] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.32, 7.86, 7.86 [22:53:15] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [23:03:14] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:18:43] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.98, 7.05, 6.77 [23:24:25] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.80, 6.71, 6.77 [23:24:36] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:56] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:26:01] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:26:25] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:40:02] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.44, 6.51, 7.85 [23:55:11] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [23:58:08] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.46, 6.27, 6.75