[00:06:09] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:08:07] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:18] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4315 files, 8.996MB, 2019-11-24 00:05:00 (1.1 weeks ago) [00:08:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:14:30] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.665 second response time [00:14:49] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.390 second response time [00:15:44] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [00:21:42] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:33:23] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [00:35:22] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 1.585 second response time [00:35:47] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:12:52] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:48] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.76, 6.87, 6.63 [01:14:51] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 3.522 second response time [01:16:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.55, 6.60, 6.58 [01:25:07] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [01:26:10] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9253] [01:32:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [01:33:00] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:34:30] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:34:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:42:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.25, 7.33, 6.81 [01:44:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.06, 6.35, 6.60 [01:44:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [01:52:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.60, 7.02, 6.74 [01:52:43] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [01:54:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.13, 6.57, 6.67 [01:56:34] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [02:18:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [02:20:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:23:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:23:30] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.84, 7.94, 7.11 [02:23:44] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:25:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:25:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.42, 6.87, 6.86 [02:27:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.01, 7.19, 6.97 [02:29:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.36, 7.09 [02:36:07] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[git_checkout_3d2png],Exec[git_pull_MediaWiki config],Exec[git_checkout_MediaWiki core],Exec[git_checkout_landing] [02:42:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [02:42:59] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.46, 7.76, 7.27 [02:43:38] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.24, 6.91, 7.08 [02:45:39] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.636 second response time [02:46:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:46:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.54, 6.08, 6.71 [02:52:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.43, 6.94, 6.71 [02:54:39] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [03:06:39] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.83, 6.09, 6.63 [03:12:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:24:49] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [03:26:04] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 6.98, 6.64 [03:28:00] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.22, 6.64, 6.58 [03:32:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:43:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.86, 7.23, 6.64 [03:47:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.08, 6.23, 6.40 [03:50:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMnG [03:50:09] [02miraheze/services] 07MirahezeSSLBot 0394425b5 - BOT: Updating services config for wikis [04:04:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [04:22:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:34:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [04:52:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:04:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [05:12:48] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [05:33:12] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.71, 6.83, 5.99 [05:35:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.75, 6.71, 6.12 [05:44:39] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [05:52:36] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:04:46] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [06:12:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:15:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.76, 7.22, 6.32 [06:17:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.87, 6.58, 6.27 [06:21:03] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 6.98, 6.60 [06:22:58] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.29, 6.78, 6.61 [06:23:35] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:35] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [06:35:52] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [06:42:34] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:46:59] Hello Kanik! If you have any questions, feel free to ask and someone should answer soon. [06:54:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [07:02:13] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.69, 7.73, 6.92 [07:02:21] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:02:48] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:03:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:04:17] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.823 second response time [07:05:14] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [07:08:04] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.76, 7.36, 7.33 [07:11:56] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.72, 5.98, 6.76 [07:14:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [07:21:54] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:22:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:34:39] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-allow-tcp-from-any-to-any-port-22] [07:36:07] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown): Package[apt-listchanges],Cron[puppet-agent],Service[puppet],Exec[ufw-default-deny] [07:42:37] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:54:33] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client] [08:02:34] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [08:13:12] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:14:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:25:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 6.84, 6.22 [08:29:02] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.62, 6.03, 6.16 [08:36:36] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JeMlq [08:36:37] [02miraheze/CreateWiki] 07translatewiki 039070cb4 - Localisation updates from https://translatewiki.net. [08:36:38] [ Main page - translatewiki.net ] - translatewiki.net. [08:42:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:52:22] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.22, 7.63, 6.77 [08:53:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [08:54:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.90, 7.69, 7.00 [08:54:55] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [08:55:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [08:56:31] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.10, 6.70, 6.79 [09:02:43] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [09:04:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.22, 7.16, 6.62 [09:06:10] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.60, 6.69, 6.54 [09:12:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.98, 7.20, 6.83 [09:12:50] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:13:20] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:14:45] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.392 second response time [09:15:23] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.657 second response time [09:17:58] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.63, 5.49, 6.30 [09:24:44] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client] [09:32:42] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [09:35:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.59, 6.95, 6.41 [09:39:05] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.66, 6.52, 6.41 [09:44:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [09:59:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 6.77, 6.26 [10:02:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [10:05:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.48, 6.13, 6.32 [10:13:16] PROBLEM - wiki.isina.ir - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [10:14:33] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [10:17:08] RECOVERY - wiki.isina.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.isina.ir' will expire on Fri 17 Jan 2020 11:24:18 AM GMT +0000. [10:22:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [10:31:43] PROBLEM - wiki.isina.ir - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [10:54:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server] [11:02:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [11:24:36] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [11:32:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [12:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMBx [12:10:09] [02miraheze/services] 07MirahezeSSLBot 03ef1449c - BOT: Updating services config for wikis [12:14:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.31, 6.33, 5.60 [12:16:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.81, 5.84, 5.56 [12:22:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.22, 6.44, 5.89 [12:24:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.00, 6.43, 6.02 [13:14:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [13:22:34] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:23:54] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.45, 7.21, 6.70 [13:29:44] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.33, 6.25, 6.56 [13:35:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.55, 6.86, 6.77 [13:36:48] !log Update for T4821: Restarting import after it was killed by MW1 [13:36:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:37:35] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.30, 5.92, 6.38 [13:53:02] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.75, 7.66, 6.96 [13:54:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [13:54:57] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.93, 6.80, 6.77 [14:02:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:24:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.47, 6.85, 6.60 [14:28:49] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.38, 6.47, 6.57 [15:17:18] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 4 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx],Service[nginx-prometheus-exporter] [15:21:49] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [15:21:50] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [15:23:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:24:16] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:49] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [15:43:28] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.75, 7.49, 6.49 [15:45:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [15:45:27] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.28, 6.49, 6.29 [15:52:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [15:53:13] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:54:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.40, 6.79 [15:54:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [15:57:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:04:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:08:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.13, 6.54, 6.75 [16:12:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.66, 7.30, 7.02 [16:14:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [16:22:41] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.05, 7.53, 7.30 [16:22:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:24:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.05, 7.39, 7.31 [16:28:56] !log switch off TUN/TAP on mw3 (not needed now) [16:29:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:29:47] paladox: what's TUN/TAP [16:30:08] RhinosF1 https://en.wikipedia.org/wiki/TUN/TAP [16:30:09] [WIKIPEDIA] TUN/TAP | "In computer networking, TUN and TAP are virtual network kernel interfaces. Being network devices supported entirely in software, they differ from ordinary network devices which are backed up by hardware network adapters.The Universal TUN/TAP Driver originated in 2000 as a merger of the corresponding..." [16:30:59] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [16:32:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:32:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [16:32:41] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.52, 4.44, 2.33 [16:33:04] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [16:34:21] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:34:49] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [16:38:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:41:59] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:42:28] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:40] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:42:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [16:42:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:43:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:07] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:44:14] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.125 second response time [16:46:05] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [16:46:50] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [16:46:52] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 0.983 second response time [16:46:53] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.701 second response time [16:49:58] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.045 second response time [16:50:19] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [16:50:51] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [16:51:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:52:39] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [16:53:21] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:53:48] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:55:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [16:55:25] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.003 second response time [16:55:42] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 4 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [16:55:47] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.467 second response time [16:57:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:57:11] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:01:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:01:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:05:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:06:22] !log depool mw3 [17:06:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:09:02] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:09:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [17:09:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [17:11:10] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 2 failures [17:11:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:11:49] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [17:13:31] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [17:13:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:14:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [17:14:21] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:14:33] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:15:39] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:37] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.712 second response time [17:16:41] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.895 second response time [17:17:38] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:17:44] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [17:19:21] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:19:38] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [17:22:16] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:23:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:25:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:29:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:31:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:32:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [17:36:14] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:36:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [17:36:32] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:38:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:40:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:40:19] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 1.774 second response time [17:40:40] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.390 second response time [17:42:44] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [17:51:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMVi [17:51:19] [02miraheze/puppet] 07paladox 0351a24a3 - php: Set process.priority only on kvm [17:54:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [17:55:42] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-redis],Service[rsyslog] [18:02:42] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:03:59] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:07:47] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [18:09:47] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [18:14:33] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [18:22:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:23:19] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [18:25:08] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [18:34:36] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [18:35:05] RECOVERY - mw3 JobRunner Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [18:35:10] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [18:36:03] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9113],Package[php7.3-apcu],Package[php7.3-redis],Service[rsyslog] [18:42:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:56] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.54, 7.68, 6.86 [18:46:51] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.88, 8.16, 7.22 [18:51:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.44, 7.67, 7.40 [18:53:13] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:04:28] Hello sario_temp! If you have any questions, feel free to ask and someone should answer soon. [19:04:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.48, 6.45, 6.76 [19:08:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.03, 6.85, 6.91 [19:16:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.02, 7.23, 7.00 [19:18:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.20, 7.25, 7.05 [19:20:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMra [19:20:10] [02miraheze/services] 07MirahezeSSLBot 03a1f73ef - BOT: Updating services config for wikis [19:22:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.50, 5.57, 6.36 [19:24:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:25:19] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-443] [19:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMov [19:30:09] [02miraheze/services] 07MirahezeSSLBot 0318f0d91 - BOT: Updating services config for wikis [19:32:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:33:36] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:53:58] Hello LakesideMiners_! If you have any questions, feel free to ask and someone should answer soon. [19:54:34] Hello A101_! If you have any questions, feel free to ask and someone should answer soon. [19:54:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [19:55:54] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing] [19:57:11] Hello Gottox_! If you have any questions, feel free to ask and someone should answer soon. [19:57:39] Hello Tegu_! If you have any questions, feel free to ask and someone should answer soon. [20:02:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:23:07] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [20:44:03] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:44:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [20:46:39] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.82, 6.73, 6.16 [20:50:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.24, 6.45, 6.28 [21:02:40] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [21:14:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [21:22:36] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [21:33:47] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.57, 6.92, 6.52 [21:34:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [21:35:30] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:37:33] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.147 second response time [21:38:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:45:31] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.13, 6.05, 6.51 [22:15:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.52, 6.99, 6.70 [22:17:31] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.45, 6.66, 6.64 [22:23:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.05, 7.14, 6.71 [22:24:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [22:25:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.02, 6.93, 6.71 [22:26:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:27:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.35, 6.18, 6.47 [22:41:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMim [22:41:33] [02miraheze/puppet] 07paladox 03796e765 - jobrunner: set nice to -0 temporarily [22:44:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.05, 6.88, 6.45 [22:46:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.01, 6.60, 6.44 [22:52:30] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [22:53:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.17, 7.23, 6.86 [22:55:28] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.22, 6.61, 6.69 [22:56:38] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 2 processes with args 'redisJobChronService' [23:03:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.32, 7.58, 7.03 [23:04:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [23:04:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [23:05:08] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:05:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.89, 6.75, 6.77 [23:06:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:06:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:09:11] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [23:15:05] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:17:09] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 6.965 second response time [23:24:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [23:24:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [23:25:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [23:26:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:26:48] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:27:26] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:30:04] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_34 [+0/-0/±2] 13https://git.io/JeMPp [23:30:05] [02miraheze/mediawiki] 07paladox 03dc16647 - Avoid using deprecated phpredis::delete() alias Bug: T227461 Change-Id: I5eb2fa42d61e4757b11b6eb909c04dafb40923a1 [23:30:07] [02miraheze/mediawiki] 07paladox 03b382a97 - Merge branch 'REL1_34' of https://github.com/wikimedia/mediawiki into REL1_34 [23:30:07] [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [23:31:28] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:39] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:50] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:32:02] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:32:10] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JeMXT [23:32:12] [02miraheze/mediawiki] 07paladox 03444e24b - Avoid using deprecated phpredis::delete() alias Bug: T227461 Change-Id: I5eb2fa42d61e4757b11b6eb909c04dafb40923a1 [23:32:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [23:33:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:34:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:36:09] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 9.244 second response time [23:36:24] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 9.244 second response time [23:36:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:37:03] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.06, 7.56, 7.34 [23:38:58] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.33, 7.52, 7.38 [23:39:42] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 3.226 second response time [23:40:20] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:55] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.54, 7.89, 7.57 [23:44:13] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 2 processes with args 'redisJobChronService' [23:44:29] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:45:01] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.53, 7.47 [23:46:27] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:48:33] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:48:36] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 8.681 second response time [23:54:33] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [23:55:00] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 1.982 second response time [23:55:22] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [23:56:33] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:56:38] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.80, 5.88, 6.48 [23:57:06] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.413 second response time