[01:15:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6b5 [01:15:08] [02miraheze/services] 07MirahezeSSLBot 03c1c04a6 - BOT: Updating services config for wikis [01:16:30] Out of curiosity. I have noticed that wikis with personalized domain and even wikis like Wikivoyage, take a while to be logged in while wikis like CommonsWiki or UcroniasWiki hosted in subdomain, that does not happen to them. Is that something from the system/login? it's just curiosity. [01:17:14] I don't know if you understand me. XD [01:45:09] !log depool mw3 temporarily [01:45:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:48:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:49:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [01:50:00] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:37] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:07] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19031 bytes in 2.068 second response time [01:56:19] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6NR [01:56:59] [02miraheze/puppet] 07paladox 03df502a5 - Revert "varnish: Remove lizardfs6" This reverts commit 7709576470f11e013fc55f3665ad2edc9416baa1. [01:57:57] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19043 bytes in 1.025 second response time [01:58:19] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.003 second response time [01:59:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:59:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:09:37] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:36] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:39] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [02:10:46] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:51] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [02:11:08] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:29] PROBLEM - mw3 SSH on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:35] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [02:11:37] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:11:42] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [02:11:52] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:11:57] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [02:12:22] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:12:40] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:55] PROBLEM - Host mw3 is DOWN: PING CRITICAL - Packet loss = 100% [02:14:54] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.647 second response time [02:15:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [02:15:55] RECOVERY - Host mw3 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [02:15:57] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.055 second response time [02:15:58] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 11070 MB (14% inode=98%); [02:15:58] RECOVERY - mw3 SSH on mw3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [02:16:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [02:16:03] RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [02:16:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.41, 2.21, 0.86 [02:16:19] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 2.187 second response time [02:16:32] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 1.960 second response time [02:16:47] RECOVERY - mw3 JobRunner Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:16:51] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [02:18:56] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:26:46] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [02:28:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [02:29:05] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:43:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.29, 7.04, 6.05 [02:45:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.45, 6.49, 6.03 [03:13:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.65, 7.01 [03:15:30] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.51, 6.77, 6.80 [03:19:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 6.98, 6.89 [03:25:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.13, 6.21, 6.60 [04:03:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:03:37] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:05:10] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 2.76, 6.07, 3.73 [04:05:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:07:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:12:59] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 1.11, 1.25, 1.92 [04:17:01] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:18:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:18:25] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:19:55] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:55] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [04:22:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:22:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:30:55] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 0.82, 0.50, 1.73 [04:32:51] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.02, 0.25, 1.36 [04:44:15] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [04:52:00] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [05:15:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.87, 6.66, 6.38 [05:17:29] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.23, 6.10, 6.19 [06:08:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.22, 7.22, 6.31 [06:14:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.53, 7.94, 7.26 [06:16:17] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.06, 7.82, 7.35 [06:20:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.95, 7.34, 7.36 [06:22:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.15, 7.53, 7.42 [06:24:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.79, 7.26, 7.36 [06:26:39] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2665 MB (11% inode=94%); [06:32:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.72, 6.25, 6.73 [06:44:39] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [07:05:22] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.13, 7.08, 6.52 [07:11:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.31, 7.66, 7.19 [07:13:00] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.16, 7.72, 7.30 [07:14:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.75, 7.04, 7.14 [07:22:35] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.74, 7.65, 7.41 [07:24:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.94, 6.90, 7.19 [07:26:26] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.82, 5.93, 6.74 [07:35:49] PROBLEM - mw3 Disk Space on mw3 is WARNING: DISK WARNING - free space: / 8400 MB (10% inode=98%); [08:08:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.50, 6.97, 6.58 [08:14:17] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.21, 6.80, 6.73 [08:18:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.43, 7.25, 6.96 [08:20:13] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.38, 7.53, 7.12 [08:22:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.45, 7.16, 7.09 [08:28:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.84, 6.29, 6.69 [09:14:44] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [09:20:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [09:22:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [09:22:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:03:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.93, 7.02, 6.28 [10:05:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.85, 6.13, 6.08 [10:14:54] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.22, 7.01, 6.49 [10:20:38] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.21, 6.58, 6.58 [10:44:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.53, 6.83, 6.51 [10:48:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.53, 6.40, 6.40 [11:04:28] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.03, 7.52, 6.70 [11:06:26] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.36, 6.32, 6.35 [11:10:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.21, 7.45, 6.88 [11:14:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.82, 7.04, 6.98 [11:18:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [11:20:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.27, 7.68, 7.32 [11:20:51] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:24:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.88, 7.71, 7.55 [11:32:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.44, 7.38, 7.31 [11:32:42] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:34:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.93, 6.89, 7.15 [11:34:45] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.642 second response time [11:38:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.26, 5.93, 6.64 [11:42:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.88, 7.18, 6.94 [11:46:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.81, 6.16, 6.57 [12:03:02] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.88, 7.66, 7.07 [12:04:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 7.61, 7.20 [12:24:44] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [12:32:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [12:44:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.17, 6.55, 6.77 [12:44:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [12:48:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.70, 6.85, 6.85 [12:49:49] Hi, is there any way to use Special:Export to export all pages in a specified namespace? [12:52:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [12:58:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.96, 6.56, 6.77 [13:04:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.58, 7.16, 6.86 [13:04:43] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [13:06:41] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:10:22] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.54, 6.50, 6.74 [13:15:21] Hi there. Can a steward assist me with a person who keeps on bypassing IP bans? [13:15:50] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:06] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.76, 7.29, 7.07 [13:16:28] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:17:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [13:17:52] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.273 second response time [13:18:11] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 7.015 second response time [13:18:30] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 5.171 second response time [13:19:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:20:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.60, 7.68, 7.27 [13:26:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.61, 7.47, 7.51 [13:26:28] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:28:29] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.410 second response time [13:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeiIC [13:30:09] [02miraheze/services] 07MirahezeSSLBot 03073bf90 - BOT: Updating services config for wikis [14:04:13] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.33, 7.73, 7.27 [14:06:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.80, 7.13, 7.13 [14:10:28] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.48, 7.21, 6.99 [14:12:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.40, 6.72, 6.81 [14:19:59] !log rm *.log* and *.log in /var/log/mediawiki on mw3 [14:20:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:20:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [14:20:56] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 47301 MB (61% inode=98%); [14:24:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.03, 7.86, 7.14 [14:24:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown): Service[postfix],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100],Exec[ufw-allow-udp-from-any-to-any-port-53] [14:24:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:26:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.52, 7.11, 6.94 [14:32:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.25, 6.05, 6.60 [14:32:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:39:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.64, 7.26, 7.00 [14:42:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [14:43:17] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.06, 7.45, 7.17 [14:44:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:45:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.98, 6.33, 6.85 [14:51:09] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.24, 7.44, 7.15 [14:53:40] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:14] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [14:54:34] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.77, 7.04, 7.19 [14:55:42] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 4.294 second response time [14:56:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:56:35] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.125 second response time [15:02:40] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.13, 6.34, 6.76 [15:12:45] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:12:51] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:45] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.10, 7.59, 7.10 [15:15:01] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 7.433 second response time [15:15:40] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.89, 6.88, 6.99 [15:17:34] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.23, 7.75, 7.33 [15:19:06] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.200 second response time [15:19:32] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.04, 7.14, 7.23 [15:21:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.07, 8.12, 7.60 [15:25:32] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.41, 7.85, 7.61 [15:33:15] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.88, 7.52, 7.27 [15:35:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.97, 7.16, 7.25 [15:37:11] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.13, 7.51, 7.35 [15:39:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.45, 7.09, 7.20 [15:50:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.43, 5.93, 6.53 [15:51:50] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeimY [15:53:52] [02miraheze/puppet] 07paladox 03bbf82e4 - Enable parsoid on misc3 [15:57:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeimZ [15:57:18] [02miraheze/puppet] 07paladox 030301133 - Update config.yaml [15:57:48] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:17] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [15:59:30] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeimc [15:59:32] [02miraheze/dns] 07paladox 03bfa37f6 - Move parsoid to misc3 [16:01:48] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:02:59] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.060 second response time [16:07:55] PROBLEM - misc4 parsoid on misc4 is CRITICAL: connect to address 185.52.3.121 and port 8142: Connection refused [16:08:01] [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_34 [+0/-0/±2] 13https://git.io/JeimB [16:08:03] [02miraheze/mediawiki] 07Pix1234 03ce3245c - Update CreatePageUw and ManageWiki from upstream [16:15:14] !log rebuild LC on test1 [16:15:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:27:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeim9 [16:27:56] [02miraheze/puppet] 07paladox 031a42a1a - Update config.yaml [16:35:26] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix] [16:40:24] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CRITICAL - load average: 4.83, 3.48, 1.68 [16:42:23] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 2.55, 2.80, 1.63 [16:44:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeiYt [16:44:11] [02miraheze/dns] 07paladox 0328e7a54 - Revert "Move parsoid to misc3" This reverts commit bfa37f6c930ae2014198b1d217dbb37e1afc557b. [16:44:33] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JeiYq [16:44:34] [02miraheze/puppet] 07paladox 0315cb651 - Revert "Enable parsoid on misc3" This reverts commit bbf82e4f7bdb2e42c01590efe93febd62f5c3675. [16:44:36] [02miraheze/puppet] 07paladox 03c328052 - Revert "Update config.yaml" This reverts commit 0301133944353bc9c96c09a942b7c1babee27a9d. [16:45:32] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:36] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CRITICAL - load average: 11.25, 9.10, 4.62 [16:47:28] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [16:47:39] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:42] RECOVERY - misc4 parsoid on misc4 is OK: TCP OK - 0.001 second response time on 185.52.3.121 port 8142 [16:48:37] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:43] PROBLEM - misc3 parsoid on misc3 is CRITICAL: connect to address 185.52.1.71 and port 8142: Connection refused [16:48:52] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:48:56] !log reboot misc3 [16:49:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:50:39] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.80, 0.37, 0.14 [16:52:46] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:26:38] JohnLewis, Zppix: FYI [[User:LotsaCarps123]] is a sockpuppet of [[User:LotsaCarps]] it has not been used abusively yet but it is something to keep an eye on [17:26:55] Examknow: nothing wrong with using multiple accounts [17:28:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeiOg [17:28:52] [02miraheze/puppet] 07paladox 039e2734f - Update elasticsearch.pp [17:29:26] !log disable ipv6 on lizardfs6 temporarily [17:29:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:34:48] PROBLEM - lizardfs6 HTTPS on lizardfs6 is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [17:35:01] err [17:35:39] should recover i guess [17:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeiOF [17:40:09] [02miraheze/services] 07MirahezeSSLBot 038227df7 - BOT: Updating services config for wikis [17:42:36] RECOVERY - lizardfs6 HTTPS on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 29244 bytes in 0.019 second response time [17:45:24] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [17:47:02] !log apt-get upgrade on ns1 [17:47:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:50:12] !log apt-get install puppet-agent puppetdb puppetdb-termini puppetserver - puppet1 [17:50:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:52:58] Hello artoo_! If you have any questions, feel free to ask and someone should answer soon. [17:53:02] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:53:03] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:53:23] hello friends! :) [17:53:46] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 126 failures. Last run 2 minutes ago with 126 failures. Failed resources (up to 3 shown) [17:53:49] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:53:49] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:04] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 2 minutes ago with 23 failures. Failed resources (up to 3 shown) [17:54:08] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:18] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:22] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:29] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:31] I am admin on crest.miraheze.org. Just had a question .. but it seems that you are busy atm [17:54:40] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:40] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:47] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:54:52] artoo_: i am around what is up [17:56:42] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:03:47] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:03:59] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:04:04] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:04:09] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:04:14] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:04:25] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:04:40] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:04:48] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:04:48] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:05:29] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:05:47] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:06:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:17:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:17:25] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:39] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:40] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:18:03] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:19:31] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 7.277 second response time [18:19:42] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.728 second response time [18:24:13] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4110 bytes in 0.022 second response time [18:24:32] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.580 second response time [18:26:10] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19032 bytes in 1.012 second response time [18:26:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:27:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:30:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jei3X [18:30:20] [02miraheze/puppet] 07paladox 0314802a0 - php: Install php-pear [18:33:02] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jei3M [18:33:04] [02miraheze/puppet] 07paladox 03084c91b - Revert "php: Install php-pear" This reverts commit 14802a0527988bd1130304ca2bbd932ef9f4f029. [18:33:05] [02miraheze/puppet] 07paladox 035a07d24 - php: Install php-mail-mime [18:34:00] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:06] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:06] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:08] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:14] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:21] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:42:11] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:43:50] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:04] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:04] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:06] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:08] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:58:21] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:06:23] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:13:00] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:52] hi [19:24:59] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [19:32:06] https://phabricator.miraheze.org/T4924#93610 [19:32:07] [ ⚓ T4924 error no server....i need them to correct ] - phabricator.miraheze.org [19:43:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:55:00] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 3 minutes ago with 14 failures. Failed resources (up to 3 shown): Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server],Service[ssh] [19:56:47] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 4 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [20:03:13] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:33] Zppix [20:17:42] just wondering all is ok .. I was sure I was browsing to our wiki yesterday and got a 503, was it down yesterday? [20:20:31] artoo_: that happens sometimes normally it just means theres been a hiccup in the servers it should resolve almost immediately [20:35:07] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [21:02:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeiZn [21:02:23] [02miraheze/puppet] 07paladox 03224ea9d - Update apt.pp [21:03:02] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [21:03:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeiZC [21:03:50] [02miraheze/puppet] 07paladox 03a70e852 - Update apt.pp [21:11:16] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[apt_update_gluster] [21:13:11] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [21:15:23] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 302755F9E22EDC1ABB62E9B56C5CDECAAA01DA2C from Apt::Source gluster_apt] [21:19:00] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JeiZM [21:19:01] [02miraheze/puppet] 07paladox 035a8bd1a - Revert "Update apt.pp" This reverts commit a70e852c5e84a45fee3cde177aa46088cdec3d01. [21:19:03] [02miraheze/puppet] 07paladox 039fb4a96 - Revert "Update apt.pp" This reverts commit 224ea9d49748a9412a01a05ee9b2ae99f3c416a0. [21:21:17] The new banner is not available for translation! [21:23:21] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:42:59] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:54:59] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [22:43:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:54:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[ufw-default-deny],Package[openssh-server],Service[ssh],Package[exim4] [23:05:47] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Package[php7.3-apcu],Exec[ops_ensure_members] [23:24:45] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures