[00:00:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [00:02:48] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.90, 6.99, 6.64 [00:04:34] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:23] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:30] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:33] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.391 second response time [00:06:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:06:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.68, 6.03, 6.47 [00:07:31] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 7.947 second response time [00:07:32] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19053 bytes in 7.298 second response time [00:12:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.84, 7.29, 6.91 [00:13:22] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:28:47] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.75, 6.24, 6.64 [00:32:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.69, 6.89, 6.83 [00:48:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.39, 6.67, 6.78 [00:55:35] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.88, 7.06, 6.96 [01:01:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.01, 6.61, 6.80 [01:05:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 7.14, 7.02 [01:06:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [01:08:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:14:50] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.30, 7.80, 7.42 [01:16:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.85, 7.09, 7.25 [01:22:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.03, 7.66, 7.44 [01:24:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.06, 7.59, 7.48 [01:24:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [01:25:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [01:27:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:32:37] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [01:48:54] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.61, 6.21, 6.77 [01:54:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [01:54:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.28, 7.06, 6.86 [01:56:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:58:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.60, 6.41, 6.67 [02:04:33] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [02:04:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.73, 6.93, 6.77 [02:06:40] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.53, 6.80, 6.75 [02:10:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.01, 7.07, 6.90 [02:12:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:12:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.03, 6.35, 6.63 [02:12:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [02:14:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:17:55] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [02:19:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [02:19:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.62, 7.01, 6.72 [02:20:08] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 2.575 second response time [02:22:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:23:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:23:12] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.71, 7.84, 7.06 [02:26:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:27:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [02:28:39] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:41] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:29:31] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:29:36] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:34:48] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 4.050 second response time [02:35:03] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.89, 7.30, 7.71 [02:35:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:35:55] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.164 second response time [02:35:57] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.639 second response time [02:52:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [02:53:13] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.68, 7.22, 7.28 [02:53:29] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:23] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.25, 7.57, 7.45 [02:55:51] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:55:52] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19053 bytes in 2.116 second response time [02:56:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:56:15] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 4 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [02:56:25] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 7.206 second response time [02:57:50] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.952 second response time [02:57:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:58:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:02:16] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:14:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [03:18:39] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.48, 6.50, 6.80 [03:22:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:24:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.14, 6.91, 6.78 [03:26:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.83, 6.48, 6.66 [03:34:40] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.66, 7.34, 6.85 [03:45:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMDN [03:45:10] [02miraheze/services] 07MirahezeSSLBot 03319d7be - BOT: Updating services config for wikis [03:58:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.15, 6.18, 6.68 [04:02:40] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.10, 7.42, 7.04 [04:04:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.22, 6.24, 6.70 [04:15:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.36, 6.99, 6.86 [04:19:41] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.67, 6.68, 6.79 [04:23:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.77, 7.39, 7.07 [04:33:26] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.48, 7.63, 7.27 [04:35:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.44, 6.75, 6.98 [04:39:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.21, 6.06, 6.61 [05:15:18] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [05:23:14] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:43:17] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.13, 7.01, 6.49 [05:45:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.16, 6.99, 6.59 [05:47:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.37, 6.77, 6.60 [05:52:57] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.58, 7.28, 6.75 [05:54:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.31, 6.94, 6.74 [05:56:46] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.25, 6.25, 6.54 [06:03:33] PROBLEM - yellowiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'yellowiki.xyz' expires in 15 day(s) (Thu 19 Dec 2019 06:00:29 AM GMT +0000). [06:03:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMS9 [06:03:48] [02miraheze/ssl] 07MirahezeSSLBot 033c3d59c - Bot: Update SSL cert for yellowiki.xyz [06:08:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.78, 7.47, 6.95 [06:12:38] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.33, 7.70, 7.20 [06:13:32] RECOVERY - yellowiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'yellowiki.xyz' will expire on Mon 02 Mar 2020 05:03:40 AM GMT +0000. [06:14:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.32, 7.79, 7.37 [06:24:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [06:32:34] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:32:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.31, 7.36, 6.99 [06:34:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.56, 7.83, 7.29 [06:42:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.16, 7.47, 7.25 [06:44:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.33, 7.81, 7.47 [06:52:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.21, 8.06, 7.53 [06:54:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.60, 7.90, 7.60 [06:56:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.29, 8.06, 7.73 [07:00:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.29, 7.57, 7.65 [07:02:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.06, 7.96, 7.80 [07:03:54] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [07:04:09] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 2 minutes ago with 23 failures. Failed resources (up to 3 shown) [07:04:10] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [07:04:22] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 425 failures. Last run 2 minutes ago with 425 failures. Failed resources (up to 3 shown) [07:04:23] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 46 failures. Last run 2 minutes ago with 46 failures. Failed resources (up to 3 shown) [07:04:24] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 2 minutes ago with 24 failures. Failed resources (up to 3 shown) [07:04:37] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [07:04:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.80, 7.77, 7.75 [07:04:44] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 420 failures. Last run 3 minutes ago with 420 failures. Failed resources (up to 3 shown) [07:04:53] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [07:05:08] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 405 failures. Last run 3 minutes ago with 405 failures. Failed resources (up to 3 shown) [07:05:35] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 4 minutes ago with 7 failures. Failed resources (up to 3 shown) [07:05:36] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 3 minutes ago with 16 failures. Failed resources (up to 3 shown) [07:05:42] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 418 failures. Last run 3 minutes ago with 418 failures. Failed resources (up to 3 shown) [07:05:56] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 244 failures. Last run 2 minutes ago with 244 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local],File[/etc/default/varnish],File[/etc/systemd/system/varnish.service],File[wildcard.miraheze.org] [07:06:26] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 422 failures. Last run 4 minutes ago with 422 failures. Failed resources (up to 3 shown): File[/root/ufw-fix],File[/usr/local/bin/gen_fingerprints],File[/etc/vim/vimrc.local],File[/usr/share/nginx/favicons/default.ico] [07:06:33] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 161 failures. Last run 3 minutes ago with 161 failures. Failed resources (up to 3 shown): File[/root/ufw-fix],File[/usr/local/bin/gen_fingerprints],File[/etc/vim/vimrc.local],File[/etc/default/varnish] [07:06:45] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:12:38] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.94, 8.24, 7.91 [07:12:53] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:13:38] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:13:39] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:13:53] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:14:02] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:16] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:26] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:40] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:47] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:14:50] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:55] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:15:02] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:13] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:15:38] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:59] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:40] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.32, 7.81, 7.89 [07:16:53] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:45] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.57, 7.91, 7.88 [07:20:23] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:22:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [07:24:29] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.701 second response time [07:24:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:24:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [07:26:40] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.67, 7.86, 8.00 [07:32:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:38:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.21, 6.01, 6.61 [07:42:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.30, 7.77, 7.24 [07:46:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.68, 7.31, 7.32 [07:52:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.68, 7.70, 7.31 [07:58:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.82, 7.42, 7.45 [08:00:47] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.01, 7.47, 7.44 [08:03:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [08:05:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [08:08:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.99, 7.56, 7.68 [08:16:39] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.04, 5.35, 6.56 [08:24:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.68, 6.86, 6.45 [08:26:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [08:26:56] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.09, 6.71, 6.52 [08:28:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:33:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.32, 7.55, 7.04 [08:35:45] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.10, 6.42, 6.66 [08:43:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 8.00, 7.58, 7.01 [08:44:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:51:20] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.49, 6.68, 6.78 [08:52:43] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:09:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.79, 7.16, 6.80 [09:22:47] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.24, 7.40, 7.22 [09:24:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.64, 7.21, 7.21 [09:30:13] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:30:32] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:32:15] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 5.501 second response time [09:32:34] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 4.520 second response time [09:32:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.10, 7.29, 7.08 [09:34:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.28, 7.44, 7.21 [09:40:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.32, 6.07, 6.63 [09:44:36] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [09:50:38] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:51:02] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:54:41] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.009 second response time [09:55:16] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.763 second response time [10:02:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [10:10:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [10:10:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [10:12:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:12:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:12:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 8.00, 7.73, 7.14 [10:14:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.04, 6.65, 6.80 [10:14:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [10:22:38] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [10:33:38] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.21, 7.52, 6.83 [10:35:32] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.80, 6.56, 6.60 [10:45:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [10:47:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:53:11] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:53:39] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:53:42] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [10:55:14] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 8.538 second response time [10:55:44] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 6.539 second response time [10:57:32] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:04:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.82, 7.01, 6.73 [11:08:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.06, 6.33, 6.54 [11:15:15] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [11:15:16] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:16:16] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:16:29] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:17:35] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [11:18:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [11:19:42] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19053 bytes in 3.634 second response time [11:21:45] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.004 second response time [11:22:44] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.51, 7.42, 6.67 [11:23:31] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:24:39] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.34, 6.88, 6.56 [11:24:46] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.642 second response time [11:24:51] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:25:26] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.623 second response time [11:26:41] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.25, 6.23, 6.39 [11:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMd6 [11:30:09] [02miraheze/services] 07MirahezeSSLBot 03e80f986 - BOT: Updating services config for wikis [11:30:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:43:04] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:43:12] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.65, 7.59, 7.08 [11:45:14] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.496 second response time [11:45:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [11:47:53] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:50:58] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.19, 5.88, 6.52 [12:05:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [12:05:29] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [12:07:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:09:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:09:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:13:00] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.14, 7.41, 6.86 [12:13:40] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:14:55] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.87, 6.58, 6.66 [12:23:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.70, 7.27, 6.88 [12:27:28] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.41, 6.70, 6.78 [12:35:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [12:43:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:45:48] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.26, 7.28, 6.76 [12:59:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.47, 6.45, 6.78 [13:07:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.84, 6.71, 6.64 [13:09:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.24, 6.34, 6.50 [13:15:18] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [13:24:48] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.41, 6.85, 6.76 [13:26:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.85, 6.42, 6.61 [13:33:25] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:20] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:36:22] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:38:29] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 5.930 second response time [13:39:30] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.744 second response time [13:43:43] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:45:30] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [13:45:40] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 2.018 second response time [13:53:32] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:15:24] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [14:17:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [14:19:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:23:00] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.14, 7.33, 6.71 [14:23:12] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:24:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.45, 7.24, 6.84 [14:26:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.72, 6.77, 6.71 [14:32:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [14:34:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:34:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.75, 6.96, 6.81 [14:36:47] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.32, 6.15, 6.52 [14:43:37] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:49] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [14:45:00] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.70, 6.88, 6.72 [14:45:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [14:46:55] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.60, 6.71, 6.69 [14:47:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:47:42] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.107 second response time [14:49:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:01:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:05:36] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:05:37] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:05:42] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [15:05:59] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:07:36] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.226 second response time [15:07:55] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.003 second response time [15:14:24] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.80, 7.17, 6.88 [15:16:24] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 5.309 second response time [15:16:45] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.86, 6.64, 6.75 [15:22:28] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:44] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:48] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:23:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.67, 6.86, 6.80 [15:24:10] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:24:33] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.753 second response time [15:24:44] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.748 second response time [15:26:12] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.649 second response time [15:29:35] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.38, 6.56, 6.71 [15:32:48] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:33:01] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:33:35] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.71, 7.19, 6.93 [15:34:56] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 7.317 second response time [15:36:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [15:36:29] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:37:16] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 7.793 second response time [15:38:33] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 8.508 second response time [15:42:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [15:43:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.39, 6.11, 6.60 [15:50:56] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:07] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:31] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:54:54] JohnLewis, paladox: mw3 seems to be having stunnel issues and mw1 rendering is failed [15:55:53] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:17] I feel my wiki (ucroniaswiki) slower than usual. [15:57:30] [02ssl] 07RhinosF1 opened pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [15:57:36] hispano76: known [15:57:56] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2141 MB (8% inode=94%); [15:58:13] [02ssl] 07RhinosF1 synchronize pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [15:58:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [15:58:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:58:32] [02ssl] 07RhinosF1 synchronize pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [15:58:41] Oh, I see. [15:58:57] hispano76: not much i can do though [15:59:05] [02ssl] 07RhinosF1 synchronize pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [15:59:27] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.644 second response time [15:59:34] No problem RhinosF1 thanks :) [15:59:40] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.393 second response time [15:59:40] [02ssl] 07RhinosF1 synchronize pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [16:00:00] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.003 second response time [16:00:59] i can get into mw3 fine [16:01:36] [02ssl] 07RhinosF1 synchronize pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [16:02:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:03:00] JohnLewis, paladox, SPF|Cloud: can we get some help here? backends keep dying [16:03:57] @Operations [16:04:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:04:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:04:28] [02ssl] 07RhinosF1 edited pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [16:06:51] Grafana is down as well [16:07:15] https://www.irccloud.com/pastebin/QDlInLuh/ [16:07:16] [ Snippet | IRCCloud ] - www.irccloud.com [16:08:07] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.14, 6.63, 6.20 [16:08:45] ^ known, nothing I know how to fix [16:09:08] hispano76: apologies for the slowness, i'm nagging to get someone here asap. [16:10:02] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.11, 6.21, 6.12 [16:11:35] no problem RhinosF1 :) [16:12:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [16:13:24] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:15:28] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.923 second response time [16:17:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:19:09] hispano76: paladox has just told there's nothing we can do [16:19:59] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:49] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:21:58] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.259 second response time [16:22:56] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 9.218 second response time [16:24:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [16:25:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.13, 7.01, 6.66 [16:25:59] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:19] Ok RhinosF1 :) [16:27:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.63, 6.21, 6.43 [16:28:05] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.980 second response time [16:28:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:33:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [16:33:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:36:35] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:32] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [16:38:34] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 0.383 second response time [16:39:27] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 0.634 second response time [16:39:45] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:39:58] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:40:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:40:15] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:41:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:43:32] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.64, 7.03, 6.60 [16:43:57] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.646 second response time [16:44:04] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.390 second response time [16:44:18] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.005 second response time [16:45:31] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.50, 5.93, 6.23 [16:48:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [16:50:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [16:50:18] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:53:58] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [16:55:55] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 1.157 second response time [16:56:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:00:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:01:54] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.73, 7.08, 6.76 [17:03:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [17:08:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [17:09:44] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:40] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 1.000 second response time [17:11:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:14:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:14:41] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:16:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.59, 5.78, 6.57 [17:16:44] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.572 second response time [17:18:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:18:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [17:23:57] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.31, 6.94, 6.81 [17:24:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:25:23] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:25:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.56, 6.74, 6.75 [17:29:39] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 5.515 second response time [17:32:03] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.44, 7.05, 6.86 [17:33:22] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:58] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.77, 6.48, 6.71 [17:35:18] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 0.984 second response time [17:35:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:37:44] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:37:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [17:38:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 6.88, 6.76 [17:39:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:43:27] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [17:44:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.41, 6.00, 6.52 [17:46:53] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:48:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:48:47] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:49:58] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 6.984 second response time [17:52:28] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:46] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [17:54:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.18, 7.08, 6.71 [17:54:30] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.933 second response time [17:54:53] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.814 second response time [17:56:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [17:56:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.32, 6.80, 6.69 [17:57:49] !log run sudo -u www-data /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maint*/runJobs.php on mw3 [17:57:57] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [17:57:59] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [17:58:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:58:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:58:25] !log stop jobrunner/jobchron temporarily on mw3 [17:59:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:59:15] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 8.079 second response time [18:01:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:01:11] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.225 second response time [18:02:14] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [18:05:45] !log cancled run on mw3 and instead are running it on mw2 [18:06:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:15:27] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server] [18:15:55] [02ssl] 07paladox closed pull request 03#242: Remove 3 certs - 13https://git.io/JeMja [18:15:56] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-3/±1] 13https://git.io/JeDJZ [18:15:58] [02miraheze/ssl] 07RhinosF1 03d329c8f - Remove 3 certs (#242) * Remove 3 certs Service: wiki.isina.ir - LetsEncrypt Service: wiki.hackdown.org - LetsEncrypt Service: endlesssea.lucyawrey.com - LetsEncrypt * Delete wiki.isina.ir.crt * Delete wiki.hackdown.org.crt * Delete endlesssea.lucyawrey.com.crt * Update certs.yaml * Update certs.yaml [18:23:21] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:25:00] paladox: can you make sure MangeWiki is updated on those wikis? [18:25:18] Sorry busy dealing with the jobrunner :( [18:25:29] Zppix ^ [18:25:35] (do you mind doing that please) [18:25:56] * RhinosF1 is mobile for likely rest of day [18:26:29] RhinosF1: yes [18:27:00] Zppix: update ManageWiki on the wikis where those certs were revoked [18:31:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [18:31:45] ik [18:32:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [18:33:36] RhinosF1: do they even exist? [18:33:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:34:00] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:37:06] RhinosF1: i cant access ivazedu.miraheze.org to even change it [18:38:01] paladox: ^ [18:38:18] paladox: i cant even access the wikis to change the settings is there a way to do it from meta or mw1? [18:43:45] argh [18:45:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [18:46:11] hmm [18:50:39] !log UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.isina.ir "', null ) WHERE wiki_dbname = 'ivazeduwiki'; [18:50:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:51:51] {{done}} [18:52:15] paladox: great now i just have to do that for the other 1 [18:52:22] i hate sql [18:52:25] i always screw it up [18:52:26] yup [18:52:38] Zppix just remember you only need to change wiki.* [18:53:13] paladox: when i run sql.php what --wiki do i run it as? the wiki im changing? [18:53:16] UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/ "', null ) WHERE wiki_dbname = ''; [18:53:26] type [18:53:31] use mhglobal; [18:53:37] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:54:11] paladox: sql.php requires a --wiki param [18:54:15] so what do i use [18:54:20] oh [18:54:24] try mhglobal? [18:54:27] ok [18:54:29] if not i can run the sql [18:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDUm [18:55:09] [02miraheze/services] 07MirahezeSSLBot 03917df8c - BOT: Updating services config for wikis [18:55:25] yeah it wont let me UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.hackdown.org"> "', null ) WHERE wiki_dbname = 'hackdownwiki'; [18:55:30] ok [18:55:36] !log UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.hackdown.org"> "', null ) WHERE wiki_dbname = 'hackdownwiki'; [18:55:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:55:49] Zppix done [18:56:23] !log UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.hackdown.org "', null ) WHERE wiki_dbname = 'hackdownwiki'; [18:56:24] works [18:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:56:40] paladox: logging the same thing twice? [18:56:45] nope [18:56:49] you made an error in the syntax [18:57:03] oh i forgot to remove a bracket [19:00:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDUZ [19:00:09] [02miraheze/services] 07MirahezeSSLBot 0316618c8 - BOT: Updating services config for wikis [19:02:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [19:04:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:15:56] PROBLEM - misc4 Prometheus on misc4 is CRITICAL: connect to address 185.52.3.121 and port 9090: Connection refused [19:17:51] RECOVERY - misc4 Prometheus on misc4 is OK: TCP OK - 0.001 second response time on 185.52.3.121 port 9090 [19:19:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDUX [19:19:33] [02miraheze/puppet] 07paladox 03b9bfae1 - Enable jobrunner on lizardfs6 and disable on mw3 temporarily [19:23:08] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:23:34] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [19:25:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDU9 [19:25:15] [02miraheze/puppet] 07paladox 03e09f896 - Update mw3.yaml [19:31:46] !log reboot mw3 [19:34:10] !log [19:31:46] <+paladox> !log reboot mw3 [19:35:05] paladox: thou shant log [19:35:12] :P [19:36:04] i'm just gonna reinstall mw3 [19:36:50] !log reinstall mw3 as buster [19:37:58] what happened to logbot paladox ? [19:38:04] good question [19:38:24] Dec 03 19:38:08 misc1 logbot[25926]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='meta.miraheze.org', port=443): Read timed out. (read timeout=30) [19:45:42] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [19:46:11] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:29] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:51] hmm [19:46:58] phab is truly down [19:47:31] network issues [19:49:16] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.007 second response time [19:49:21] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:58] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:24] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [19:50:31] rip [19:53:05] https://meta.miraheze.org/wiki/Requests_for_Comment/Define_images_policy_for_Miraheze_Commons. (preliminary version) comments? paladox Reception123 RhinosF1 SPF|Cloud Voidwalker Zppix [19:53:06] [ Requests for Comment/Define images policy for Miraheze Commons. - Miraheze Meta ] - meta.miraheze.org [19:54:53] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.302 second response time [19:55:38] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [19:55:39] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 4 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:56:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:56:29] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.507 second response time [19:57:06] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:58:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [19:58:41] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 6.612 second response time [19:59:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [20:00:47] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [20:01:48] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.674 second response time [20:02:29] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [20:02:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:05:06] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:05:07] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [20:05:31] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.005 second response time [20:08:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [20:09:17] PROBLEM - mw3 Puppet on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [20:09:58] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [20:14:41] Note: 504 Gateway Time-out phabricator.miraheze.org / Not problem :) [20:14:47] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 5.960 second response time [20:19:43] !log restart networking service on misc4 [20:20:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:20:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:21:45] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.012 second response time [20:22:55] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19074 bytes in 0.130 second response time [20:23:30] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17725 bytes in 0.062 second response time [20:26:50] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:28:16] .op [20:28:16] Please wait... [20:28:56] !log restarting networking on cp* and mw* (temporarily hack by crossing out 8.8.8.8 in /etc/resolv.conf) [20:29:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:31:14] !log begin import for brokenwindowswiki (T4946) [20:31:20] !log restart nginx and php7.3-fpm on mw[12] [20:31:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:31:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:32:33] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 1.074 second response time [20:32:55] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.075 second response time [20:33:18] !log restarting networking on misc* (temporarily hack by crossing out 8.8.8.8 in /etc/resolv.conf) [20:33:19] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.099 second response time [20:33:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:33:32] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.608 second response time [20:48:04] PROBLEM - mw3 Puppet on mw3 is UNKNOWN: NRPE: Unable to read output [20:48:28] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 69771 MB (96% inode=98%); [20:48:29] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 1.07, 0.75, 0.36 [20:53:29] RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3' [20:54:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [20:55:56] !log run sudo -u www-data /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maint*/runJobs.php on mw2 [20:57:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:58:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:58:40] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:59:06] !log run rebuildrecentchanges.php and initsitestats.php on brokenwindowswiki on mw2 following completion of request import [20:59:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:00:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:00:50] !log stopped script [21:01:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:02:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [21:02:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:07:13] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 458 bytes in 0.009 second response time [21:08:36] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.638 second response time [21:08:52] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.390 second response time [21:08:59] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.004 second response time [21:11:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDIX [21:11:49] [02miraheze/puppet] 07paladox 030f712e4 - php: Pin packages.sury.org [21:12:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [21:12:48] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:14:49] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 5.884 second response time [21:16:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:18:19] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 54 seconds ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],Package[glusterfs-client] [21:18:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [21:18:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:19:26] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [21:19:27] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [21:28:04] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:34:15] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [21:34:22] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [21:34:28] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [21:34:45] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [21:34:46] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:34:46] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [21:35:11] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:37:18] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [21:43:57] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.011 second response time [21:44:13] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.656 second response time [21:44:27] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.390 second response time [21:44:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [21:44:55] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [21:44:56] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [21:44:57] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.027 second response time [21:46:33] "Repooled" no work! paladox [21:47:59] No spanish my wiki :( [21:48:40] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:48:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [21:49:03] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:06] hispano76: ¿Cuál es el problema? [21:49:59] Zppix: UcroniasWiki aparecía en inglés pero parece que ya está arreglado :) Siempre que falla "Repooled", los wikis en otros idiomas aparecen en inglés [21:50:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:51:04] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 5.279 second response time [21:53:20] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [21:53:29] !log repool mw3 [21:53:31] hispano76 oh? [21:53:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:54:49] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 5.288 second response time [21:54:53] example on hispano.miraheze.org https://usercontent.irccloud-cdn.com/file/Aw3cvVYV/Anotaci%C3%B3n%202019-12-03%20155406.png [21:57:33] paladox: did you rebuildlc on mw3 after resintall [21:57:38] oh [21:58:10] !log rebuild lc on mw3 [21:58:12] hum, confused you? [21:58:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:01:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:03:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:04:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDLa [22:04:36] [02miraheze/puppet] 07paladox 037d2a6d0 - Update config.yaml [22:05:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [22:09:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:25:25] !log switched mw* to use 1.1.1.1 [22:25:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:25:38] !log that's in /etc/resolv.conf [22:25:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:26:13] paladox: any reason 1.1.1.1 and not 8.8.4.4? [22:27:03] JohnLewis we had 8.8.4.4, i'm seeing if this will help with performance. Since i noticed when 8.8.8.8 went down requests were hanging. [22:27:07] i can revert [22:27:17] !log revert back to 8.8.4.4 on mw* [22:27:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:43:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [22:45:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:55:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDtw [22:55:09] [02miraheze/services] 07MirahezeSSLBot 03be58455 - BOT: Updating services config for wikis [23:04:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [23:04:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [23:06:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:06:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:12:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [23:14:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:15:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [23:18:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [23:18:40] !log rebuildrc, initsitestats after import completed for wfhbwiki [23:19:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:22:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:23:38] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:33:00] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:43:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100]