[00:14:22] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [00:20:50] !log root@mw2:/var/log/mediawiki/debuglogs# rm * [00:20:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:22:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [00:35:09] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:35:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [00:37:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:37:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:43:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [00:45:38] !log set 8.8.8.8 back in resolv.conf on mw[123], cp[24] and misc[1234] [00:45:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:46:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDiZ [00:46:56] [02miraheze/puppet] 07paladox 035e1860d - mediawiki: Increase php mysql timeout to 8 [00:49:51] !log restart php7.3-fpm on mw[123] and lizardfs6 [00:49:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:03:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:33:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [01:43:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:35:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [02:51:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [03:10:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDPd [03:10:11] [02miraheze/services] 07MirahezeSSLBot 03d02f41b - BOT: Updating services config for wikis [03:33:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [03:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:50:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDX9 [03:50:09] [02miraheze/services] 07MirahezeSSLBot 03e169b1e - BOT: Updating services config for wikis [03:53:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [04:11:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:43:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [04:48:10] Hello Agent! If you have any questions, feel free to ask and someone should answer soon. [05:03:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:07:33] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Full, 882487 files, 53.21GB, 2019-11-20 05:04:00 (2.1 weeks ago) [05:36:11] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1442 MB (5% inode=94%); [05:38:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:39:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [05:40:32] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:32] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:44:44] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 5.898 second response time [05:46:50] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Feb 2020 08:50:14 AM GMT +0000. [05:47:36] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [05:50:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:52:39] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 15% [05:53:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:12:11] PROBLEM - bacula1 Bacula Databases db5 on bacula1 is WARNING: WARNING: Full, 2161 files, 71.19GB, 2019-11-20 06:10:00 (2.1 weeks ago) [06:13:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [06:19:12] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Full, 80934 files, 2.828GB, 2019-11-20 06:16:00 (2.1 weeks ago) [06:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:26:26] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2443 MB (10% inode=94%); [07:00:00] RhinosF1: check other headphones [08:08:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [08:10:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:43:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:51:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:03:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [09:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:33:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [09:41:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [10:15:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [10:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [11:05:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [11:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [12:43:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [12:51:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:13:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [14:23:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:31:05] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JeDdk [14:31:06] [02miraheze/MirahezeMagic] 07translatewiki 03096a32c - Localisation updates from https://translatewiki.net. [14:31:07] [ Main page - translatewiki.net ] - translatewiki.net. [14:31:09] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDdI [14:31:10] [02miraheze/ManageWiki] 07translatewiki 03b1d6b25 - Localisation updates from https://translatewiki.net. [14:31:11] [ Main page - translatewiki.net ] - translatewiki.net. [14:31:13] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeDdL [14:31:15] [02miraheze/WikiDiscover] 07translatewiki 03b42857f - Localisation updates from https://translatewiki.net. [14:31:15] [ Main page - translatewiki.net ] - translatewiki.net. [16:06:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [16:06:52] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:08:52] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 2.228 second response time [16:12:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:15:19] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:34] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [16:17:23] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.329 second response time [16:21:50] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:25:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [16:26:24] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:42] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:50] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:28:40] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.390 second response time [16:29:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:30:01] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 3.343 second response time [16:30:30] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 7.564 second response time [18:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:43:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [21:03:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:54:46] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [21:56:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:14:23] !log deleteBatch.php on mw1 for T4954 [22:14:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:23:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [22:31:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:23:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [23:25:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:26:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [23:27:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online