[00:56:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [01:25:46] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CRITICAL: Diff, 46119 files, 36.42GB, 2019-10-20 01:25:00 (4.4 weeks ago) [01:43:30] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [01:59:10] Voidwalker: ping [01:59:17] hi [01:59:24] PM [01:59:32] sure [02:13:57] !log root@bacula1:/bacula/backup# rm * [02:14:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:15:02] RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 471203 MB (98% inode=99%); [02:17:13] !log apt-get upgrade on bacula1 [02:17:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:21:03] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Package[dirmngr],Package[nagios-plugins],Package[postfix],Package[puppet-agent] [02:23:30] !log reboot bacula1 - kernel update [02:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:23:49] PROBLEM - bacula1 Bacula Databases db5 on bacula1 is CRITICAL: connect to address 172.245.38.205 port 5666: Connection refusedconnect to host 172.245.38.205 port 5666: Connection refused [02:26:07] PROBLEM - bacula1 SSH on bacula1 is CRITICAL: connect to address 172.245.38.205 and port 22: Connection refused [02:26:41] PROBLEM - bacula1 Bacula Daemon on bacula1 is CRITICAL: connect to address 172.245.38.205 port 5666: Connection refusedconnect to host 172.245.38.205 port 5666: Connection refused [02:27:02] PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: connect to address 172.245.38.205 port 5666: Connection refusedconnect to host 172.245.38.205 port 5666: Connection refused [02:28:05] RECOVERY - bacula1 SSH on bacula1 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u1 (protocol 2.0) [02:28:37] RECOVERY - bacula1 Bacula Daemon on bacula1 is OK: PROCS OK: 2 processes with UID = 116 (bacula) [02:29:02] RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 471063 MB (98% inode=99%); [02:31:34] !log regenerating backups [02:31:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:33:05] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:58:12] Voidwalker: Hi [03:58:19] hi [05:06:21] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 882487 files, 53.21GB, 2019-11-20 05:04:00 (2.3 minutes ago) [05:33:55] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [05:34:25] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 401 failures. Last run 2 minutes ago with 401 failures. Failed resources (up to 3 shown) [05:34:40] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 412 failures. Last run 2 minutes ago with 412 failures. Failed resources (up to 3 shown) [05:34:42] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 19 failures. Last run 2 minutes ago with 19 failures. Failed resources (up to 3 shown) [05:34:46] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown) [05:34:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [05:34:52] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 386 failures. Last run 3 minutes ago with 386 failures. Failed resources (up to 3 shown) [05:35:05] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 243 failures. Last run 2 minutes ago with 243 failures. Failed resources (up to 3 shown): File[/etc/default/stunnel4],File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate] [05:35:06] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [05:35:08] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 39 failures. Last run 3 minutes ago with 39 failures. Failed resources (up to 3 shown) [05:35:09] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [05:35:21] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [05:35:37] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 405 failures. Last run 3 minutes ago with 405 failures. Failed resources (up to 3 shown) [05:35:40] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 399 failures. Last run 3 minutes ago with 399 failures. Failed resources (up to 3 shown) [05:35:40] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown) [05:35:58] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 402 failures. Last run 3 minutes ago with 402 failures. Failed resources (up to 3 shown): File[/opt/texvc_3.0.0+git20160613-1_amd64.deb],File[/etc/mathoid/config.yaml],File[/etc/apt/trusted.gpg.d/php.gpg],File[/usr/lib/php/20180731/luasandbox.so] [05:37:20] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 166 failures. Last run 3 minutes ago with 166 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org],File[wildcard.miraheze.org_private],File[/etc/default/stunnel4],File[/etc/stunnel/mediawiki.conf] [05:43:01] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [05:43:05] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [05:43:20] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [05:43:33] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:43:54] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:44:12] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [05:44:14] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [05:44:33] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:44:44] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [05:44:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:44:49] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:44:54] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:44:55] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:45:04] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:45:24] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:45:28] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:45:42] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:12:02] RECOVERY - bacula1 Bacula Databases db5 on bacula1 is OK: OK: Full, 2161 files, 71.19GB, 2019-11-20 06:10:00 (2.0 minutes ago) [06:12:21] RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 4311 files, 8.982MB, 2019-11-20 06:10:00 (2.4 minutes ago) [06:18:09] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 80934 files, 2.828GB, 2019-11-20 06:16:00 (2.1 minutes ago) [06:26:39] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2697 MB (11% inode=94%); [07:28:44] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [09:03:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [09:03:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [09:05:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:05:47] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:26:36] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [10:28:44] Hello TWfighter! If you have any questions, feel free to ask and someone should answer soon. [12:13:32] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:25:37] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx],Service[nginx-prometheus-exporter] [12:45:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6Y0 [12:45:10] [02miraheze/services] 07MirahezeSSLBot 036e7cdb3 - BOT: Updating services config for wikis [13:04:46] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [13:04:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [13:06:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:06:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:33:30] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx] [14:53:27] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:17:01] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:17:36] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:19:07] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.220 second response time [15:19:39] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 7.531 second response time [15:23:42] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:27:40] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:32:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [15:34:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [15:36:00] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:36:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:55:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [15:59:59] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:01:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [16:04:09] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [16:04:27] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:05:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6na [16:05:57] [02miraheze/puppet] 07paladox 033a09e13 - nginx: Disable keepalive_timeout temporarily [16:06:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:06:25] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.674 second response time [16:09:48] !log restart nginx and php7.3 on mw2 [16:09:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:09:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:10:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6nK [16:10:36] [02miraheze/puppet] 07paladox 034fd8ece - Revert "nginx: Disable keepalive_timeout temporarily" This reverts commit 3a09e13a5caa49bf6ea74218949303fa1a8c0421. [16:15:33] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [16:21:00] @Stewards pm? [16:21:54] go ahead [16:25:20] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [16:26:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:26:20] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:35:12] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [16:45:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:47:24] hello [16:47:48] im bored [16:48:27] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:54:53] ErtasVideos: come say hello (in #miraheze-offtopic ) [16:57:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:59:36] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:59:48] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:59:52] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:00:00] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:01:34] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.395 second response time [17:01:43] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.666 second response time [17:01:58] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.004 second response time [17:02:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:04:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:13:02] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:29] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:39] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:40] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:14:31] well [17:14:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:14:41] what just happened? [17:15:34] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19033 bytes in 0.630 second response time [17:15:59] idk [17:17:46] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19033 bytes in 0.451 second response time [17:18:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:20:40] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.12, 6.69, 5.72 [17:20:48] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [17:21:48] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 3.97, 11.15, 9.50 [17:22:06] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [17:22:07] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:36] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.76, 5.89, 5.53 [17:22:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:23:20] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.955 second response time [17:23:56] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 3.69, 6.10, 7.72 [17:24:11] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.692 second response time [17:24:50] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [17:26:07] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [17:26:37] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19034 bytes in 2.135 second response time [17:27:55] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.05, 5.16, 6.57 [17:39:32] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:41:27] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:43:37] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:53] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:18] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:45:44] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.004 second response time [17:45:55] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.461 second response time [17:52:29] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:54:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:54:32] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 3.172 second response time [18:02:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [18:04:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [18:05:25] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:05:46] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:06:27] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:07:33] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.100 second response time [18:12:11] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:16:18] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 6.710 second response time [18:16:53] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.902 second response time [18:17:13] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 8.000 second response time [18:19:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [18:21:31] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:16] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:23:31] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.004 second response time [18:24:22] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.699 second response time [18:29:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:29:54] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:35:18] Reception123: Te he respondido [18:47:26] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [18:53:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:54:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [18:58:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:58:49] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:59:04] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:00:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:03:02] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 4.643 second response time [19:03:14] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.402 second response time [19:05:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:11:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:12:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [19:13:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:14:04] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:31:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [19:33:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:42:52] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [19:44:51] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.310 second response time [19:45:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:52:42] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [19:53:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [19:56:46] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.390 second response time [19:57:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:02:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [20:04:01] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:05:10] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:09] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.671 second response time [20:08:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:14:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:16:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:39:32] !log deleted all the logs apart from debuglogs in /var/log/mediawiki on mw2 [20:39:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:56:42] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-80],Package[php7.3-redis],Service[rsyslog] [21:03:04] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [21:18:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [21:20:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:30:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [21:32:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:37:08] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 4 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9253],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9113],Service[nginx-prometheus-exporter] [21:48:35] !log depool mw2 [21:48:42] !log reimage mw2 as buster (10) [21:48:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:48:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:50:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6RR [21:50:27] [02miraheze/puppet] 07paladox 03521b7a5 - Revert "varnish: Remove lizardfs6" This reverts commit 366be5dd29df689605ee399ce6f7da0a3faf0342. [21:57:26] PROBLEM - mw2 Disk Space on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [21:57:49] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [21:57:55] PROBLEM - mw2 Puppet on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [21:58:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [21:58:14] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [21:58:19] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: connect to address 185.52.2.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:58:26] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [21:58:31] PROBLEM - mw2 php-fpm on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [21:58:37] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [21:58:55] PROBLEM - mw2 Current Load on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [21:59:11] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [22:03:45] PROBLEM - mw2 Puppet on mw2 is UNKNOWN: NRPE: Unable to read output [22:03:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:05:05] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 1.25, 0.85, 0.40 [22:05:14] RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 78520 MB (96% inode=98%); [22:08:34] RECOVERY - mw2 php-fpm on mw2 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3' [22:21:46] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.008 second response time [22:22:14] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.404 second response time [22:22:20] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.641 second response time [22:22:21] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 458 bytes in 0.016 second response time [22:23:24] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 26 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[glusterfs-client] [22:25:51] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.003 second response time [22:27:12] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [22:27:43] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [22:27:51] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.004 second response time [22:28:09] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [22:29:23] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 1 failures [22:31:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [22:31:43] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [22:32:09] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [22:33:12] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [22:33:25] PROBLEM - mw2 Puppet on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [22:33:43] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [22:36:10] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [22:40:24] [02ssl] 07Pix1234 opened pull request 03#239: Regenerate SSL cert - 13https://git.io/Je60V [22:40:51] [02ssl] 07paladox closed pull request 03#239: Regenerate SSL cert - 13https://git.io/Je60V [22:40:52] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je60o [22:40:54] [02miraheze/ssl] 07Pix1234 033b8715d - Regenerate SSL cert (#239) [22:57:00] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:01:42] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [23:01:59] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [23:02:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [23:03:42] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.007 second response time [23:03:54] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [23:04:04] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [23:07:23] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [23:09:24] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [23:09:44] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [23:09:53] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [23:13:27] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:13:45] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [23:13:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [23:15:15] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [23:17:21] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:01] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown): Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server],Service[ssh] [23:33:02] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6Er [23:35:09] [02miraheze/services] 07MirahezeSSLBot 030bce66b - BOT: Updating services config for wikis [23:36:58] !log repooled mw2 [23:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:43:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6EM [23:43:24] [02miraheze/puppet] 07paladox 034f8c2ed - varnish: Remove lizardfs6 This reverts commit 521b7a520f5093f59912eabc22a22c6badf0e6cf. [23:45:12] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.