[00:02:19] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3681 MB (19% inode=93%); [00:09:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvV3G [00:09:03] [02miraheze/puppet] 07paladox 03a508fed - phabricator: Set read only [00:54:13] !log sysctl -w net.ipv6.conf.all.disable_ipv6=1 - db4 [00:54:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:58:07] !log increase phab1 disk by 5g [00:58:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:01:21] !log set write back cache on mon1 [01:02:07] !log set write back cache on mon1 [01:02:13] !log set write back cache on mon1 [01:02:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:03:08] PROBLEM - mon1 Puppet on mon1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 days ago with 0 failures [01:03:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVsc [01:03:43] [02miraheze/puppet] 07paladox 03dcd10ea - phabricator: Switch to db6 and turn off read only [01:04:12] RECOVERY - bacula2 Bacula Static on bacula2 is OK: OK: Full, 6035529 files, 556.7GB, 2020-03-03 04:54:00 (1.8 days ago) [01:06:05] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [01:07:11] PROBLEM - bacula2 Bacula Databases db6 on bacula2 is CRITICAL: CRITICAL: no terminated jobs [01:15:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVsz [01:15:10] [02miraheze/puppet] 07paladox 03bc89bdb - Update site.pp [01:17:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVsa [01:17:01] [02miraheze/puppet] 07paladox 036fb4fe7 - Update nrpe.cfg.erb [01:21:46] RECOVERY - bacula2 Bacula Databases db6 on bacula2 is OK: OK: Full, 4604 files, 21.26GB, 2020-03-01 05:00:00 (3.8 days ago) [01:38:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVsQ [01:38:07] [02miraheze/puppet] 07paladox 03aa62813 - Update init.pp [01:52:16] PROBLEM - gluster1 Disk Space on gluster1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:40] PROBLEM - lizardfs6 GlusterFS port 49152 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 49152: Connection refused [01:55:12] PROBLEM - gluster1 Disk Space on gluster1 is WARNING: DISK WARNING - free space: / 48255 MB (7% inode=82%); [02:05:49] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [02:05:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVGW [02:06:01] [02miraheze/mw-config] 07paladox 03bc55aa9 - Switch staffwiki to db6 [02:06:54] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVG8 [02:06:55] [02miraheze/dns] 07paladox 033c585b3 - Switch staffwiki to cp7 [02:18:13] !log drop staffwiki on db4 [02:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:18:55] !log drop phabricator_% on db4 [02:19:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:21:10] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 26693 MB (7% inode=95%); [02:25:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVZC [02:25:14] [02miraheze/puppet] 07paladox 03a2af1ad - db: Increase table cache a bit [05:25:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVWv [05:25:08] [02miraheze/services] 07MirahezeSSLBot 033839668 - BOT: Updating services config for wikis [06:00:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVlv [06:00:10] [02miraheze/services] 07MirahezeSSLBot 03388ee9f - BOT: Updating services config for wikis [06:25:20] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:27:19] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3148 MB (13% inode=94%); [06:31:04] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:50:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVld [06:50:09] [02miraheze/services] 07MirahezeSSLBot 035b73db5 - BOT: Updating services config for wikis [08:45:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVBH [08:45:09] [02miraheze/services] 07MirahezeSSLBot 03c6a392e - BOT: Updating services config for wikis [08:58:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:1056::2/cpweb [08:58:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:29:19] PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Sat 21 Mar 2020 11:25:43 AM GMT +0000). [11:29:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVay [11:29:36] [02miraheze/ssl] 07MirahezeSSLBot 03ec9d851 - Bot: Update SSL cert for wiki.cdntennis.ca [11:32:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [11:33:09] PROBLEM - vault.aics.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vault.aics.cf' expires in 15 day(s) (Sat 21 Mar 2020 11:29:23 AM GMT +0000). [11:33:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVa7 [11:33:24] [02miraheze/ssl] 07MirahezeSSLBot 03b8b3f53 - Bot: Update SSL cert for vault.aics.cf [11:34:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:41:13] RECOVERY - vault.aics.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'vault.aics.cf' will expire on Wed 03 Jun 2020 10:33:17 AM GMT +0000. [11:43:19] RECOVERY - wiki.cdntennis.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cdntennis.ca' will expire on Wed 03 Jun 2020 10:29:28 AM GMT +0000. [12:26:14] PROBLEM - wiki.iufs.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.iufs.cf' expires in 15 day(s) (Sat 21 Mar 2020 12:23:38 PM GMT +0000). [12:26:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVwZ [12:26:30] [02miraheze/ssl] 07MirahezeSSLBot 03d3753cc - Bot: Update SSL cert for wiki.iufs.cf [12:34:13] RECOVERY - wiki.iufs.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.iufs.cf' will expire on Wed 03 Jun 2020 11:26:22 AM GMT +0000. [14:31:41] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [16:34:42] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVyM [16:34:43] [02miraheze/MirahezeMagic] 07translatewiki 03992638d - Localisation updates from https://translatewiki.net. [16:34:44] [ Main page - translatewiki.net ] - translatewiki.net. [16:34:45] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVyD [16:34:46] [02miraheze/ManageWiki] 07translatewiki 032c5e799 - Localisation updates from https://translatewiki.net. [16:34:47] [ Main page - translatewiki.net ] - translatewiki.net. [16:34:48] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVyy [16:34:49] [02miraheze/CreateWiki] 07translatewiki 035e82428 - Localisation updates from https://translatewiki.net. [16:34:50] [ Main page - translatewiki.net ] - translatewiki.net. [16:34:51] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVyS [16:34:52] [02miraheze/WikiDiscover] 07translatewiki 03a7bea40 - Localisation updates from https://translatewiki.net. [16:34:53] [ Main page - translatewiki.net ] - translatewiki.net. [17:22:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVSA [17:22:41] [02miraheze/puppet] 07paladox 033c93a5a - Update init.pp [17:26:13] RECOVERY - lizardfs6 GlusterFS port 49152 on lizardfs6 is OK: TCP OK - 0.019 second response time on 54.36.165.161 port 49152 [17:26:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvV9e [17:26:42] [02miraheze/puppet] 07paladox 034419981 - Update gluster1.yaml [17:30:17] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:48:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvV90 [17:49:00] [02miraheze/puppet] 07paladox 039e46d74 - puppetdb: Fix support for puppetdb >=6.8 [17:52:46] !log apt-get upgrade - puppet1 (puppetserver, puppetdb and a few other packages) [17:54:22] Hello eir86! If you have any questions, feel free to ask and someone should answer soon. [17:54:49] paladox PLEASE STPO [17:54:50] STOP [18:04:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown) [18:05:09] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [18:05:15] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 414 failures. Last run 2 minutes ago with 414 failures. Failed resources (up to 3 shown) [18:05:15] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [18:05:40] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 423 failures. Last run 2 minutes ago with 423 failures. Failed resources (up to 3 shown) [18:05:45] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 414 failures. Last run 3 minutes ago with 414 failures. Failed resources (up to 3 shown) [18:05:49] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 420 failures. Last run 2 minutes ago with 420 failures. Failed resources (up to 3 shown) [18:06:11] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 418 failures. Last run 2 minutes ago with 418 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/salt-common],File[/usr/local/bin/prometheus-phpfpm-exporter],File[/etc/systemd/system/prometheus-php-fpm.service],File[/etc/logrotate.d/puppet] [18:06:14] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown) [18:06:36] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 441 failures. Last run 3 minutes ago with 441 failures. Failed resources (up to 3 shown) [18:06:47] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 434 failures. Last run 3 minutes ago with 434 failures. Failed resources (up to 3 shown) [18:06:47] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [18:06:54] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 3 minutes ago with 31 failures. Failed resources (up to 3 shown) [18:06:56] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 368 failures. Last run 2 minutes ago with 368 failures. Failed resources (up to 3 shown): File[/etc/sudoers],File[/home/nagiosre],File[/etc/nginx/mime.types],File[/etc/nginx/fastcgi_params] [18:12:12] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [18:12:36] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:12:54] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:12:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:12:56] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:13:15] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [18:13:33] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:13:37] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:13:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:14:22] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:14:40] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:15:02] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:15:08] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:15:13] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:54:04] PROBLEM - mw2 Current Load on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [18:54:31] PROBLEM - mw2 Puppet on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [18:54:55] PROBLEM - mw2 php-fpm on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [18:55:40] PROBLEM - mw2 Disk Space on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [18:56:55] RECOVERY - mw2 php-fpm on mw2 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [18:57:38] RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 69263 MB (85% inode=93%); [18:58:03] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.27, 3.20, 3.14 [18:58:24] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [19:57:25] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.01, 1.38, 0.90 [19:59:24] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.42, 1.44, 0.99 [20:53:06] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.37, 1.77, 1.11 [20:57:04] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.58, 1.36, 1.11 [22:25:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVAv [22:25:51] [02miraheze/puppet] 07paladox 0323e245f - letsencrypt: Add support for buster [22:26:08] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/etc/letsencrypt/cli.ini] [22:28:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVAU [22:28:26] [02miraheze/puppet] 07paladox 038299c6c - letsencrypt::web: Add mon1 to firewall [22:29:04] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:55] [02puppet] 07paladox opened pull request 03#1274: varnish: Switch backend to jobrunner1 for .well-known - 13https://git.io/JvVA3 [22:35:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvVAG [22:35:59] [02miraheze/puppet] 07paladox 035c487cc - varnish: Switch backend to jobrunner1 for .well-known [22:36:01] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:36:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvVAn [22:36:28] [02miraheze/puppet] 07paladox 03576080d - Update stunnel.conf [22:36:30] [02puppet] 07paladox synchronize pull request 03#1274: varnish: Switch backend to jobrunner1 for .well-known - 13https://git.io/JvVA3 [22:36:42] [02puppet] 07paladox closed pull request 03#1270: HACK: Due to a buggy 7.3.15 release, we are forcing the use of debian's own repo - 13https://git.io/Jv2ML [22:36:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [22:36:53] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:38:08] [02puppet] 07paladox synchronize pull request 03#1274: varnish: Switch backend to jobrunner1 for .well-known - 13https://git.io/JvVA3 [22:38:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvVAW [22:38:11] [02miraheze/puppet] 07paladox 032159196 - Update default.vcl [23:32:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb [23:34:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online