[00:02:24] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 3013 MB (15% inode=93%); [00:02:34] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3288 MB (17% inode=93%); [01:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5kD [01:55:09] [02miraheze/services] 07MirahezeSSLBot 03429f83a - BOT: Updating services config for wikis [02:38:31] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.90, 1.58, 1.25 [02:40:27] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.95, 1.33, 1.19 [03:54:11] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.94, 1.77, 1.57 [04:02:02] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.17, 1.85, 1.67 [04:03:59] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.56, 1.73, 1.64 [04:05:57] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.48, 1.62, 1.61 [05:06:56] [02mw-config] 07Pix1234 closed pull request 03#2960: Reconfigurations on Hispanowiki, Privadowiki and Ucroniaswiki - 13https://git.io/Jv5f8 [05:06:58] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5OW [05:06:59] [02miraheze/mw-config] 07Hispano76 0305aeeb5 - Reconfigurations on Hispanowiki, Privadowiki and Ucroniaswiki (#2960) [06:27:10] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3175 MB (13% inode=93%); [06:33:16] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.42, 1.92, 1.63 [06:34:45] !log reception@jobrunner1:~/wiki/images$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively --skip-dupes /home/reception/wiki/images [06:34:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:39:10] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.47, 1.90, 1.76 [06:43:05] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.28, 1.62, 1.67 [08:14:21] Reception123: where is MirahezeRC set in puppet? [08:19:32] Reception123: I think I found it, now I confused [08:19:44] paladox: why have we got 2 RC bots again? [08:19:57] puppet only shows one on mon1 [08:21:10] hm, no idea [08:23:30] Reception123: MirahezeRC_ looks to be coming from the old infra, but why and where it's told to? I don't knoow [08:25:04] RhinosF1: oh, that's probably because misc1 is still online [08:25:10] because mail1 hasn't been setup yet [08:26:00] Reception123: https://github.com/miraheze/puppet/commit/dad34f2177625367184a0b9546226df720800ecc#diff-99ec8ea2d6f28c30f9a0d57bb66577e6 [08:26:00] [ misc1: Remove irc role · miraheze/puppet@dad34f2 · GitHub ] - github.com [08:26:11] it shouldn't run on misc1 but seems to be [08:26:21] let me check [08:33:13] Reception123: any idea? [08:33:41] nope [09:24:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [09:32:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:34:32] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2117 MB (10% inode=93%); [09:45:38] !log stop ircrcbot on misc1 [09:45:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:53:53] paladox: thanks, why was it running when puppet doesn't show the role [09:54:13] because the systemd service thing was still there. [09:54:47] oh [09:54:55] is it gone now? [09:55:38] yes [10:01:46] Good [10:04:37] [02puppet] 07paladox closed pull request 03#1309: remove some old infra config - 13https://git.io/Jv7N7 [10:04:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-23/±8] 13https://git.io/Jv58c [10:04:40] [02miraheze/puppet] 07Reception123 0340f92e9 - remove some old infra config (#1309) * remove some old infra config * change per paladox [10:04:42] [02puppet] 07paladox deleted branch 03Reception123-patch-1 - 13https://git.io/vbiAS [10:04:43] [02miraheze/puppet] 07paladox deleted branch 03Reception123-patch-1 [10:12:00] PROBLEM - bacula2 Bacula Static on bacula2 is UNKNOWN: NRPE: Unable to read output [10:14:01] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: no terminated jobs [10:24:21] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [10:26:21] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.2 weeks ago) [10:27:35] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jv540 [10:27:37] [02miraheze/dns] 07Reception123 038be8c15 - add housing.wiki zone [10:44:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv5Bm [10:44:38] [02miraheze/ssl] 07Reception123 0338088d4 - add housing.wiki cert [11:05:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5BF [11:05:09] [02miraheze/services] 07MirahezeSSLBot 03920216c - BOT: Updating services config for wikis [11:22:21] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/Jv5RX [11:22:22] [02miraheze/ssl] 07Reception123 03671cb53 - add pubwiki.lab.co.pl cert [11:22:24] [02miraheze/ssl] 07Reception123 03ced3506 - add index.pdcommunity.ir cert [11:39:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv50E [11:39:51] [02miraheze/puppet] 07paladox 032395dde - Update mon1.yaml [11:46:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv50N [11:46:43] [02miraheze/puppet] 07paladox 03acd3c6e - Update mon1.yaml [11:47:43] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.45, 2.08, 1.72 [11:49:42] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.42, 1.81, 1.66 [11:53:40] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.56, 1.66, 1.63 [12:06:25] !log reboot services2 [12:06:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:06:46] !log reboot services1 [12:06:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:10:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5ES [12:10:11] [02miraheze/services] 07MirahezeSSLBot 03fd01b99 - BOT: Updating services config for wikis [12:17:38] !log restart php7.3-fpm on mw6 and chown php-error.log as www-data [12:17:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:18:44] !log restart php7.3-fpm on mw7 and chown php-error.log as www-data [12:18:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:19:29] !log restart php7.3-fpm on mw4 [12:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:23:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5u8 [12:23:39] [02miraheze/mw-config] 07paladox 03be12d61 - Log also VisualEditor [12:32:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5uF [12:32:40] [02miraheze/puppet] 07paladox 03ced78e0 - Update config.yaml.erb [12:46:33] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:50:36] PROBLEM - test2 HTTPS on test2 is CRITICAL: connect to address 51.77.107.211 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:52:36] RECOVERY - test2 HTTPS on test2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.006 second response time [12:54:36] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:55:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5g3 [12:56:01] [02miraheze/puppet] 07paladox 033237035 - Update config.yaml [13:02:33] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:03:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5gi [13:03:10] [02miraheze/puppet] 07paladox 03c351caa - Update config.yaml [13:14:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv523 [13:14:02] [02miraheze/puppet] 07paladox 03a02f522 - parsoid: Upgrade to 0.11 [13:28:09] !log restart varnish + nginx on cp7 [13:28:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:31:02] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:32:43] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5aq [13:32:44] [02miraheze/dns] 07paladox 03edb0268 - Depool cp6 [13:33:02] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:28] PROBLEM - cp6 Stunnel Http for test2 on cp6 is CRITICAL: connect to address localhost and port 8090: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:34:39] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=93%); [13:34:45] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5an [13:34:46] [02miraheze/dns] 07paladox 030a63755 - Revert "Depool cp6" This reverts commit edb0268a196f10c8aa5196043ee357e481ac03e4. [13:34:57] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5aC [13:34:59] [02miraheze/dns] 07paladox 034630545 - Update admin_state [13:35:38] PROBLEM - cp7 Stunnel Http for test2 on cp7 is CRITICAL: connect to address localhost and port 8090: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:36:29] PROBLEM - cp6 Stunnel Http for test2 on cp6 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.001 second response time [13:37:38] PROBLEM - cp7 Stunnel Http for test2 on cp7 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.001 second response time [13:49:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5VU [13:49:49] [02miraheze/puppet] 07paladox 039811b1f - nodejs: Upgrade to NodeJS 12 [13:55:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5V4 [13:55:59] [02miraheze/puppet] 07paladox 03e8d68aa - Fix [13:56:24] PROBLEM - services2 restbase on services2 is CRITICAL: connect to address 51.89.160.141 and port 7231: Connection refused [13:57:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5Vz [13:57:30] [02miraheze/puppet] 07paladox 03b074511 - Update miraheze_project.yaml [14:00:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5Vo [14:00:11] [02miraheze/puppet] 07paladox 036db1d1d - nodejs: Downgrade NodeJS to 10 (still an upgrade 8->10) [14:07:10] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jv5VN [14:07:11] [02miraheze/puppet] 07paladox 03f4ce2c5 - Revert "nodejs: Downgrade NodeJS to 10 (still an upgrade 8->10)" This reverts commit 6db1d1d141e3e64039d5b513ceb355e2d4b581b0. [14:07:13] [02miraheze/puppet] 07paladox 033298dac - Revert "nodejs: Upgrade to NodeJS 12" This reverts commit 9811b1fcb2cb71d2ac54b0760f22b442a03c84d4. [14:24:49] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.75, 1.63, 1.49 [14:26:47] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.44, 1.53, 1.46 [14:32:47] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.18, 1.91, 1.63 [14:35:16] PROBLEM - cp6 Disk Space on cp6 is WARNING: DISK WARNING - free space: / 2084 MB (10% inode=93%); [14:36:47] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.36, 1.70, 1.61 [14:37:13] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 2140 MB (11% inode=93%); [15:09:15] PROBLEM - cp6 Disk Space on cp6 is WARNING: DISK WARNING - free space: / 2081 MB (10% inode=93%); [15:25:32] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jv5Ko [15:25:34] [02miraheze/landing] 07paladox 0374a21fd - Update matomo tracker [15:25:36] [02landing] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fxEQA [15:25:37] [02landing] 07paladox opened pull request 03#30: Update matomo tracker - 13https://git.io/Jv5KK [15:26:22] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jv5KP [15:26:23] [02miraheze/landing] 07paladox 031d637c9 - Update donate.php [15:26:25] [02landing] 07paladox synchronize pull request 03#30: Update matomo tracker - 13https://git.io/Jv5KK [15:27:47] [02landing] 07paladox closed pull request 03#30: Update matomo tracker - 13https://git.io/Jv5KK [15:27:49] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv5KD [15:27:50] [02miraheze/landing] 07paladox 03a03cdb6 - Update matomo tracker (#30) * Update matomo tracker * Update donate.php [15:27:52] [02miraheze/landing] 07paladox deleted branch 03paladox-patch-1 [15:27:53] [02landing] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fxEQA [15:34:32] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jv56T [15:34:33] [02miraheze/mediawiki] 07paladox 03a6245fb - Update Cargo [15:36:33] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.38, 1.73, 1.62 [15:38:15] My wiki was closed due to inactivity. I re-opened it, but am still unable to create pages. Is this a bug, or is some other issue at work here? [15:38:32] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.14, 1.53, 1.56 [15:44:37] what wiki? quirc [15:45:02] quircwiki [15:45:08] hispano76: QuIRC, but handled [15:45:25] It was just lagging from the reopen [15:50:06] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:50:36] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:50:44] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:52:31] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:52:41] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:54:06] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:04] PROBLEM - misc1 Disk Space on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:37:22] PROBLEM - misc1 SMTP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:28] PROBLEM - misc1 SSH on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:34] uh [16:38:13] PROBLEM - misc1 IMAP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:26] PROBLEM - ping4 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 315.53 ms [16:39:03] RECOVERY - misc1 Disk Space on misc1 is OK: DISK OK - free space: / 34419 MB (84% inode=98%); [16:39:20] RECOVERY - misc1 SMTP on misc1 is OK: SMTP OK - 0.087 sec. response time [16:39:29] RECOVERY - misc1 SSH on misc1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [16:40:07] RECOVERY - misc1 IMAP on misc1 is OK: IMAP OK - 0.048 second response time on 185.52.1.76 port 143 [* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.] [16:40:26] RECOVERY - ping4 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 18.69 ms [16:41:11] !log apt-get dist upgrade - misc1 [16:41:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:45:35] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 1 minute ago with 6 failures. Failed resources (up to 3 shown): Package[php7.3-common],Package[php7.3-opcache],Package[php7.3-cli],Package[php7.3-fpm] [16:49:26] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:59:38] PROBLEM - services2 Puppet on services2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [17:06:19] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv514 [17:06:21] [02miraheze/dns] 07paladox 03e044dee - Remove services2 temporarily [17:15:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv51M [17:15:10] [02miraheze/services] 07MirahezeSSLBot 03309cce5 - BOT: Updating services config for wikis [17:34:05] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [17:36:46] PROBLEM - bacula2 Current Load on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:38:44] RECOVERY - bacula2 Current Load on bacula2 is OK: OK - load average: 0.23, 0.19, 0.19 [17:44:58] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [17:52:42] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:54:22] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [17:56:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5MQ [17:56:57] [02miraheze/puppet] 07paladox 03c3dd76a - services2: Enable proton [17:57:12] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:57:50] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:01:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5DJ [18:01:02] [02miraheze/puppet] 07paladox 03dccd5d6 - Services: Upgrade restbase config to v1 [18:01:03] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [18:01:04] [02puppet] 07paladox opened pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:02:16] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [18:02:23] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:02:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5DL [18:02:23] [02miraheze/puppet] 07paladox 03a76c02b - Update config.yaml.erb [18:02:30] PROBLEM - services2 Puppet on services2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [18:02:39] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:02:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5Dt [18:02:41] [02miraheze/puppet] 07paladox 03cf0005a - Update config.yaml.erb [18:04:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5Dm [18:04:07] [02miraheze/puppet] 07paladox 036f528dc - Update restbase.pp [18:04:08] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:04:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+1/-1/±0] 13https://git.io/Jv5DO [18:04:28] [02miraheze/puppet] 07paladox 035ea2f4e - Rename miraheze_project.yaml to miraheze_project_v1.yaml [18:04:29] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:05:13] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:06:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+1/-0/±0] 13https://git.io/Jv5D3 [18:06:02] [02miraheze/puppet] 07paladox 036664e27 - Create miraheze_project_sys.yaml [18:06:03] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:07:13] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [18:10:12] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:12:27] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [18:19:20] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:21:29] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 3987 files, 11.59MB, 2020-03-29 00:05:00 (2.8 days ago) [18:22:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5Dw [18:23:00] [02miraheze/puppet] 07paladox 032276197 - Update miraheze_project_v1.yaml [18:23:02] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:25:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5DK [18:25:27] [02miraheze/puppet] 07paladox 03563180d - Update miraheze_project_v1.yaml [18:25:29] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:32:32] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/Jv5Dy [18:32:33] [02miraheze/CreateWiki] 07JohnFLewis 037b2e249 - add wiki_url to CreateWiki [18:33:42] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:35:55] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [18:38:20] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5Db [18:38:20] [02miraheze/CreateWiki] 07JohnFLewis 03ad2dba0 - add url to databases.json and wiki.json [18:43:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5yf [18:43:40] [02miraheze/puppet] 07paladox 03c2f4f8e - Update miraheze_project_v1.yaml [18:43:41] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:45:12] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:46:17] RECOVERY - services2 restbase on services2 is OK: TCP OK - 0.000 second response time on 51.89.160.141 port 7231 [18:47:13] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [18:49:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5yI [18:49:54] [02miraheze/puppet] 07paladox 035ad6eab - Update miraheze_project_sys.yaml [18:49:55] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [18:51:26] PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [18:52:24] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:25] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [19:07:11] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:08:37] PROBLEM - cp8 Disk Space on cp8 is CRITICAL: DISK CRITICAL - free space: / 1154 MB (5% inode=93%); [19:09:12] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 3987 files, 11.59MB, 2020-03-29 00:05:00 (2.8 days ago) [19:25:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5y5 [19:25:40] [02miraheze/puppet] 07paladox 031d383b0 - Update mon1.yaml [19:30:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5Se [19:30:11] [02miraheze/services] 07MirahezeSSLBot 03c9334ff - BOT: Updating services config for wikis [19:31:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5Sf [19:31:05] [02miraheze/puppet] 07paladox 03665e383 - Update mon1.yaml [19:37:56] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:40:02] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [19:50:09] * hispano76 greetings :) [19:54:38] Hey :) [19:58:03] PROBLEM - ping4 on ns1 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:59:21] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:00:03] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 108.75 ms [20:00:55] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:04:29] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [20:08:58] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 3987 files, 11.59MB, 2020-03-29 00:05:00 (2.8 days ago) [20:27:48] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.32, 1.94, 1.32 [20:29:48] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.32, 1.63, 1.27 [20:35:55] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:38:00] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [20:40:57] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:14] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [21:06:09] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.78, 1.69, 1.38 [21:08:10] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.80, 1.39, 1.31 [21:08:43] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:10:43] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [21:29:54] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:31:59] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 3987 files, 11.59MB, 2020-03-29 00:05:00 (2.9 days ago) [21:46:45] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:51:44] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [21:55:30] RhinosF1: testwiki is all cleaned up. Tomorrow I will work on a rollback feature. [22:13:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5QP [22:13:01] [02miraheze/puppet] 07paladox 03ce90810 - Update mediawiki-includes.conf.erb [22:13:03] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [22:18:05] PROBLEM - services2 proton on services2 is CRITICAL: connect to address 51.89.160.141 and port 3000: Connection refused [22:18:31] PROBLEM - services2 electron on services2 is CRITICAL: connect to address 51.89.160.141 and port 3000: Connection refused [22:20:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5Q7 [22:20:37] [02miraheze/puppet] 07paladox 034288a73 - Update config.yaml [22:20:38] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [22:21:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5Q5 [22:21:27] [02miraheze/puppet] 07paladox 03a12ad97 - Update proton.systemd.erb [22:21:29] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [22:22:13] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [22:22:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv5Qx [22:22:16] [02miraheze/puppet] 07paladox 03ee5e50d - Update miraheze_project_sys.yaml [22:23:48] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.55, 1.76, 1.36 [22:24:42] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:25:48] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.86, 1.39, 1.27 [22:26:44] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.3 weeks ago) [22:29:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv57J [22:29:35] [02miraheze/puppet] 07paladox 03159c818 - Update services.pp [22:29:37] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [22:33:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv57m [22:33:06] [02miraheze/puppet] 07paladox 034f4f25f - Update restbase.pp [22:33:08] [02puppet] 07paladox synchronize pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [22:33:41] [02puppet] 07paladox closed pull request 03#1311: Services: Upgrade restbase config to v1 - 13https://git.io/Jv5DU [22:33:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-1/±6] 13https://git.io/Jv57Y [22:33:44] [02miraheze/puppet] 07paladox 0316835df - Services: Upgrade restbase config to v1 (#1311) [22:34:51] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:35:58] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv57G [22:35:59] [02miraheze/dns] 07paladox 03fcbb588 - Revert "Remove services2 temporarily" This reverts commit e044dee3209acbcbd82548cd665bc0773c530c92. [22:41:05] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:41:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [22:41:13] PROBLEM - services1 electron on services1 is CRITICAL: connect to address 51.89.160.132 and port 3000: Connection refused [22:43:18] PROBLEM - services1 proton on services1 is CRITICAL: connect to address 51.89.160.132 and port 3000: Connection refused [22:43:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv57V [22:43:45] [02miraheze/puppet] 07paladox 0338dd770 - Add transform back to restbase [22:49:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv575 [22:49:07] [02miraheze/puppet] 07paladox 03a511bd2 - services: Fix proton monitoring port [22:50:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-1/±0] 13https://git.io/Jv57x [22:50:38] [02miraheze/puppet] 07paladox 0363aeb48 - Decom electron Replaced by proton [22:50:40] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:50:41] [02puppet] 07paladox opened pull request 03#1312: Decom electron - 13https://git.io/Jv57p [22:51:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-1/±0] 13https://git.io/Jv57j [22:51:04] [02miraheze/puppet] 07paladox 0321ea79e - Delete electron.pp [22:51:06] [02puppet] 07paladox synchronize pull request 03#1312: Decom electron - 13https://git.io/Jv57p [22:51:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-1/±0] 13https://git.io/Jv55v [22:51:15] [02miraheze/puppet] 07paladox 03b71c57c - Delete electron.systemd.erb [22:51:16] [02puppet] 07paladox synchronize pull request 03#1312: Decom electron - 13https://git.io/Jv57p [22:51:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv55f [22:51:40] [02miraheze/puppet] 07paladox 037dcb4dc - Update services.pp [22:51:41] [02puppet] 07paladox synchronize pull request 03#1312: Decom electron - 13https://git.io/Jv57p [22:52:21] [02puppet] 07paladox closed pull request 03#1312: Decom electron - 13https://git.io/Jv57p [22:52:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-3/±1] 13https://git.io/Jv55J [22:52:24] [02miraheze/puppet] 07paladox 0336d50e4 - Decom electron (#1312) * Decom electron Replaced by proton * Delete electron.pp * Delete electron.systemd.erb * Update services.pp [22:52:25] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:52:27] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [22:52:37] RECOVERY - services2 proton on services2 is OK: TCP OK - 0.000 second response time on 51.89.160.141 port 3030 [22:52:50] RECOVERY - services1 proton on services1 is OK: TCP OK - 0.000 second response time on 51.89.160.132 port 3030 [22:53:31] [02puppet] 07paladox closed pull request 03#1310: remove lizardfs - 13https://git.io/Jv7Nh [22:53:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-14/±1] 13https://git.io/Jv55I [22:53:34] [02miraheze/puppet] 07Reception123 035d1be61 - remove lizardfs (#1310) [22:53:36] [02puppet] 07paladox deleted branch 03Reception123-patch-2 - 13https://git.io/vbiAS [22:53:37] [02miraheze/puppet] 07paladox deleted branch 03Reception123-patch-2 [23:01:54] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:04:00] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 3987 files, 11.59MB, 2020-03-29 00:05:00 (3.0 days ago) [23:18:38] !log rm /srv/mediawiki/w/cache/managewiki/*realtimefandubgameswikiwiki* - mw* [23:18:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:58:33] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.69, 2.08, 1.38