[00:00:10] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-6 [00:00:11] [02mw-config] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbvb3 [00:04:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jv4Nx [00:04:06] [02miraheze/puppet] 07paladox 03e19cede - varnish: Use mw[4567] only on cp7 [00:04:08] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [00:04:09] [02puppet] 07paladox opened pull request 03#1250: varnish: Use mw[4567] only on cp7 - 13https://git.io/Jv4Nh [00:08:08] [02puppet] 07paladox closed pull request 03#1250: varnish: Use mw[4567] only on cp7 - 13https://git.io/Jv4Nh [00:08:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4Am [00:08:11] [02miraheze/puppet] 07paladox 037977cc5 - varnish: Use mw[4567] only on cp7 (#1250) [00:08:12] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [00:08:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [00:12:17] RECOVERY - cp7 HTTPS on cp7 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1526 bytes in 0.008 second response time [00:22:47] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw5 mw6 [00:24:47] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 11 backends are healthy [00:26:02] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [00:27:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4A6 [00:27:50] [02miraheze/mw-config] 07paladox 03128cf21 - Take testwiki out of read only [00:42:49] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [01:01:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4xu [01:01:25] [02miraheze/mw-config] 07paladox 0318b2a66 - Update Redis.php [01:02:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jv4xa [01:02:45] [02miraheze/puppet] 07paladox 035b97387 - mw[567]: Set new_servers to true [01:02:47] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [01:02:48] [02puppet] 07paladox opened pull request 03#1251: mw[567]: Set new_servers to true - 13https://git.io/Jv4xV [01:03:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jv4xw [01:03:08] [02miraheze/puppet] 07paladox 03ba05a5b - Update mw6.yaml [01:03:09] [02puppet] 07paladox synchronize pull request 03#1251: mw[567]: Set new_servers to true - 13https://git.io/Jv4xV [01:03:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jv4xr [01:03:19] [02miraheze/puppet] 07paladox 03391733e - Update mw7.yaml [01:03:21] [02puppet] 07paladox synchronize pull request 03#1251: mw[567]: Set new_servers to true - 13https://git.io/Jv4xV [01:03:28] [02puppet] 07paladox closed pull request 03#1251: mw[567]: Set new_servers to true - 13https://git.io/Jv4xV [01:03:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jv4xo [01:03:31] [02miraheze/puppet] 07paladox 039f31d31 - mw[567]: Set new_servers to true (#1251) * mw[567]: Set new_servers to true * Update mw6.yaml * Update mw7.yaml [01:08:33] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [01:08:34] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [01:09:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4x9 [01:09:36] [02miraheze/puppet] 07paladox 03e9e3980 - nutcracker: Restart service when nutcracker.yml changes [01:36:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4pc [01:36:09] [02miraheze/mw-config] 07paladox 030e19169 - Disable sitenotice, no longer needed and also the db is being reinstalled [01:43:55] PROBLEM - db6 SSH on db6 is CRITICAL: connect to address 51.89.160.130 and port 22: Connection refused [01:44:51] expected ^ [01:52:57] RECOVERY - db6 SSH on db6 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [02:07:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4pP [02:07:59] [02miraheze/mw-config] 07paladox 031c19a31 - Switch testwiki to db6 [02:12:48] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv4p9 [02:12:49] [02miraheze/dns] 07paladox 0313aeabd - Make dreamverse use cp7 [02:12:51] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [02:12:52] [02dns] 07paladox opened pull request 03#130: Make dreamverse use cp7 - 13https://git.io/Jv4pH [02:13:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4p5 [02:13:58] [02miraheze/mw-config] 07paladox 03be07d4d - Switch dreamversewiki to db6 [02:14:01] [02dns] 07paladox closed pull request 03#130: Make dreamverse use cp7 - 13https://git.io/Jv4pH [02:14:02] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4pd [02:14:04] [02miraheze/dns] 07paladox 03c4f58bd - Make dreamverse use cp7 (#130) [02:14:05] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [02:14:07] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl [02:21:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/Jv4px [02:21:49] [02miraheze/mw-config] 07paladox 03136c8fd - Move rotompediawiki to new infra [02:21:51] [02mw-config] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbvb3 [02:21:55] [02mw-config] 07paladox opened pull request 03#2891: Move rotompediawiki to new infra - 13https://git.io/Jv4pp [02:23:05] [02mw-config] 07paladox closed pull request 03#2891: Move rotompediawiki to new infra - 13https://git.io/Jv4pp [02:23:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4ph [02:23:08] [02miraheze/mw-config] 07paladox 03a77dde5 - Move rotompediawiki to new infra (#2891) [02:23:09] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-6 [02:23:11] [02mw-config] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbvb3 [02:23:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4pj [02:23:14] [02miraheze/dns] 07paladox 031ab5b1f - Make rotompedia use cp7 [02:32:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4hL [02:32:45] [02miraheze/puppet] 07paladox 03c22b1a7 - Update nginx.conf.erb [03:09:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4hw [03:10:00] [02miraheze/puppet] 07paladox 03b4d2afd - nrpe: Allow mon1.miraheze.org [03:12:40] RECOVERY - jobrunner1 Redis Process on jobrunner1 is OK: PROCS OK: 1 process with args 'redis-server' [03:12:53] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 0.15, 0.14, 0.12 [03:13:20] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:13:20] RECOVERY - jobrunner1 Disk Space on jobrunner1 is OK: DISK OK - free space: / 17203 MB (60% inode=84%); [03:13:29] RECOVERY - mw7 php-fpm on mw7 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [03:13:30] RECOVERY - jobrunner1 JobRunner Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:13:47] RECOVERY - test2 Current Load on test2 is OK: OK - load average: 0.03, 0.08, 0.04 [03:13:56] RECOVERY - jobrunner1 JobChron Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [03:13:56] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 1.01, 1.12, 1.15 [03:13:57] RECOVERY - jobrunner1 php-fpm on jobrunner1 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [03:14:14] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:14:20] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:14:20] RECOVERY - test2 Disk Space on test2 is OK: DISK OK - free space: / 8219 MB (43% inode=77%); [03:14:24] RECOVERY - mw7 Disk Space on mw7 is OK: DISK OK - free space: / 8401 MB (44% inode=77%); [03:14:38] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 4.01, 1.24, 0.43 [03:14:44] RECOVERY - test2 php-fpm on test2 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [03:16:38] RECOVERY - puppet2 Disk Space on puppet2 is OK: DISK OK - free space: / 5240 MB (39% inode=88%); [03:16:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4hy [03:16:42] [02miraheze/puppet] 07paladox 031203c16 - Remove os check in salt::apt Let's install version 2019.2 on the new infra (which run buster). [03:17:57] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:19:16] RECOVERY - mw5 Disk Space on mw5 is OK: DISK OK - free space: / 8108 MB (42% inode=76%); [03:19:30] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 0.29, 0.19, 0.12 [03:19:52] RECOVERY - mw5 php-fpm on mw5 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [03:19:53] RECOVERY - mw4 php-fpm on mw4 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [03:20:01] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 0.60, 0.30, 0.18 [03:20:36] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:20:41] RECOVERY - mw4 Disk Space on mw4 is OK: DISK OK - free space: / 8852 MB (46% inode=76%); [03:21:03] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [03:21:04] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [03:21:15] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [03:21:18] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:21:18] RECOVERY - rdb2 Disk Space on rdb2 is OK: DISK OK - free space: / 7635 MB (80% inode=89%); [03:21:18] RECOVERY - ns2 Current Load on ns2 is OK: OK - load average: 0.11, 0.03, 0.01 [03:21:20] RECOVERY - mail1 Current Load on mail1 is OK: OK - load average: 0.36, 0.16, 0.06 [03:21:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:21:23] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mw5' not defined [03:21:25] RECOVERY - mail1 Disk Space on mail1 is OK: DISK OK - free space: / 5487 MB (57% inode=84%); [03:21:25] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 6254 MB (32% inode=93%); [03:21:25] RECOVERY - db6 Disk Space on db6 is OK: DISK OK - free space: / 616033 MB (99% inode=99%); [03:21:25] PROBLEM - gluster1 Disk Space on gluster1 is WARNING: DISK WARNING - free space: / 56148 MB (9% inode=83%); [03:21:25] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:21:26] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:21:28] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:21:28] RECOVERY - rdb1 Redis Process on rdb1 is OK: PROCS OK: 1 process with args 'redis-server' [03:21:43] RECOVERY - db7 Disk Space on db7 is OK: DISK OK - free space: / 617381 MB (99% inode=99%); [03:21:43] RECOVERY - services1 Current Load on services1 is OK: OK - load average: 0.22, 0.11, 0.03 [03:21:43] RECOVERY - cp6 Stunnel Http for misc2 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 43457 bytes in 0.115 second response time [03:21:44] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mw7' not defined [03:21:49] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:21:53] PROBLEM - cp6 Stunnel Http for test2 on cp6 is UNKNOWN: NRPE: Command 'check_stunnel_test2' not defined [03:21:53] RECOVERY - rdb2 Redis Process on rdb2 is OK: PROCS OK: 1 process with args 'redis-server' [03:21:53] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [03:22:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 2.60, 2.15, 1.71 [03:22:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4h7 [03:22:33] [02miraheze/puppet] 07paladox 0360eae36 - Update minions.pp [03:22:42] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:22:49] RECOVERY - cloud2 Disk Space on cloud2 is OK: DISK OK - free space: / 1116579 MB (62% inode=99%); [03:22:49] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 0.29, 0.22, 0.19 [03:22:53] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:22:53] RECOVERY - db6 Current Load on db6 is OK: OK - load average: 0.04, 0.04, 0.01 [03:22:54] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [03:22:54] PROBLEM - cp7 Stunnel Http for test2 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_test2' not defined [03:22:57] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 3% [03:22:59] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:23:07] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 3% [03:23:07] RECOVERY - rdb1 Disk Space on rdb1 is OK: DISK OK - free space: / 7299 MB (76% inode=88%); [03:23:07] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:23:12] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [03:23:12] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is UNKNOWN: NRPE: Command 'check_stunnel_mw7' not defined [03:23:12] RECOVERY - mon1 IRC Log Bot on mon1 is OK: PROCS OK: 1 process with args 'adminlogbot.py' [03:23:21] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is UNKNOWN: NRPE: Command 'check_stunnel_mw5' not defined [03:23:21] RECOVERY - mw6 Disk Space on mw6 is OK: DISK OK - free space: / 8402 MB (44% inode=77%); [03:23:22] RECOVERY - mon1 Disk Space on mon1 is OK: DISK OK - free space: / 24550 MB (85% inode=92%); [03:23:22] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 2.64, 3.50, 3.34 [03:23:22] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:23:22] RECOVERY - services2 Current Load on services2 is OK: OK - load average: 0.02, 0.02, 0.00 [03:23:22] RECOVERY - rdb1 Current Load on rdb1 is OK: OK - load average: 0.04, 0.05, 0.01 [03:23:26] RECOVERY - mon1 IRC RC Bot on mon1 is OK: PROCS OK: 1 process with args 'ircrcbot.py' [03:23:26] RECOVERY - phab1 phd on phab1 is OK: PROCS OK: 1 process with args 'phd' [03:23:26] RECOVERY - phab1 Current Load on phab1 is OK: OK - load average: 0.22, 0.24, 0.15 [03:23:26] RECOVERY - ns2 Disk Space on ns2 is OK: DISK OK - free space: / 7453 MB (78% inode=88%); [03:23:30] RECOVERY - db7 Current Load on db7 is OK: OK - load average: 0.05, 0.07, 0.02 [03:23:30] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 0.07, 0.15, 0.11 [03:23:30] RECOVERY - cp7 Disk Space on cp7 is OK: DISK OK - free space: / 6653 MB (35% inode=94%); [03:23:34] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 0.59, 0.66, 0.33 [03:23:43] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 0.03, 0.04, 0.01 [03:23:43] RECOVERY - cp7 Stunnel Http for misc2 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 43457 bytes in 0.122 second response time [03:23:53] RECOVERY - services1 Disk Space on services1 is OK: DISK OK - free space: / 5402 MB (56% inode=72%); [03:23:53] RECOVERY - services2 Disk Space on services2 is OK: DISK OK - free space: / 5728 MB (60% inode=73%); [03:23:54] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:24:09] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is UNKNOWN: NRPE: Command 'check_stunnel_mw6' not defined [03:24:09] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mw4' not defined [03:24:11] RECOVERY - phab1 Disk Space on phab1 is OK: DISK OK - free space: / 3090 MB (17% inode=84%); [03:24:13] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:24:13] RECOVERY - cloud1 Disk Space on cloud1 is OK: DISK OK - free space: / 1678626 MB (94% inode=99%); [03:24:13] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mw6' not defined [03:24:14] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:24:24] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:24:24] RECOVERY - rdb2 Current Load on rdb2 is OK: OK - load average: 0.05, 0.05, 0.01 [03:24:29] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is UNKNOWN: NRPE: Command 'check_stunnel_mw4' not defined [03:24:29] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 0.04, 0.09, 0.05 [03:24:33] RECOVERY - mw6 php-fpm on mw6 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [03:28:27] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[apt_update_salt],Exec[apt_update] [03:31:44] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:35:25] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:36:38] PROBLEM - cp8 Disk Space on cp8 is CRITICAL: DISK CRITICAL - free space: / 1132 MB (5% inode=93%); [03:38:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4jf [03:38:27] [02miraheze/puppet] 07paladox 03cf929b4 - cp8: Lower cache_file_size to 9G [03:41:50] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:42:38] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2168 MB (11% inode=93%); [03:45:28] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.368 second response time [03:45:44] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.385 second response time [03:45:45] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.369 second response time [03:47:24] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18664 bytes in 0.953 second response time [03:47:41] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18663 bytes in 0.886 second response time [03:47:45] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18665 bytes in 0.959 second response time [04:12:38] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2115 MB (10% inode=93%); [04:36:27] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.401 second response time [04:36:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [04:36:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [04:38:23] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18664 bytes in 1.387 second response time [04:38:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [04:38:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:02:38] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 4238 MB (21% inode=93%); [06:07:05] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:13:43] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:26:04] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3281 MB (13% inode=94%); [06:28:32] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:31:48] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:37:37] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:40:55] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:46:30] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mrlove.wiki' expires in 15 day(s) (Fri 06 Mar 2020 06:42:55 AM GMT +0000). [06:46:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBfr [06:46:46] [02miraheze/ssl] 07MirahezeSSLBot 03bbad620 - Bot: Update SSL cert for mrlove.wiki [06:54:28] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Tue 19 May 2020 05:46:38 AM GMT +0000. [07:15:11] Reception123: testwiki is down [07:51:40] Reception123: It's back [11:25:18] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [11:33:17] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:03:24] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBOv [13:03:25] [02miraheze/dns] 07paladox 031fd01e0 - Put test.m.org on cp7 [13:22:41] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBO2 [13:22:42] [02miraheze/dns] 07paladox 03eee651d - Update miraheze.org [13:54:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [13:55:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [13:58:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:58:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:58:49] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.36, 1.99, 1.31 [14:00:50] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.68, 1.51, 1.22 [14:04:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [14:08:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:11:06] Hello Rots61! If you have any questions, feel free to ask and someone should answer soon. [14:12:13] why does miraheze not work? It happens quite often. [14:16:19] Rots61: which wiki? [14:16:37] publictestwiki [14:17:23] now he works [14:17:35] paladox: [14:17:59] Rots61: it’s running on new infrastructure. [14:18:45] that doesn't work good. [14:19:14] Rots61: it seems that way [14:19:39] It’s been on it all day [14:19:51] it works for me [14:19:54] what does not what with publictestwiki? [14:20:03] RhinosF1 ^ [14:20:07] *work [14:20:13] paladox: slow, db connection errors this morning [14:20:23] he give an error. [14:20:44] Rots61: what does that say? [14:21:09] an unknown error. [14:21:23] and a part in Dutch [14:21:41] Rots61: can u paste the full error? [14:22:56] the error is already past time 2 minutes ago. [14:23:44] Rots61: if you can paste the error it will be helpful [14:23:46] That'll be due to cache i imagine [14:24:00] paladox: would the error show in logs? [14:24:05] yes [14:24:08] database.log [14:24:16] do i have access to that? [14:24:39] Wed Feb 19 14:21:56 UTC 2020 mw1.miraheze.org testwiki Error connecting to db6.miraheze.org as user mediawiki: :real_connect(): (HY000/2002): Connection timed out [14:25:22] i/o related maybe paladox? [14:25:27] no [14:25:37] as you can see it's using the wrong server [14:25:52] oh yes it is, i see that now [14:25:52] testwiki is to use the new servers, so your itermittently being sent through cp4 [14:26:19] cp4 is using the old infra, cp7 is using the new infra (uses mw[4567]) [14:27:26] paladox: out of curiousity is it possible to force it to use cp7? [14:28:13] that's what it should be doing, publictestwiki has changed it so it uses cp7 rather then mw-lb [14:31:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBsF [14:31:21] [02miraheze/puppet] 07paladox 031e3a829 - Update db.pp [14:40:53] PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:42:49] PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Full, 2336892 files, 199.8GB, 2020-01-30 17:58:00 (2.8 weeks ago) [15:00:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBZ8 [15:00:45] [02miraheze/puppet] 07paladox 0324409cf - Update db.pp [15:08:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBZd [15:08:43] [02miraheze/puppet] 07paladox 038f85044 - Add ipv6 address for mw[123] [15:09:28] Rots61 should be fixed now. [15:35:01] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:18] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:19] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:21] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:21] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:25] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:25] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:45] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:36:03] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:36:11] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:04] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:08] paladox: “all works now” followed by a massive wall of icinga alerts [15:37:19] that's puppet failure Zppix [15:37:24] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:24] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:24] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:24] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:24] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:24] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:24] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:25] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:25] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:28] Holy spam [15:37:30] Zppix: give it 10 mins [15:37:42] it's caused by me anyways [15:37:43] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:44] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:38:13] paladox: well i think you made it angry [15:39:08] Zppix: https://meta.miraheze.org/w/index.php?diff=96707&oldid=96662&rcid=378807#A_suspected_sock [15:39:13] [ Difference between revisions of "Stewards' noticeboard" - Miraheze Meta ] - meta.miraheze.org [15:39:35] RhinosF1: a steward has to handle that [15:43:42] SPF|Cloud, PuppyKun: ^ [15:48:22] that's not really something causing ongoing issues I don't think that you have to ping anyone, let alone multiple people for that [15:51:06] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [15:51:22] Incoming alert spam [15:51:25] PuppyKun: I didn't massively look, I'm on a call. I knew you were online stewards who could look [15:51:49] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:51:49] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:51:49] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [15:51:53] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:51:55] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [15:51:59] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:24] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:31] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:31] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:36] Wee [15:53:19] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:21] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:36] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:36] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:36] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:59] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:59] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:53:59] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:53:59] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:54:00] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:54:00] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:54:00] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:55:57] Zppix: lol [15:58:28] big spam [15:58:41] Such spam [15:59:24] much wow! [15:59:33] Data a lotta spam [15:59:47] s/data/dats [16:01:39] ZppixBot: no correction? [16:01:48] S/?/?? [16:01:48] Zppix meant to say: ZppixBot: no correction?? [16:01:53] lol [16:01:56] Interesting [16:10:14] * paladox spams PuppyKun & Zppix :P [16:11:58] * RhinosF1 sends all the spam to paladox [16:15:09] Heh [16:32:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 51.161.32.127/cpweb [16:34:27] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2639 MB (10% inode=94%); [16:36:40] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:42:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [16:44:37] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:17:37] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [17:19:32] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:00:14] Zppix: see discord if around [19:00:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBBU [19:00:20] [02miraheze/mw-config] 07paladox 03990eb0d - Set nbdbwiki and nonbinarywiki into read only [19:01:07] RhinosF1: probably not today [19:01:15] (re your discord msg) [19:01:54] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvBBq [19:01:55] [02miraheze/dns] 07paladox 031b17787 - Set nonbinary.wiki and data.nonbinary.wiki to use new cache proxy Does not migrate pl to use the new cache proxy. [19:01:57] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [19:01:58] [02dns] 07paladox opened pull request 03#131: Set nonbinary.wiki and data.nonbinary.wiki to use new cache proxy - 13https://git.io/JvBBm [19:05:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvBBG [19:05:23] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2607:5300:205:200::17f6/cpweb [19:05:24] [02miraheze/mw-config] 07paladox 032f49c43 - Move nbdbwiki and nonbinarywiki to db6 [19:05:25] [02mw-config] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbvb3 [19:05:26] [02mw-config] 07paladox opened pull request 03#2892: Move nbdbwiki and nonbinarywiki to db6 - 13https://git.io/JvBBn [19:06:44] [02mw-config] 07paladox closed pull request 03#2892: Move nbdbwiki and nonbinarywiki to db6 - 13https://git.io/JvBBn [19:06:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBB4 [19:06:47] [02miraheze/mw-config] 07paladox 037e39526 - Move nbdbwiki and nonbinarywiki to db6 (#2892) [19:07:02] [02dns] 07paladox closed pull request 03#131: Set nonbinary.wiki and data.nonbinary.wiki to use new cache proxy - 13https://git.io/JvBBm [19:07:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBBB [19:07:05] [02miraheze/dns] 07paladox 03da84204 - Set nonbinary.wiki and data.nonbinary.wiki to use new cache proxy (#131) Does not migrate pl to use the new cache proxy. [19:08:12] [02miraheze/mediawiki] 07Pix1234 deleted branch 03REL1_33 [19:08:13] [02mediawiki] 07Pix1234 deleted branch 03REL1_33 - 13https://git.io/vbL5b [19:08:29] [02miraheze/mw-config] 07Pix1234 deleted branch 03paladox-patch-6 [19:08:30] [02mw-config] 07Pix1234 deleted branch 03paladox-patch-6 - 13https://git.io/vbvb3 [19:08:32] [02mw-config] 07Pix1234 deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [19:08:33] [02miraheze/mw-config] 07Pix1234 deleted branch 03paladox-patch-3 [19:08:35] [02mw-config] 07Pix1234 deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [19:08:36] [02miraheze/mw-config] 07Pix1234 deleted branch 03paladox-patch-2 [19:08:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:09:31] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl [19:09:33] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [19:10:31] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [19:11:11] paladox: ^ [19:12:28] yup, aware [19:12:32] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:13:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [19:15:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [19:15:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [19:17:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:18:07] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [19:19:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:20:20] RhinosF1: want me to nuke https://meta.miraheze.org/wiki/Translations:Template:Mental_Health_Resources? [19:20:23] [ Translations:Template:Mental Health Resources - Miraheze Meta ] - meta.miraheze.org [19:21:21] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:22:01] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBB9 [19:22:02] [02miraheze/dns] 07paladox 033826934 - Fix [19:22:17] RhinosF1: nvm it doesnt seem i can [19:23:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:29:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [19:31:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:32:17] Zppix: you can or leave them in case I ever work out how to do Translateable Templates [19:32:24] k [19:32:30] It’s still very WIP [19:35:32] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBR4 [19:35:33] [02miraheze/dns] 07paladox 030794c69 - Fix part 2 [19:39:02] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvBRg [19:39:04]