'; [01:33:05] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is UNKNOWN: NRPE: Unable to read output [01:33:08] PROBLEM - cp8 Varnish Backends on cp8 is UNKNOWN: NRPE: Unable to read output [01:33:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:33:20] PROBLEM - cp8 HTTPS on cp8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:28] PROBLEM - cp8 Stunnel Http for lizardfs6 on cp8 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8203: HTTP/1.1 200 OK [01:34:36] [02miraheze/puppet] 07Southparkfan created branch 03cp8 13https://git.io/JvCLk [01:34:38] [02puppet] 07Southparkfan created branch 03cp8 - 13https://git.io/vbiAS [01:35:07] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [01:35:10] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03cp8 [+0/-0/±1] 13https://git.io/JvCLI [01:35:11] [02miraheze/puppet] 07Southparkfan 03fac49bc - add cp8 to monitoring [01:35:43] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03cp8 [+0/-0/±1] 13https://git.io/JvCLL [01:35:44] [02miraheze/puppet] 07Southparkfan 03a77ebbe - add cp8 to lizardfs ufw [01:36:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [01:36:02] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [01:36:03] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [01:36:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [01:36:06] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [01:36:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [01:36:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [01:36:11] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:36:21] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03cp8 [+0/-0/±1] 13https://git.io/JvCLq [01:36:22] [02miraheze/puppet] 07Southparkfan 03bb4549f - add cp8 port to mw [01:36:54] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03cp8 [+0/-0/±1] 13https://git.io/JvCLm [01:36:56] [02miraheze/puppet] 07Southparkfan 03b012540 - add cp8 to nginx real-ip [01:37:08] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [01:37:19] [02puppet] 07Southparkfan opened pull request 03#1197: Introduce cp8 - 13https://git.io/JvCLY [01:37:54] [02puppet] 07Southparkfan closed pull request 03#1197: Introduce cp8 - 13https://git.io/JvCLY [01:37:56] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JvCLs [01:37:57] [02miraheze/puppet] 07Southparkfan 03616b2ce - Introduce cp8 (#1197) * add cp8 to monitoring * add cp8 to lizardfs ufw * add cp8 port to mw * add cp8 to nginx real-ip [01:38:08] [02puppet] 07paladox deleted branch 03cp8 - 13https://git.io/vbiAS [01:38:09] [02miraheze/puppet] 07paladox deleted branch 03cp8 [01:38:12] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCLG [01:38:14] [02miraheze/mw-config] 07Southparkfan 03016f17d - Add cp8.miraheze.org [01:43:08] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 54% [01:43:08] RECOVERY - cp8 HTTPS on cp8 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1542 bytes in 0.374 second response time [01:43:44] SPF|Cloud we should probs lower the cache file? [01:44:11] paladox|UKInEU: maybe, depends on the log file size (which is what I'm looking at now) [01:44:11] 15G -> 10g? [01:44:20] SPF|Cloud seems cp8 has 1.8g left [01:44:28] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1853 MB (9% inode=93%); [01:45:28] yeah, 1.8 won't do it unfortunately [01:45:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [01:46:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [01:47:18] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: connect to address 107.191.126.23 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:47:32] hmm [01:47:54] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:49:13] I see no obvious errors on cp2 [01:49:40] [02miraheze/puppet] 07Southparkfan created branch 03allow-cache-size-change 13https://git.io/JvCLw [01:49:42] [02puppet] 07Southparkfan created branch 03allow-cache-size-change - 13https://git.io/vbiAS [01:49:49] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:50:22] SPF|Cloud nginx -t is taking a long time, same for service nginx restart... [01:50:53] i'll reboot cp2 [01:51:02] !log reboot cp2 [01:51:06] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1550 bytes in 0.383 second response time [01:51:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:51:29] did you depool it? [01:52:02] SPF|Cloud it depooled it's self. [01:52:04] ^ nevermind [01:52:17] eh yes, I see [01:53:11] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 39% [01:53:51] SPF|Cloud nginx still not starting... i guess someones abusing the resources on the host cp2 is on [01:54:42] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03allow-cache-size-change [+0/-0/±1] 13https://git.io/JvCLo [01:54:44] [02miraheze/puppet] 07Southparkfan 03f73b93d - Update init.pp [01:55:27] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: connect to address 107.191.126.23 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:55:34] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03allow-cache-size-change [+0/-0/±1] 13https://git.io/JvCL6 [01:55:35] [02miraheze/puppet] 07Southparkfan 03edf2b23 - Update varnish.default [01:57:03] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03allow-cache-size-change [+1/-0/±0] 13https://git.io/JvCLP [01:57:04] [02miraheze/puppet] 07Southparkfan 0327cf6f2 - Create cp8.yaml [01:57:18] [02puppet] 07Southparkfan opened pull request 03#1198: Allow cache size change - 13https://git.io/JvCLX [01:57:36] paladox ^ [01:57:54] you forgot to rename varnish.default SPF|Cloud [01:58:00] and to move it to templates/ [01:58:22] oops ;) [01:59:06] cp2 nginx just failed on me, so seems dead at the moment [01:59:20] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03allow-cache-size-change [+1/-1/±0] 13https://git.io/JvCLM [01:59:21] [02miraheze/puppet] 07Southparkfan 03de146bf - Rename modules/varnish/files/varnish/varnish.default to modules/varnish/templates/varnish.default.erb [01:59:23] [02puppet] 07Southparkfan synchronize pull request 03#1198: Allow cache size change - 13https://git.io/JvCLX [01:59:37] should be better now? [02:01:37] [02puppet] 07paladox synchronize pull request 03#1198: Allow cache size change - 13https://git.io/JvCLX [02:01:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03allow-cache-size-change [+0/-0/±1] 13https://git.io/JvCLS [02:01:40] [02miraheze/puppet] 07paladox 0396ec062 - Update varnish.default.erb [02:01:44] [02puppet] 07paladox closed pull request 03#1198: Allow cache size change - 13https://git.io/JvCLX [02:01:45] SPF|Cloud merged [02:01:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-1/±1] 13https://git.io/JvCL9 [02:01:47] [02miraheze/puppet] 07Southparkfan 03632a650 - Allow cache size change (#1198) * Update init.pp * Update varnish.default * Create cp8.yaml * Rename modules/varnish/files/varnish/varnish.default to modules/varnish/templates/varnish.default.erb * Update varnish.default.erb Co-authored-by: paladox [02:01:52] [02puppet] 07paladox deleted branch 03allow-cache-size-change - 13https://git.io/vbiAS [02:01:53] [02miraheze/puppet] 07paladox deleted branch 03allow-cache-size-change [02:02:41] whatever is causing the cp2 issues, it only seems to impact cp2 [02:02:52] but fortunately we're installing its replacement right now [02:05:25] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCLb [02:05:26] [02miraheze/puppet] 07Southparkfan 03cde16cd - fix puppet error [02:06:07] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 6973 MB (36% inode=93%); [02:06:10] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:06:52] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [02:06:54] awesome :P [02:07:04] SPF|Cloud i think we're ready to pool in cp8 now :) [02:08:07] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:10:25] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCLp [02:10:26] [02miraheze/dns] 07Southparkfan 03dafe51b - Pool in cp8.. for real [02:14:28] \o/ [02:14:29] SPF|Cloud: should probs also remove cp2 [02:14:45] paladox|UKInEU: want to see how cp8 is performing for now [02:16:50] Ok [02:17:19] it's too late for me, so cya [02:19:09] I’m [02:19:10] Going to go too [02:54:19] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1538 bytes in 0.485 second response time [02:54:25] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:54:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:55:27] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:13:32] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:15:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [03:15:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [03:16:08] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [03:16:49] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:17:28] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:17:30] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 12% [03:19:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:20:18] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43457 bytes in 6.226 second response time [03:20:37] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15287 bytes in 5.780 second response time [03:21:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:22:49] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:23:01] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:24:59] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:25:01] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [03:25:47] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 5.221 second response time [03:27:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [03:30:06] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:31:22] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:31:41] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [03:32:45] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [03:35:58] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:37:13] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43457 bytes in 5.801 second response time [03:37:25] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:38:47] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 7.453 second response time [03:40:29] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:43:34] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [03:43:35] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:45:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:46:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:46:30] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15289 bytes in 4.994 second response time [03:47:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:49:27] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 6.613 second response time [03:53:33] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:55:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [04:00:58] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:03:16] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15289 bytes in 1.640 second response time [04:07:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [04:21:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:22:01] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [04:24:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:28:59] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [04:30:28] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:36:05] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:37:31] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:38:24] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15287 bytes in 1.580 second response time [04:39:56] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:42:36] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [04:47:13] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [04:54:53] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:56:58] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [05:00:08] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:02:47] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [05:06:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:08:00] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.35, 2.55, 1.71 [05:10:02] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.50, 1.92, 1.58 [05:11:46] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:12:20] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.02, 1.61, 1.51 [05:17:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [05:18:57] PROBLEM - wiki.kujum.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kujum.org' expires in 15 day(s) (Thu 27 Feb 2020 05:14:27 AM GMT +0000). [05:19:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCYN [05:19:12] [02miraheze/ssl] 07MirahezeSSLBot 03919ada1 - Bot: Update SSL cert for wiki.kujum.org [05:22:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:27:50] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [05:29:27] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [05:32:10] RECOVERY - wiki.kujum.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kujum.org' will expire on Mon 11 May 2020 04:19:05 AM GMT +0000. [05:34:43] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.16, 2.96, 2.14 [05:39:19] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.41, 1.44, 1.70 [05:39:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:44:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [05:48:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:51:16] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:53:59] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 2.687 second response time [05:59:31] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [06:04:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:05:05] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:07:30] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [06:13:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [06:14:13] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:15:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:20:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [06:23:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:23:27] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:23:29] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown) [06:25:01] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:26:58] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2948 MB (12% inode=94%); [06:27:23] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 1.155 second response time [06:27:48] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [06:32:13] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:32:46] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:19] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [06:42:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [06:44:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:04:42] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:10:23] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:12:22] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [07:45:31] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:50:27] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.45, 2.08, 1.25 [07:52:37] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.33, 1.42, 1.11 [07:53:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:14:35] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:31:11] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:51:15] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:53:45] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [09:25:18] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:27:20] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [09:47:19] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.50, 2.70, 1.46 [09:53:25] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.28, 1.61, 1.45 [10:17:04] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:18:20] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[sdiy.info],File[sdiy.info_private] [10:19:02] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (1.6 weeks ago) [10:24:32] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:33:39] PROBLEM - www.bushcraftpedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.bushcraftpedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [10:52:52] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2646 MB (10% inode=94%); [11:06:18] RECOVERY - www.bushcraftpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'server.notrace.nl' will expire on Fri 08 May 2020 11:29:19 AM GMT +0000. [11:31:35] PROBLEM - www.bushcraftpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:05] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [11:43:04] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [11:58:32] RECOVERY - www.bushcraftpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'server.notrace.nl' will expire on Fri 08 May 2020 11:29:19 AM GMT +0000. [12:22:58] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.54, 2.76, 1.55 [12:24:55] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.55, 1.89, 1.38 [12:26:53] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.13, 1.30, 1.22 [13:11:45] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.79, 1.93, 1.12 [13:13:43] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.55, 1.41, 1.03 [15:35:23] PROBLEM - wiki.tallguysfree.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tallguysfree.com' expires in 15 day(s) (Thu 27 Feb 2020 03:31:39 PM GMT +0000). [15:35:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCEV [15:35:38] [02miraheze/ssl] 07MirahezeSSLBot 03494d28d - Bot: Update SSL cert for wiki.tallguysfree.com [15:36:52] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.77, 2.75, 1.49 [15:38:53] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.57, 1.93, 1.35 [15:40:35] * hispano76 greetings [15:40:55] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.56, 1.47, 1.24 [15:43:23] RECOVERY - wiki.tallguysfree.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tallguysfree.com' will expire on Mon 11 May 2020 02:35:30 PM GMT +0000. [15:48:53] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.47, 2.05, 1.53 [15:56:52] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.47, 1.82, 1.84 [15:58:52] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.41, 1.38, 1.68 [16:02:06] paladox|UKInEU: JohnLewis it seems cp8 maybe having some load issue [16:08:52] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.32, 1.82, 1.63 [16:10:52] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.51, 1.35, 1.48 [16:14:26] paladox|UKInEU: what's matomo going to? [16:15:34] and salt master [16:19:18] Matomo is mont1 and salt master I think is the same [16:22:03] Hello flu-pm! If you have any questions, feel free to ask and someone should answer soon. [16:22:52] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.37, 2.05, 1.67 [16:23:21] thx paladox|UKInEU [16:24:55] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.53, 1.46, 1.49 [16:31:26] Hi, I'm observing /zero/ Google visibility of wiki https://anwaltspostfach.miraheze.org Anything that can be done about that? [16:31:35] [ Anwaltspostfach unter Linux ] - anwaltspostfach.miraheze.org [16:33:19] Example search terms are "signaturanwendungskomponente icedtea" that should find https://anwaltspostfach.miraheze.org/wiki/Signaturanwendungskomponente [16:33:21] [ Signaturanwendungskomponente – Anwaltspostfach unter Linux ] - anwaltspostfach.miraheze.org [16:33:33] But Google doesn't. [16:36:08] flu-pm: when was the page(s)/wiki created [16:36:57] Zppix, beginning to mid january. [16:37:47] Hmm let me see if theres anything obvious if not someone else will have to look into it if its something we can do anything about or i can look into it in a few hours more in depth [16:38:49] Zppix, ok, I'll check back here. Thanks! [16:39:34] flu-pm: note that google controls how it indexes so we may not be able to do anything [16:40:47] Zppix: spf does the google indexes [16:41:40] Zppix, yeah, but why should Google not index that page? My deep hope is that you can do something about that. Otherwise, the wiki's audience would be rather small. :-/. [16:44:00] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.38, 2.00, 1.22 [16:45:58] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.67, 1.13, 1.03 [16:49:00] flu-pm: We've probably not sent new indexes over yet. Give it time and advertise your wiki in places it does index. [16:52:47] RhinosF1, I did advertise it mid Jan. at https://forum.ubuntuusers.de/post/9126099/ Given, Google indexes that forum, would that be enough or should I post a new link somewhere? [16:52:49] [ beaClientSecurity lässt sich nicht installieren › Programme › Ubuntu verwenden › Forum › ubuntuusers.de ] - forum.ubuntuusers.de [16:53:21] You could add it to Gazetters of Wikis on meta.miraheze.org flu-pm [16:54:51] Zppix, will look into that. I gather the problem will be fixed within a couple hours our days. Is that right? [16:54:52] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.32, 1.85, 1.24 [16:56:50] RhinosF1: https://meta.miraheze.org/w/index.php?title=Tech:Cloud_Server_Comparison&diff=96461&oldid=96460 [16:56:53] [ Difference between revisions of "Tech:Cloud Server Comparison" - Miraheze Meta ] - meta.miraheze.org [16:56:55] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.37, 1.31, 1.12 [16:56:57] fyi for you [16:59:59] JohnLewis: thx [17:00:12] and I removed Salt from mon just now [17:00:24] ok [17:07:05] flu-pm: i cant see anything obvious as far as i know probably will have to wait for SPF|Cloud [17:10:46] Zppix: it will be indexed when Google finds it, all that can be said really [17:10:58] This is a problem to be taken to Google and to ask them about really [17:11:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvC2Z [17:11:44] [02miraheze/puppet] 07paladox 03562f1ac - Add test2 to varnish (this can only be used with headers set) [17:11:46] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:11:47] [02puppet] 07paladox opened pull request 03#1199: Add test2 to varnish - 13https://git.io/JvC2n [17:12:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvC2C [17:12:26] [02miraheze/puppet] 07paladox 0371a1603 - Update stunnel.conf [17:12:27] [02puppet] 07paladox synchronize pull request 03#1199: Add test2 to varnish - 13https://git.io/JvC2n [17:13:17] [02puppet] 07paladox closed pull request 03#1199: Add test2 to varnish - 13https://git.io/JvC2n [17:13:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvC28 [17:13:20] [02miraheze/puppet] 07paladox 034feb408 - Add test2 to varnish (#1199) * Add test2 to varnish (this can only be used with headers set) * Update stunnel.conf [17:13:21] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [17:13:23] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:13:29] JohnLewis, I thought you already found something on your side when RhinosF1 said "we've probably not sent new indexes over yet". No? [17:14:09] There's nothing on our side for the matter [17:14:24] OK. Will wait and see. [17:17:25] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [17:17:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvC2g [17:17:56] [02miraheze/puppet] 07paladox 033831a03 - fix port [17:19:21] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:39:01] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.69, 2.20, 1.33 [17:40:59] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.58, 1.80, 1.28 [17:42:59] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.20, 1.54, 1.25 [17:49:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [17:51:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:05:07] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCVk [18:05:09] [02miraheze/dns] 07paladox 0383a4259 - Add mw5 to dns [18:12:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCVc [18:13:01] [02miraheze/dns] 07paladox 034561107 - Add mw[67] to dns [18:23:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCVV [18:23:38] [02miraheze/puppet] 07paladox 0373a67ed - Update mw4.yaml [18:24:04] I understand that "Download as PDF" does not work on private sites. Is that a Miraheze or MediaWiki restriction? Is it likely to be fixed? [18:30:49] MikeV: It’s to do with parsoid. [18:37:03] It's nothing to do with parsoid [18:37:27] MikeV: It's an upstream issue with the service developers (Wikimedia) [18:38:44] ^ [18:41:53] JohnLewis, Thanks. Is it likely to be addressed in future releases? [18:42:07] paladox|UKInEU: ^ [18:42:27] MikeV upstream have a task for it, but there's no fixed date for them to add support. [18:59:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvCwg [18:59:30] [02miraheze/puppet] 07paladox 032b44448 - Create mw5.yaml [18:59:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvCw2 [18:59:40] [02miraheze/puppet] 07paladox 03886f721 - Create mw6.yaml [18:59:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvCwV [18:59:53] [02miraheze/puppet] 07paladox 03f9ce93d - Create mw7.yaml [19:21:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [19:23:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:24:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCr8 [19:24:38] [02miraheze/puppet] 07paladox 03e38fb58 - Add mw[4567] to firewalls [19:24:40] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:24:41] [02puppet] 07paladox opened pull request 03#1200: Add mw[4567] to firewalls - 13https://git.io/JvCrB [19:27:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCro [19:27:42] [02miraheze/puppet] 07paladox 03ec993ae - Update varnish.pp [19:27:43] [02puppet] 07paladox synchronize pull request 03#1200: Add mw[4567] to firewalls - 13https://git.io/JvCrB [19:30:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCrD [19:30:55] [02miraheze/puppet] 07paladox 035371760 - Update irc.pp [19:30:57] [02puppet] 07paladox synchronize pull request 03#1200: Add mw[4567] to firewalls - 13https://git.io/JvCrB [19:33:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCrF [19:33:13] [02miraheze/puppet] 07paladox 035dac1b1 - Update firewall.yaml [19:33:14] [02puppet] 07paladox synchronize pull request 03#1200: Add mw[4567] to firewalls - 13https://git.io/JvCrB [19:35:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCrj [19:35:59] [02miraheze/puppet] 07paladox 0359cdabf - Update services.pp [19:36:01] [02puppet] 07paladox synchronize pull request 03#1200: Add mw[4567] to firewalls - 13https://git.io/JvCrB [19:36:48] [02puppet] 07paladox closed pull request 03#1200: Add mw[4567] to firewalls - 13https://git.io/JvCrB [19:36:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JvCov [19:36:51] [02miraheze/puppet] 07paladox 039969741 - Add mw[4567] to firewalls (#1200) * Add mw[4567] to firewalls * Update varnish.pp * Update irc.pp * Update firewall.yaml * Update services.pp [19:36:52] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:36:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:40:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCos [19:40:40] [02miraheze/puppet] 07paladox 030827230 - Add mw[4567] to varnish (not pulling this in till we are ready to move all prod to these new machines) [19:40:41] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:40:43] [02puppet] 07paladox opened pull request 03#1201: Add mw[4567] to varnish - 13https://git.io/JvCoG [19:42:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCo8 [19:42:16] [02miraheze/puppet] 07paladox 031cb0ac5 - Update default.vcl [19:42:17] [02puppet] 07paladox synchronize pull request 03#1201: Add mw[4567] to varnish - 13https://git.io/JvCoG [19:42:58] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:43:01] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:43:10] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:43:14] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:44:04] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:44:15] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:44:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvCo0 [19:44:28] [02miraheze/puppet] 07paladox 034ef4667 - Update default.vcl [19:44:28] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:44:30] [02puppet] 07paladox synchronize pull request 03#1201: Add mw[4567] to varnish - 13https://git.io/JvCoG [19:45:04] oh [19:46:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvCoK [19:46:25] [02miraheze/puppet] 07paladox 030ca0170 - Fix dupes [19:46:26] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [19:46:28] [02puppet] 07paladox opened pull request 03#1202: Fix dupes - 13https://git.io/JvCo6 [19:47:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvCoy [19:47:13] [02miraheze/puppet] 07paladox 036f552f1 - Update irc.pp [19:47:15] [02puppet] 07paladox synchronize pull request 03#1202: Fix dupes - 13https://git.io/JvCo6 [19:48:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvCoH [19:48:07] [02miraheze/puppet] 07paladox 03ec96d58 - Update varnish.pp [19:48:08] [02puppet] 07paladox synchronize pull request 03#1202: Fix dupes - 13https://git.io/JvCo6 [19:48:21] [02puppet] 07paladox closed pull request 03#1202: Fix dupes - 13https://git.io/JvCo6 [19:48:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JvCo7 [19:48:24] [02miraheze/puppet] 07paladox 035dc3f71 - Fix dupes (#1202) * Fix dupes * Update irc.pp * Update varnish.pp [19:48:44] paladox|UKInEU: how's the migration going? [19:48:53] It's going fine [19:49:08] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:49:08] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [19:49:10] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [19:51:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvCop [19:51:04] [02miraheze/puppet] 07paladox 039bc43f1 - Add mediawiki role to mw4 [19:52:02] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:52:16] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:53:10] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:53:16] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:54:26] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:55:08] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvC6Y [20:23:09] [02miraheze/puppet] 07paladox 0378e1600 - Temporarily add 'new_server' hiera config [20:23:11] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [20:23:12] [02puppet] 07paladox opened pull request 03#1203: Temporarily add 'new_server' hiera config - 13https://git.io/JvC6O [20:24:21]