[00:05:09] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.57, 1.99, 1.53 [00:07:15] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.86, 1.60, 1.44 [01:22:00] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.55, 1.84, 1.41 [01:26:14] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.67, 1.74, 1.47 [01:28:15] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.08, 1.41, 1.37 [03:12:12] PROBLEM - spiral.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:19:05] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Thu 20 Aug 2020 15:07:11 GMT +0000. [03:24:06] PROBLEM - yellowiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'yellowiki.xyz' expires in 15 day(s) (Tue 28 Jul 2020 03:20:38 GMT +0000). [03:31:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJYFe [03:31:03] [02miraheze/ssl] 07MirahezeSSLBot 030eb308f - Bot: Update SSL cert for yellowiki.xyz [03:44:13] RECOVERY - yellowiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'yellowiki.xyz' will expire on Sat 10 Oct 2020 02:30:55 GMT +0000. [04:40:08] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 4.72, 3.73, 2.25 [05:00:29] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.16, 1.59, 2.00 [05:02:31] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.29, 1.83, 2.02 [05:04:31] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.72, 1.40, 1.84 [05:06:31] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.03, 1.55, 1.83 [05:20:46] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.12, 1.79, 1.95 [05:28:55] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.89, 1.26, 1.67 [06:20:34] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.83, 1.52, 1.21 [06:22:38] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.90, 1.25, 1.15 [06:58:44] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:26] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 20 Aug 2020 14:54:45 GMT +0000. [08:23:27] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:23:41] Reception123: ^ [08:23:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [08:24:03] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time [08:24:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [08:24:14] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:24:15] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [08:24:51] Reception123, SPF|Cloud, paladox: down [08:24:54] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [08:25:14] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 2 backends are down. mw5 mw6 [08:25:28] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw5 mw6 [08:25:39] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:25:40] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [08:25:50] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.303 second response time [08:26:21] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [08:26:26] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw7 [08:27:13] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 42% [08:29:47] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 20% [08:29:56] Reception123, SPF|Cloud, paladox: widespread 502/3 reports and slow service. Please assist. [08:30:31] RhinosF1: I got no access until the afternoon today [08:30:33] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:30:50] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:30:58] Reception123: ack [08:31:06] RhinosF1: Have you emailed SRE? [08:31:21] Maybe it's related to the new db servers [08:31:23] Reception123: will do now [08:31:31] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:31:55] RhinosF1: I got to leave in 5 minutes so I will take a very quick look for anything obvious but I can't do more unfortunately [08:32:01] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:32:05] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 5.031 second response time [08:32:55] Email and thanks Reception123 [08:33:02] emailed [08:33:14] RhinosF1: Any suggestions on what I can quickly check? [08:33:43] Reception123: not a clue, 3 mws down and 2 cps [08:33:58] Nothing even consitent [08:34:09] RhinosF1: I can SSH to them fine [08:34:35] Servers are flapping like mad [08:34:43] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [08:34:46] Reception123: see -staff for icinga status [08:34:46] I can check Promox but that's it [08:35:11] Ok [08:35:31] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:35:41] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.657 second response time [08:36:40] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.344 second response time [08:37:44] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [08:37:59] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [08:38:36] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:38:47] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 0.994 second response time [08:40:09] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 59% [08:40:27] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 41% [08:40:47] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:41:13] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [08:41:37] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:42:50] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:42:55] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:43:28] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.012 second response time [08:43:43] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 34% [08:43:44] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.005 second response time [08:44:21] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:45:05] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:45:08] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:45:35] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 36% [08:46:20] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.318 second response time [08:46:34] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [08:47:11] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:47:11] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 8.441 second response time [08:47:37] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 0.003 second response time [08:47:48] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 0.313 second response time [08:49:13] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [08:49:30] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 3.894 second response time [08:50:05] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:50:09] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 9.091 second response time [08:50:11] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.000 second response time [08:50:15] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:52:40] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 35% [08:52:51] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:53:18] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:53:23] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.004 second response time [08:55:25] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:55:28] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 4.528 second response time [08:55:48] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:55:55] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.007 second response time [08:55:59] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.752 second response time [08:56:29] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:57:43] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time [08:58:54] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 0.326 second response time [08:59:23] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:00:27] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 3.715 second response time [09:00:28] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 4.108 second response time [09:03:11] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.002 second response time [09:03:20] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:03:44] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 1.154 second response time [09:04:28] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.008 second response time [09:04:34] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.238 second response time [09:05:09] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [09:05:13] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.997 second response time [09:05:48] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:05:52] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:06:46] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.315 second response time [09:08:13] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 8.929 second response time [09:09:31] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:10:43] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 5.447 second response time [09:10:53] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.017 second response time [09:10:55] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.004 second response time [09:10:57] !log restarted php7.3-fpm on all mediawiki servers [09:11:26] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [09:11:42] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 0.998 second response time [09:11:47] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [09:12:11] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [09:15:26] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time [09:15:52] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.237 second response time [09:16:16] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:16:38] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:16:43] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 3 backends are down. mw4 mw6 mw7 [09:16:44] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:16:54] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:17:21] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [09:17:39] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 44% [09:17:46] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [09:17:48] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [09:17:52] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 0.003 second response time [09:17:59] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.004 second response time [09:18:16] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 0.325 second response time [09:18:42] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.315 second response time [09:19:13] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 6.239 second response time [09:20:00] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 37% [09:20:07] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 21% [09:21:29] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:23:06] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time [09:23:23] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.757 second response time [09:23:42] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:24:01] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:24:20] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:24:25] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.004 second response time [09:24:25] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [09:25:24] PROBLEM - jobrunner1 Redis Process on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server' [09:25:27] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [09:25:50] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 1.004 second response time [09:27:52] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [09:28:34] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 6.028 second response time [09:28:44] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.004 second response time [09:28:46] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 0.355 second response time [09:28:56] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:30:12] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:30:23] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 4.267 second response time [09:30:47] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:31:22] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:32:49] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:33:03] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:35:27] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:35:29] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:36:08] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 4.030 second response time [09:36:29] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [09:36:29] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:37:55] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 6.388 second response time [09:38:16] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 0.309 second response time [09:38:43] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:41:37] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 7.720 second response time [09:42:16] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 1.015 second response time [09:42:37] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:42:57] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:42:57] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [09:42:57] RECOVERY - jobrunner1 Redis Process on jobrunner1 is OK: PROCS OK: 1 process with args 'redis-server' [09:43:15] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.000 second response time [09:44:40] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 0.221 second response time [09:45:50] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.238 second response time [09:48:08] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:48:16] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:49:15] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:49:25] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:50:25] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.013 second response time [09:50:31] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.004 second response time [09:50:37] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.332 second response time [09:52:46] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:53:04] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:54:19] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15603 bytes in 1.003 second response time [09:54:29] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.007 second response time [09:54:30] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15609 bytes in 0.310 second response time [09:54:52] PROBLEM - jobrunner1 Redis Process on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server' [09:55:05] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.003 second response time [09:55:15] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.004 second response time [09:55:30] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 1.000 second response time [09:55:35] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.316 second response time [09:55:41] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15595 bytes in 0.319 second response time [09:56:12] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [09:56:23] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [09:56:23] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [09:56:44] PROBLEM - db9 MariaDB on db9 is CRITICAL: Can't connect to MySQL server on 'db9.miraheze.org' (111) [09:57:05] JohnLewis, SPF|Cloud: ^ [09:57:29] That's me [09:57:29] JohnLewis: 2020-07-08 - 23:28:04UTC tell JohnLewis can you please investigate why https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L1802 and https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L224 is still resulting in https://i.imgur.com/dggv3eZ.jpg [09:57:29] JohnLewis: 2020-07-08 - 23:29:14UTC tell JohnLewis sorry, second link should be [09:57:31] JohnLewis: 2020-07-08 - 23:30:59UTC tell JohnLewis sorry, my phone is screwing up. The second link in my first message should be https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L2224 [09:59:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [10:00:27] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 56% [10:01:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 51% [10:02:53] RECOVERY - jobrunner1 Redis Process on jobrunner1 is OK: PROCS OK: 1 process with args 'redis-server' [10:04:13] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 2% [10:04:31] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 2% [10:04:34] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 3% [10:04:39] RECOVERY - db9 MariaDB on db9 is OK: Uptime: 461 Threads: 25 Questions: 35569 Slow queries: 1099 Opens: 3012 Flush tables: 1 Open tables: 3006 Queries per second avg: 77.156 [10:04:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:04:46] JohnLewis: we're coming back [10:04:51] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [10:04:55] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [10:05:29] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 7% [10:05:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:05:37] Yeah, errors in logs are significantly down - if nothing flares up in the next 10 minutes I think this is resolved [10:05:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [10:05:49] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [10:06:23] JohnLewis: agreed [10:06:53] We're back to 0 unknown alerts [10:07:04] 2 handled and John stopping puppet on test2 [10:08:30] We still need to work out why [10:30:19] Hello, gadgets doesn't work for me on preferences page. Its normal? https://meta.miraheze.org/wiki/Special:Preferences [10:30:20] [ Log in - Miraheze Meta ] - meta.miraheze.org [10:30:37] Oh, on WMF it is same. [10:32:15] Yes it is expected [12:42:54] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJOfo [12:42:56] [02miraheze/CreateWiki] 07JohnFLewis 03cff908e - Introduce $wikiDBClusters to WikiInitialise [13:10:00] Can be gadget in css, or js only? [13:11:20] Both I think [13:11:36] Ok, Thanks. [13:21:25] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.00, 3.15, 1.85 [13:21:43] Yeah [13:21:56] After simple test it works [13:25:27] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.59, 1.73, 1.56 [13:26:08] Do you have any skin usage statistics? [13:26:42] Vector, Modern,... [13:29:26] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.86, 1.46, 1.52 [13:36:12] @MrJaroslavik: for meta? [13:37:30] Yes [13:39:42] Someone can get them [13:40:10] JohnLewis, Zppix: i'm mobile, could someone check the prefs table? ^ [13:52:59] https://www.irccloud.com/pastebin/vjzBb3uH/ [13:53:00] [ Snippet | IRCCloud ] - www.irccloud.com [13:53:20] @MrJaroslavik ^ for all skins that are not 'vector' [13:53:49] Thanks JohnLewis [13:54:04] Im mobile too [13:54:24] Thanks! [13:55:06] So other users have vector like as default skin, right? [13:55:32] If they havent changed it and the wiki doesnt set another default iirc yes [13:56:06] @MrJaroslavik: for meta, yes [13:56:16] On Meta is vector as default, propadly. Thanks again! [13:57:28] Yes it is [14:04:16] This can be good feature request on WMF phab, like is Special:GadgetUsage, can be Special:SkinsUsage. [14:04:50] Could be [14:05:57] Added to my todo list :D. [14:06:39] Nice [14:18:38] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJOTs [14:18:39] [02miraheze/mediawiki] 07JohnFLewis 037a167e7 - CW [14:42:01] Hi [14:42:56] Someone is work on phabricator requests? [14:47:25] @Avengium (Ángel) I am on SRE duty yes (for today). But which requests do you have in mind? [14:56:28] i was waiting for the t5890 , but i was asking to know if someone was resolving requests [14:57:00] in general, not only mine ^^ [15:00:19] I'd like to adopt this wiki because the only admin/owner hasn't been active for half a year. https://atrociousgameplay.miraheze.org [15:00:20] [ Atrocious Gameplay Miraheze ] - atrociousgameplay.miraheze.org [15:07:06] @Void Hello Void, can you delete cvt group on Meta, please? [15:11:08] Except for anything particularly urgent, I'm going to be away for the day [15:11:39] Will be back late tomorrow [15:11:54] Ok [15:12:10] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJOky [15:12:12] [02miraheze/CreateWiki] 07JohnFLewis 03fcd20af - Implement multi-db support fully into WikiManager [15:12:46] I'd like to adopt this wiki because the only admin/owner hasn't been active for half a year. https://atrociousgameplay.miraheze.org [15:12:47] [ Atrocious Gameplay Miraheze ] - atrociousgameplay.miraheze.org [15:16:56] Hi [15:17:06] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:17:29] When are more active the staff and admins? weekdays or weekends? [15:17:41] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:17:44] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:18:03] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:20:41] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.45, 2.05, 1.40 [15:22:44] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.82, 1.60, 1.32 [15:24:46] If it's the wiki I want to adopt, the admin is never active. [15:51:32] about wiki templates, why if i export a infobox, when importing the file in my wiki appear several templates? [15:55:58] Because the infobox depends on them [15:58:00] ok [16:26:06] @Inkster I've messaged Void regarding the outstanding adoption requests; he will work through them in the order received. [16:27:11] for adopting wikis i think is better to put a request on meta RfA request for adoption [16:27:17] A pro tip to a successful adoption, make sure you proactively editing on the wiki, adding pages, uploading files, and so forth, in addition to vandalism reverts. [16:35:36] I edit there. [16:43:48] | I edit there. @Inkster Okay, that's great actually. As long as you're active there, it shouldn't be an issue. 🙂 [16:56:53] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 4.59, 2.75, 1.57 [16:57:58] hi [16:58:08] someone's here? [16:58:19] Me [16:58:58] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.08, 1.97, 1.43 [17:01:43] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJOqW [17:01:45] [02miraheze/CreateWiki] 07JohnFLewis 030ee4826 - Implement multi-db support fully into WikiManager [17:03:07] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.81, 1.46, 1.35 [17:03:33] I have a question. A friend told me that there are wikis that have templates transcluded from other wikis, without having to import them. That's right? [17:03:43] ye [17:04:01] It's the enable scary transcluding option [17:05:15] and where is it? the option [17:06:17] in ManageWiki/settings [17:07:54] i supposed in settings, but i was about to ask in which section [17:08:27] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJOq0 [17:08:28] [02miraheze/mediawiki] 07JohnFLewis 030443f43 - CW [17:08:45] JohnLewis: it's prod then? (Once puppet runs) [17:09:14] RhinosF1: the backend code is, but needs config to make it actually be used [17:09:29] but most of that code has been there for months [17:10:07] i found the option. it was enabled [17:10:50] so i could transclude templates like infoboxes or like [[Template:Listen]] for the audios [17:10:57] JohnLewis: okay [17:11:09] You could Avengium [17:12:42] My friend is saying me that the template he is trying to put still don't work [17:13:08] You might need to get an interwiki admin turn the transclude option on [17:13:14] hispano76, JohnLewis: ^ [17:13:29] but the setting is already ticked on [17:13:41] here it appears on: https://tuscriaturas.miraheze.org/wiki/Especial:ManageWiki/settings#mw-section-edit [17:13:43] [ Administra los ajustes adicionales de este wiki - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [17:15:09] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJOqr [17:15:11] [02miraheze/mw-config] 07JohnFLewis 03a1ae1af - Add multi-db setup for CreateWiki [17:16:29] Avengium debe habilitarse también acá https://tuscriaturas.miraheze.org/wiki/Especial:Interwiki [17:16:29] Avengium: it'll be the interwiki table [17:16:31] [ Ver datos de interwikis - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [17:16:37] Thanks hispano76 [17:17:18] oh, i didn't know we needed interwiki too [17:17:45] hispano76 can turn it on for your wikis and advise [17:18:00] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:19:28] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 5.14, 3.02, 1.94 [17:19:44] Avengium necesitaría más detalles ¿de que wiki quieres transcluir? [17:21:17] de Wikipedia en español para tuscriaturas wiki [17:21:52] y de wikipedia en ingles para yourcreatures.miraheze (de paso) por ahorrarme otra conversación [17:22:09] *wikimedia (creo) [17:22:54] ¿Cuál es el prefijo que quieren usar en cada interwiki? [17:23:33]