[00:03:04] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.91, 21.66, 16.79 [00:05:03] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.65, 19.06, 16.42 [00:35:55] PROBLEM - gluster2 APT on gluster2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [00:39:03] [02mw-config] 07Universal-Omega edited a comment on pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUI8s [00:42:46] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [01:14:20] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:52] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [01:14:58] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 4.09, 2.14, 0.91 [01:15:07] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 97% [01:15:18] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is WARNING: WARNING - NGINX Error Rate is 55% [01:15:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [01:15:35] PROBLEM - cp9 Current Load on cp9 is CRITICAL: CRITICAL - load average: 4.87, 2.63, 1.28 [01:15:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [01:17:04] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 50% [01:17:15] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 0% [01:17:23] PROBLEM - cp9 HTTPS on cp9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:24] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: HTTP CRITICAL - No data received from host [01:18:28] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: HTTP CRITICAL - No data received from host [01:18:31] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: HTTP CRITICAL - No data received from host [01:18:32] PROBLEM - cp7 Stunnel Http for test2 on cp7 is CRITICAL: HTTP CRITICAL - No data received from host [01:18:36] PROBLEM - cp7 Stunnel Http for mon1 on cp7 is CRITICAL: HTTP CRITICAL - No data received from host [01:18:53] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [01:19:05] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 6% [01:20:35] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 9.295 second response time [01:20:41] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 9.76, 8.06, 4.91 [01:21:18] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [01:22:00] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.95, 7.58, 5.04 [01:22:32] RECOVERY - cp7 Stunnel Http for test2 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15655 bytes in 0.030 second response time [01:22:36] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.76, 7.86, 5.21 [01:22:55] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [01:23:12] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.95, 7.38, 4.80 [01:23:38] RECOVERY - cp9 HTTPS on cp9 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1936 bytes in 7.256 second response time [01:23:58] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.39, 6.72, 5.06 [01:24:31] RECOVERY - cp7 Stunnel Http for mon1 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 30379 bytes in 0.013 second response time [01:24:33] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.020 second response time [01:24:33] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15661 bytes in 0.009 second response time [01:24:51] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [01:25:03] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1936 bytes in 8.375 second response time [01:25:09] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.18, 5.68, 4.48 [01:25:20] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [01:26:36] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 4.38, 6.17, 5.14 [01:27:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:27:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:30:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.42, 7.83, 6.09 [01:31:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [01:31:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2607:5300:205:200::2ac4/cpweb [01:33:09] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.41, 6.47, 5.15 [01:33:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.85, 21.12, 17.00 [01:33:54] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.23, 6.94, 5.97 [01:34:36] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 5.98, 7.22, 6.29 [01:35:11] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.30, 5.21, 4.84 [01:35:12] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.84, 21.98, 17.79 [01:36:36] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 10.23, 8.75, 6.98 [01:38:02] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 13.15, 8.89, 6.88 [01:38:24] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 7.49, 6.47, 4.57 [01:44:03] @System Administrators I am experiencing very slow page load times and even timeouts on Meta, and have been for the last half hour or so. I'm not sure if there could be an issue with one of the servers or perhaps the cache proxies. [01:44:15] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 8.40, 7.21, 5.48 [01:44:35] @System Administrators I do see socket timeouts on mw7 and cp9 on Icinga (https://icinga.miraheze.org/dashboard), so that, perhaps, possibly, explains the timeouts and very slow response times. [01:44:36] [ Icinga Web 2 Login ] - icinga.miraheze.org [01:45:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:45:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:46:14] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 5.40, 6.54, 5.45 [01:46:34] ns1 and ns2 were also showing as "down" or unreachable in Icinga (https://icinga.miraheze.org/dashboard#!/monitoring/service/show?host=ns2&service=ns2 GDNSD Datacenters) [01:46:36] [ Icinga Web 2 Login ] - icinga.miraheze.org [01:48:00] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 2.83, 6.57, 7.23 [01:48:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 4.34, 6.70, 7.47 [01:50:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.70, 7.72, 7.76 [01:54:36] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.98, 6.75, 7.44 [01:56:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [01:56:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.81, 7.18, 7.49 [01:57:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [01:58:20] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:00:36] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 6.52, 6.92, 7.33 [02:00:49] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [02:01:56] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.15, 6.96, 6.99 [02:02:36] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 9.51, 8.00, 7.67 [02:03:55] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.85, 7.44, 7.18 [02:04:36] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 4.56, 6.90, 7.33 [02:05:53] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 2.86, 5.72, 6.58 [02:07:01] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [02:08:36] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.99, 7.49, 7.40 [02:08:55] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [02:09:54] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.44, 7.04, 7.01 [02:12:36] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.23, 7.40, 7.38 [02:13:58] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.60, 6.33, 6.71 [02:15:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:16:02] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 7.22, 7.04, 6.21 [02:16:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:17:58] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.16, 5.29, 5.67 [02:18:36] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 2.87, 5.60, 6.67 [02:21:33] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (11 critical updates). [02:23:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.15, 21.81, 23.74 [02:26:55] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 0.21, 0.54, 1.96 [02:27:35] PROBLEM - cp9 Current Load on cp9 is WARNING: WARNING - load average: 0.48, 0.70, 1.86 [02:29:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.99, 15.27, 20.14 [02:29:34] RECOVERY - cp9 Current Load on cp9 is OK: OK - load average: 0.24, 0.55, 1.66 [02:30:55] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.17, 0.31, 1.54 [02:36:54] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [03:05:28] PROBLEM - rdb2 APT on rdb2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [03:20:56] [02mw-config] 07Universal-Omega commented on pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUIrU [03:22:35] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [03:44:15] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 2.54, 6.13, 3.35 [03:44:28] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [03:46:17] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [03:50:06] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.18, 0.66, 1.60 [03:55:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/services/compare/ce05eaef4a3d...26f9e5d87832 [03:55:16] [ Comparing ce05eaef4a3d...26f9e5d87832 · miraheze/services · GitHub ] - github.com [03:55:16] [02miraheze/services] 07MirahezeSSLBot 0326f9e5d - BOT: Updating services config for wikis [04:01:01] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [04:02:26] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [04:09:41] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 94 packages available for upgrade (11 critical updates). [05:01:36] PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.esporo.net' expires in 15 day(s) (Sun 13 Sep 2020 04:55:21 GMT +0000). [05:05:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUI6J [05:05:51] [02miraheze/ssl] 07MirahezeSSLBot 031da4f07 - Bot: Update SSL cert for guia.esporo.net [05:08:32] PROBLEM - rdb1 APT on rdb1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [05:33:40] PROBLEM - cp9 Current Load on cp9 is CRITICAL: CRITICAL - load average: 5.94, 3.69, 1.77 [05:37:38] PROBLEM - cp9 Current Load on cp9 is WARNING: WARNING - load average: 0.50, 1.82, 1.43 [05:39:36] RECOVERY - cp9 Current Load on cp9 is OK: OK - load average: 0.34, 1.31, 1.28 [06:01:15] PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 94 packages available for upgrade (11 critical updates). [06:02:18] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:05:04] RECOVERY - gluster1 APT on gluster1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:05:48] RECOVERY - cp3 APT on cp3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:07:06] RECOVERY - mon1 APT on mon1 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [06:10:26] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:10:51] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [06:14:51] RECOVERY - ldap1 APT on ldap1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:15:28] RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:15:42] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIij [06:15:44] [02miraheze/ssl] 07Reception123 03ef1c49f - attempt to fix www.battlecats.ml cert [06:16:01] RECOVERY - mw4 APT on mw4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:16:45] RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:20:32] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:21:34] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [06:28:13] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [06:32:35] RECOVERY - mw7 APT on mw7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:32:51] RECOVERY - db11 APT on db11 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:34:13] RECOVERY - services1 APT on services1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:37:56] RECOVERY - phab1 APT on phab1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:38:36] RECOVERY - test2 APT on test2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:42:11] RECOVERY - cp6 APT on cp6 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:45:09] RECOVERY - db13 APT on db13 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:47:56] RECOVERY - gluster2 APT on gluster2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:50:28] RECOVERY - services2 APT on services2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:56:39] RECOVERY - mw5 APT on mw5 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:59:15] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [06:59:38] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [06:59:52] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIXB [06:59:53] [02miraheze/ssl] 07Reception123 0394ac74f - rm battlecats.ml redirect [07:03:31] [02mw-config] 07Reception123 commented on pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUIXD [07:04:27] [02mw-config] 07Reception123 commented on pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUIX9 [07:12:23] RECOVERY - guia.esporo.net - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.esporo.net' will expire on Thu 26 Nov 2020 04:05:42 GMT +0000. [07:12:23] RECOVERY - lodge.jsnydr.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lodge.jsnydr.com' will expire on Wed 25 Nov 2020 16:40:25 GMT +0000. [07:12:32] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:12:36] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Wed 25 Nov 2020 16:39:29 GMT +0000. [07:12:38] RECOVERY - wiki.consentcraft.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consentcraft.uk' will expire on Wed 25 Nov 2020 14:31:44 GMT +0000. [07:12:42] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 25 Nov 2020 14:43:55 GMT +0000. [07:13:00] RECOVERY - wiki.bullshit.systems - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bullshit.systems' will expire on Wed 25 Nov 2020 16:50:26 GMT +0000. [07:13:07] RECOVERY - disabled.life - LetsEncrypt on sslhost is OK: OK - Certificate 'disabled.life' will expire on Wed 25 Nov 2020 15:48:38 GMT +0000. [07:13:17] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Wed 25 Nov 2020 15:00:30 GMT +0000. [07:13:18] RECOVERY - wiki.hanaoto.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hanaoto.me' will expire on Wed 25 Nov 2020 14:34:27 GMT +0000. [07:13:19] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Wed 25 Nov 2020 14:37:32 GMT +0000. [07:13:25] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:13:29] RECOVERY - wiki.drones4nature.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.drones4nature.info' will expire on Wed 25 Nov 2020 16:29:07 GMT +0000. [07:13:42] RECOVERY - froggy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'froggy.info' will expire on Wed 25 Nov 2020 16:38:41 GMT +0000. [07:13:42] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:13:44] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Wed 25 Nov 2020 14:45:30 GMT +0000. [07:13:54] RECOVERY - meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meregos.com' will expire on Wed 25 Nov 2020 16:47:51 GMT +0000. [07:14:02] RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worlduniversityandschool.org' will expire on Wed 25 Nov 2020 16:37:45 GMT +0000. [07:14:31] RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Wed 25 Nov 2020 16:33:07 GMT +0000. [07:14:44] RECOVERY - wiki.patriam.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.patriam.cc' will expire on Wed 25 Nov 2020 15:59:24 GMT +0000. [07:14:45] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:14:58] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Wed 25 Nov 2020 15:14:32 GMT +0000. [07:15:00] RECOVERY - wiki.ameciclo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ameciclo.org' will expire on Wed 25 Nov 2020 15:40:45 GMT +0000. [07:15:01] RECOVERY - wikipuk.cl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipuk.cl' will expire on Wed 25 Nov 2020 15:04:11 GMT +0000. [07:15:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.81, 19.00, 16.29 [07:16:08] RECOVERY - garrettcountyguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'garrettcountyguide.com' will expire on Wed 11 Nov 2020 17:20:21 GMT +0000. [07:16:26] RECOVERY - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pablojprieto.com.es' will expire on Wed 25 Nov 2020 16:07:38 GMT +0000. [07:16:26] RECOVERY - wiki.staraves-no.cz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.staraves-no.cz' will expire on Wed 25 Nov 2020 15:23:19 GMT +0000. [07:17:06] RECOVERY - madgenderscience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madgenderscience.wiki' will expire on Wed 25 Nov 2020 15:35:11 GMT +0000. [07:17:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.61, 22.85, 17.98 [07:17:14] RECOVERY - bconnected.aidanmarkham.com - LetsEncrypt on sslhost is OK: OK - Certificate 'bconnected.aidanmarkham.com' will expire on Wed 25 Nov 2020 14:25:07 GMT +0000. [07:17:23] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 25 Nov 2020 16:47:33 GMT +0000. [07:17:23] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Wed 25 Nov 2020 15:19:36 GMT +0000. [07:17:23] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:36] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Wed 25 Nov 2020 16:04:37 GMT +0000. [07:17:56] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Wed 25 Nov 2020 15:56:48 GMT +0000. [07:18:08] RECOVERY - wiki.inebriation-confederation.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.inebriation-confederation.com' will expire on Wed 25 Nov 2020 16:38:27 GMT +0000. [07:18:16] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Wed 25 Nov 2020 15:25:13 GMT +0000. [07:18:37] RECOVERY - wiki.besuccess.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.besuccess.com' will expire on Wed 25 Nov 2020 14:16:46 GMT +0000. [07:18:37] RECOVERY - secularknowledge.com - LetsEncrypt on sslhost is OK: OK - Certificate 'secularknowledge.com' will expire on Wed 25 Nov 2020 14:54:34 GMT +0000. [07:18:43] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Wed 25 Nov 2020 16:21:04 GMT +0000. [07:19:02] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Wed 25 Nov 2020 16:14:38 GMT +0000. [07:23:01] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:07] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:25:26] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:32:59] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.39, 23.18, 22.39 [07:35:12] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:35:22] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:36:53] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.78, 17.68, 20.28 [07:46:35] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.47, 20.16, 19.93 [07:48:32] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.10, 17.51, 18.99 [08:00:45] PROBLEM - gluster1 Disk Space on gluster1 is WARNING: DISK WARNING - free space: / 75658 MB (10% inode=84%); [08:03:08] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [09:13:55] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [09:40:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIHj [09:40:22] [02miraheze/services] 07MirahezeSSLBot 0359fe3eb - BOT: Updating services config for wikis [09:54:15] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.39, 20.28, 17.13 [09:56:15] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.71, 18.20, 16.75 [10:48:26] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Sun 13 Sep 2020 10:40:41 GMT +0000). [10:50:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUIdT [10:50:09] [02miraheze/ssl] 07MirahezeSSLBot 033933e2f - Bot: Update SSL cert for wiki.villagecollaborative.net [11:01:55] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Thu 26 Nov 2020 09:50:00 GMT +0000. [11:02:15] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.59, 20.66, 17.74 [11:04:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.28, 21.10, 18.26 [11:06:15] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.23, 19.01, 17.83 [11:17:57] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JUIFO [11:17:58] [02miraheze/ssl] 07Reception123 032a17e7a - add wiki.danvs.net cert [11:35:41] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [12:32:09] PROBLEM - puppet2 APT on puppet2 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (11 critical updates). [12:41:43] JohnLewis: hi. around? [12:42:53] (regarding ManageWiki) [12:48:06] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 18 packages available for upgrade (11 critical updates). [13:23:08] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [13:23:57] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.19, 20.65, 18.47 [13:25:57] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.40, 19.37, 18.29 [13:32:25] PROBLEM - db7 APT on db7 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [13:55:33] * hispano76 greetings [13:58:15] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [13:58:16] PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt [13:58:24] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 28%, RTA = 155.02 ms [13:58:46] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 50% [13:58:59] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 1.74, 2.09, 1.23 [13:59:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [13:59:13] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 40% [13:59:23] PROBLEM - cp9 Current Load on cp9 is WARNING: WARNING - load average: 1.31, 2.00, 1.17 [14:00:01] PROBLEM - speleo.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for speleo.wiki could not be found [14:00:16] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [14:00:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.75, 23.42, 19.97 [14:00:24] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 156.52 ms [14:00:43] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 12% [14:00:58] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 0.94, 1.71, 1.20 [14:01:04] hispano76: Hello! [14:01:08] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 14% [14:01:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:01:23] RECOVERY - cp9 Current Load on cp9 is OK: OK - load average: 0.73, 1.52, 1.09 [14:02:06] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 30% [14:02:56] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.44, 1.32, 1.11 [14:04:21] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.87, 23.33, 21.04 [14:06:58] RECOVERY - speleo.wiki - reverse DNS on sslhost is OK: rDNS OK - speleo.wiki reverse DNS resolves to cp7.miraheze.org [14:07:15] :) [14:08:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.81, 22.60, 21.16 [14:18:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 11.47, 19.92, 21.72 [14:24:17] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.88, 18.36, 20.39 [15:16:50] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [15:18:14] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.50, 23.07, 20.24 [15:24:15] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.02, 23.07, 21.33 [15:34:16] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.23, 17.56, 19.58 [16:12:29] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [16:16:41] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [16:16:45] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [16:48:54] PROBLEM - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [16:50:52] RECOVERY - cp9 HTTP 4xx/5xx ERROR Rate on cp9 is OK: OK - NGINX Error Rate is 12% [17:07:29] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:13] RECOVERY - wiki.seamly.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.net' will expire on Tue 15 Sep 2020 10:08:22 GMT +0000. [17:32:00] PROBLEM - cp9 APT on cp9 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [17:43:49] PROBLEM - gluster2 Disk Space on gluster2 is WARNING: DISK WARNING - free space: / 36717 MB (10% inode=77%); [18:17:10] This a very welcomed change, if it indeed works. [18:22:17] Yeah @Hypercane...I explained why it didn't work...but I'm still working on it. https://phabricator.miraheze.org/T5991#119447 [18:22:18] [ ⚓ T5991 Add $wgRevokePermissions and $wgRestrictionLevels to ManageWiki ] - phabricator.miraheze.org [18:22:53] That's fine, I didn't mean to criticize you if I did. [18:23:16] Just saying this change I've been looking forward to the most. [18:23:17] lol [18:24:10] You didn't. I am still working on getting it working and really hope I can though. Do you have any other config suggestions to add to ManageWiki in the meantime? @Hypercane [18:24:37] Hmm. Not that I can think of. Other than wgRevokePermissions. [18:24:47] Which I assume is going to be worked on next. [18:25:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULLF [18:25:12] [02miraheze/services] 07MirahezeSSLBot 036d6f477 - BOT: Updating services config for wikis [18:26:59] > Hmm. Not that I can think of. Other than wgRevokePermissions. @Hypercane well I don't see the need for that. Do you see a use-case? If so I'll work on a PR for it otherwise I'm not sure why it's needed. But yeah, was going to work on that next, but don't know if I should. [18:27:43] You're right. But its always nice to have it just incase you really need it. [18:27:51] So idk [18:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULtB [18:45:13] [02miraheze/services] 07MirahezeSSLBot 037a745be - BOT: Updating services config for wikis [18:59:15] [02mw-config] 07Universal-Omega synchronize pull request 03#3215: Add wgRestrictionLevels to ManageWiki per T5991 - 13https://git.io/JUICd [19:17:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.40, 20.72, 19.02 [19:19:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.05, 21.44, 19.41 [19:23:17] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.56, 20.26, 19.55 [19:32:17] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.45, 19.47, 19.29 [19:36:16] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.34, 18.66, 19.08 [20:14:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.22, 20.45, 19.25 [20:16:08] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Sun 13 Sep 2020 20:10:11 GMT +0000). [20:16:21] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.29, 18.67, 18.78 [20:18:36] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Sun 13 Sep 2020 20:13:23 GMT +0000). [20:19:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JULY0 [20:19:47] [02miraheze/ssl] 07MirahezeSSLBot 03fbd8697 - Bot: Update SSL cert for wiki.macc.nyc [20:31:10] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:34:48] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:34:59] PROBLEM - orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:36:11] PROBLEM - meta.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:36:11] PROBLEM - miraheze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.com' expires in 15 day(s) (Sun 13 Sep 2020 20:28:55 GMT +0000). [20:37:04] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Thu 26 Nov 2020 19:19:36 GMT +0000. [20:52:02] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Sun 13 Sep 2020 20:48:06 GMT +0000). [21:07:40] [02ManageWiki] 07Universal-Omega opened pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [21:53:01] [02ManageWiki] 07RhinosF1 reviewed pull request 03#179 commit - 13https://git.io/JUL3P [21:54:35] [02ManageWiki] 07Universal-Omega reviewed pull request 03#179 commit - 13https://git.io/JUL3X [22:08:54] [02ManageWiki] 07Universal-Omega synchronize pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [22:11:32] [02ManageWiki] 07RhinosF1 reviewed pull request 03#179 commit - 13https://git.io/JULsU [22:11:34] [02ManageWiki] 07RhinosF1 reviewed pull request 03#179 commit - 13https://git.io/JULsT [22:12:02] [02ManageWiki] 07RhinosF1 commented on pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULsk [22:12:12] [02ManageWiki] 07RhinosF1 edited a comment on pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULsk [23:40:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JULZB [23:40:43] [02miraheze/puppet] 07paladox 031ceadfe - varnish: Fix rate limiter * Whitelist all of Miraheze ips * Do not rate limit static.miraheze.org as it can handle bursts. [23:40:45] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [23:40:46] [02puppet] 07paladox opened pull request 03#1486: varnish: Fix rate limiter - 13https://git.io/JULZR [23:40:56] [02puppet] 07paladox edited pull request 03#1486: varnish: Fix rate limiter - 13https://git.io/JULZR [23:42:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JULZu [23:42:33] [02miraheze/puppet] 07paladox 03432e075 - Update nginx.pp [23:42:35] [02puppet] 07paladox synchronize pull request 03#1486: varnish: Fix rate limiter - 13https://git.io/JULZR