[00:05:50] @blackwidowmovie0 That would need an RfC to establish. It's something we could consider in the future, certainly, for discussion, but I don't see it as urgency. We have a new Steward who is extremely active. Additionally, we also have @JohnLewis and @Voidwalker, both of whom are also quite active. @NDKilla is also extremely active on Discord and in other ways and can be pinged to check the rename queue as needed [00:50:25] I'll ask [00:51:30] @blackwidowmovie0 Ask what? [02:13:56] RECOVERY - madzebrascience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madzebrascience.wiki' will expire on Fri 29 Jan 2021 13:43:32 GMT +0000. [02:17:23] Command sent from Discord by Zppix: [02:17:24] .whoami [02:17:39] Command sent from Discord by Zppix: [02:17:40] .accesslevel [02:17:40] The access level for MH-Discord is User. [02:22:52] PROBLEM - madzebrascience.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address madzebrascience.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:27:05] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.climatechange.ai' expires in 15 day(s) (Tue 05 Jan 2021 03:23:25 GMT +0000). [03:31:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLzS2 [03:31:55] [02miraheze/ssl] 07MirahezeSSLBot 032396f02 - Bot: Update SSL cert for wiki.climatechange.ai [04:01:45] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Sat 20 Mar 2021 02:31:48 GMT +0000. [04:33:56] RECOVERY - madzebrascience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madzebrascience.wiki' will expire on Fri 29 Jan 2021 13:43:32 GMT +0000. [04:37:09] PROBLEM - wiki.helioss.co - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.helioss.co' expires in 15 day(s) (Tue 05 Jan 2021 04:33:57 GMT +0000). [04:42:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLzpW [04:42:15] [02miraheze/ssl] 07MirahezeSSLBot 0361474f8 - Bot: Update SSL cert for wiki.helioss.co [04:42:52] PROBLEM - madzebrascience.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address madzebrascience.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:50:28] PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.macc.nyc All nameservers failed to answer the query. [05:03:45] RECOVERY - wiki.helioss.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.helioss.co' will expire on Sat 20 Mar 2021 03:42:07 GMT +0000. [05:21:37] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.macc.nyc and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:22:38] RECOVERY - cp9 APT on cp9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:24:51] RECOVERY - cp3 APT on cp3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:34:49] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [06:40:09] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 98 packages available for upgrade (0 critical updates). [07:01:16] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 99 packages available for upgrade (0 critical updates). [08:18:30] PROBLEM - wiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.miraheze.org could not be found [08:25:27] RECOVERY - wiki.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - wiki.miraheze.org reverse DNS resolves to cp7.miraheze.org [09:34:46] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.80, 6.50, 5.29 [09:36:02] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.92, 6.83, 5.73 [09:36:44] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.92, 6.54, 5.45 [09:38:00] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.76, 6.49, 5.74 [09:38:42] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.57, 5.74, 5.28 [10:05:30] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.10, 6.01, 5.19 [10:07:28] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.13, 5.55, 5.11 [11:05:23] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.63, 6.28, 5.17 [11:07:22] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.75, 5.61, 5.06 [11:57:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.16, 19.78, 18.61 [11:59:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.93, 19.00, 18.42 [12:05:00] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.10, 6.75, 5.96 [12:06:59] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.98, 5.94, 5.75 [12:11:41] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.04, 7.03, 6.09 [12:13:39] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.14, 6.88, 6.13 [12:15:37] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.96, 6.21, 5.97 [12:20:40] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.50, 7.11, 6.44 [12:22:38] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.34, 6.46, 6.28 [13:18:56] RECOVERY - madzebrascience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madzebrascience.wiki' will expire on Fri 29 Jan 2021 13:43:32 GMT +0000. [13:27:51] PROBLEM - madzebrascience.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address madzebrascience.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:35:17] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.85, 6.58, 5.64 [13:35:42] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.15, 6.68, 5.79 [13:37:16] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.22, 6.21, 5.62 [13:37:40] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.50, 6.12, 5.69 [14:00:32] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.00, 6.86, 6.11 [14:02:28] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.58, 6.37, 6.02 [14:10:16] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.87, 6.72, 6.19 [14:12:13] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.41, 6.55, 6.19 [14:20:04] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.91, 6.48, 6.20 [14:22:02] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.45, 5.95, 6.04 [14:32:40] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.31, 20.42, 19.76 [14:34:38] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.83, 20.28, 19.81 [14:35:22] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.68, 6.84, 5.97 [14:35:36] PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found [14:37:22] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.75, 6.41, 5.92 [14:41:40] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 9.87, 7.36, 6.50 [14:42:33] RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - www.erikapedia.com reverse DNS resolves to cp6.miraheze.org [14:43:39] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.78, 7.27, 6.56 [14:43:47] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.66, 6.89, 6.18 [14:47:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.06, 21.40, 20.21 [14:47:38] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.61, 6.47, 6.40 [14:47:44] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.22, 6.22, 6.09 [14:49:23] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.73, 6.95, 6.33 [14:49:27] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.53, 20.31, 19.94 [14:51:22] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.53, 6.46, 6.22 [14:58:19] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.33, 22.37, 20.65 [15:00:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.26, 22.19, 20.80 [15:12:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.31, 18.87, 20.10 [15:16:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.96, 22.36, 21.32 [15:31:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JL2X6 [15:31:36] [02miraheze/mw-config] 07Universal-Omega 030689485 - Update comment [15:33:04] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.87, 7.86, 6.79 [15:34:20] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.24, 7.47, 6.67 [15:35:53] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.80, 17.42, 19.69 [15:36:18] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.20, 6.97, 6.58 [15:37:00] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 5.65, 7.27, 6.86 [15:38:16] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.28, 6.39, 6.42 [15:40:56] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.52, 6.49, 6.64 [15:48:47] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.16, 6.65, 6.51 [15:50:16] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.54, 7.50, 6.56 [15:52:16] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.89, 7.33, 6.60 [15:54:41] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.32, 6.56, 6.55 [15:56:16] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.71, 6.16, 6.31 [16:04:29] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 9.26, 7.40, 6.72 [16:06:16] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.97, 7.05, 6.54 [16:06:27] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.82, 7.11, 6.69 [16:08:25] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.18, 6.76, 6.61 [16:11:05] !log shutdown -r on db7 [16:11:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:16] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 6.49, 6.59, 6.50 [16:12:52] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JL27T [16:12:53] [02miraheze/mw-config] 07Southparkfan 03540ebf9 - Disable compression flag for database With few to none bandwidth constraints, compression isn't likely to be useful here, but instead wasting clock cycles [16:12:57] !log root@gluster1:/home/paladox# gluster volume set mvol performance.nl-cache on [16:13:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:31] SPF|Cloud thanks, i forgot that was there! [16:14:02] let's see what the impact is on resource usage [16:15:36] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.24, 21.08, 17.52 [16:16:18] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.71, 6.94, 6.41 [16:17:42] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.31, 18.18, 16.91 [16:18:19] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.90, 6.58, 6.34 [16:18:42] PROBLEM - ping6 on db7 is CRITICAL: CRITICAL - Destination Unreachable (2001:41d0:800:105a::11) [16:18:45] PROBLEM - db7 APT on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:18:49] PROBLEM - db7 Current Load on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:19:12] PROBLEM - db7 Puppet on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:19:21] PROBLEM - ping4 on db7 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:19:23] PROBLEM - db7 SSH on db7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:34] PROBLEM - db7 Disk Space on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:19:40] PROBLEM - db7 NTP time on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:07] known ^ [16:20:11] PROBLEM - Host db7 is DOWN: PING CRITICAL - Packet loss = 100% [16:20:12] will set a downtime [16:20:34] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.57, 6.94, 6.63 [16:24:30] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.17, 6.62, 6.58 [16:29:28] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.29, 6.76, 6.16 [16:31:28] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.45, 6.08, 5.98 [16:35:25] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.47, 7.36, 6.64 [16:36:34] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 4.95, 6.83, 6.67 [16:37:24] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.81, 6.53, 6.44 [16:37:58] !log depooled/repooled mw[45] (experimenting with 20 rather than 26 php childs to see if it'll help the load) [16:38:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:33] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.01, 5.90, 6.34 [16:38:38] !log install atop on test2 [16:38:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:48] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [16:39:27] PROBLEM - mw5 Puppet on mw5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [16:42:21] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.38, 7.24, 6.54 [16:42:46] !log install fatrace as well on test2 [16:42:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:36] !log set php childs to 18 on mw[4567] (experimenting with different settings) [16:46:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:22] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 4.86, 6.14, 6.35 [16:50:09] PROBLEM - mw6 Puppet on mw6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [16:50:58] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 9.18, 7.65, 6.85 [16:51:56] PROBLEM - mw7 Puppet on mw7 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [16:52:54] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.31, 7.33, 6.81 [16:54:51] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.00, 6.57, 6.61 [16:59:49] @dmehus per conversation yesterday, when you said `@blackwidowmovie0 Ask what?`, I meant begin drafting an RfC [17:01:08] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 9.07, 7.11, 6.31 [17:03:06] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.43, 7.06, 6.38 [17:05:47] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [17:07:01] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.38, 6.05, 6.18 [17:07:45] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [17:07:47] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JL2je [17:07:48] [02miraheze/puppet] 07Southparkfan 032e5a0c8 - mediawiki: install fatrace [17:08:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JL2jI [17:08:19] [02miraheze/mw-config] 07paladox 03f08a9b4 - Switch wgLanguageConverterCacheType to redis [17:08:43] !log remove atop from test2 [17:09:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:57] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.07, 6.63, 6.37 [17:12:55] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.10, 6.46, 6.34 [17:23:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.11, 6.35, 6.11 [17:25:52] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.36, 5.70, 5.91 [17:31:35] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw7 [17:32:27] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 2 backends are down. mw5 mw7 [17:32:41] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 2 backends are down. mw5 mw7 [17:32:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2607:5300:205:200::2ac4/cpweb [17:33:10] PROBLEM - www.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fraterniterebellion.com could not be found [17:33:11] PROBLEM - nenawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nenawiki.org could not be found [17:33:13] PROBLEM - wiki.velaan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.velaan.org could not be found [17:33:23] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 2 backends are down. mw4 mw7 [17:33:39] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:34:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:35:37] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 1.742 second response time [17:36:51] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLaJD [17:36:52] [02miraheze/puppet] 07JohnFLewis 03a7849bc - T6615: Drop support for TLSv1 and TLSv1.1 [17:38:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb, 2607:5300:205:200::2ac4/cpweb [17:39:52] RECOVERY - www.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - www.fraterniterebellion.com reverse DNS resolves to cp6.miraheze.org [17:39:56] RECOVERY - nenawiki.org - reverse DNS on sslhost is OK: rDNS OK - nenawiki.org reverse DNS resolves to cp7.miraheze.org [17:39:58] RECOVERY - wiki.velaan.org - reverse DNS on sslhost is OK: rDNS OK - wiki.velaan.org reverse DNS resolves to cp6.miraheze.org [17:40:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:41:36] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.06, 6.56, 6.08 [17:42:29] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 11.00, 8.45, 6.85 [17:42:35] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [17:43:20] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [17:43:34] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.73, 6.48, 6.13 [17:43:37] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [17:44:22] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [17:44:27] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.99, 7.76, 6.79 [17:48:24] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.66, 6.66, 6.52 [17:59:14] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [18:01:14] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15969 bytes in 0.052 second response time [18:09:53] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JLaLo [18:09:54] [02miraheze/landing] 07paladox 035428fee - Update site notice [18:09:56] [02landing] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fxEQA [18:09:57] [02landing] 07paladox opened pull request 03#56: Update site notice - 13https://git.io/JLaL6 [18:09:59] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.62, 7.02, 6.39 [18:10:16] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JLaLi [18:10:17] [02miraheze/landing] 07paladox 03f3f735f - Update donate.php [18:10:19] [02landing] 07paladox synchronize pull request 03#56: Update site notice - 13https://git.io/JLaL6 [18:10:38] [02landing] 07paladox closed pull request 03#56: Update site notice - 13https://git.io/JLaL6 [18:10:40] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JLaL1 [18:10:41] [02miraheze/landing] 07paladox 03cbebfbb - Update site notice (#56) * Update site notice * Update donate.php [18:10:43] [02landing] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fxEQA [18:10:44] [02miraheze/landing] 07paladox deleted branch 03paladox-patch-2 [18:11:11] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.43, 7.85, 6.68 [18:12:57] @dmehus I would again submit a request for you to respond to my message on my talk page, given that it's been 10 days now [18:13:57] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.97, 7.24, 6.65 [18:14:53] miraheze/landing/paladox-patch-2/f3f735f - paladox The build has errored. https://travis-ci.org/miraheze/landing/builds/750710837 [18:15:08] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 5.45, 6.91, 6.60 [18:17:53] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 14.16, 9.16, 7.45 [18:19:04] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.54, 6.53, 6.54 [18:19:59] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 5.37, 7.41, 7.01 [18:20:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::2ac4/cpweb [18:21:06] PROBLEM - wiki.erisly.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.erisly.com could not be found [18:21:57] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.88, 6.54, 6.74 [18:21:57] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberfurs.org could not be found [18:22:02] PROBLEM - kk.uncyc.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kk.uncyc.tk could not be found [18:22:02] PROBLEM - www.opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.opendatascot.org could not be found [18:22:03] PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found [18:22:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:27:45] RECOVERY - wiki.erisly.com - reverse DNS on sslhost is OK: rDNS OK - wiki.erisly.com reverse DNS resolves to cp7.miraheze.org [18:28:52] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: rDNS OK - wiki.cyberfurs.org reverse DNS resolves to cp7.miraheze.org [18:28:55] RECOVERY - kk.uncyc.tk - reverse DNS on sslhost is OK: rDNS OK - kk.uncyc.tk reverse DNS resolves to cp6.miraheze.org [18:28:56] RECOVERY - orain.org - reverse DNS on sslhost is OK: rDNS OK - orain.org reverse DNS resolves to cp7.miraheze.org [18:28:56] RECOVERY - www.opendatascot.org - reverse DNS on sslhost is OK: rDNS OK - www.opendatascot.org reverse DNS resolves to cp6.miraheze.org [18:30:38] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Mon 08 Feb 2021 18:24:51 GMT +0000. [18:46:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:105a::10/cpweb [18:46:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb [18:48:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:48:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:02:40] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.05, 20.89, 17.77 [19:04:40] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.41, 20.13, 17.93 [19:05:55] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:06:09] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:06:45] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:07:21] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:41:45] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.07, 6.55, 6.04 [19:41:55] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 6.94, 6.98, 6.10 [19:43:43] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.63, 6.20, 5.97 [19:43:55] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.79, 6.53, 6.04 [19:45:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JLaln [19:45:44] [02miraheze/mw-config] 07paladox 03616b2de - Redis: Remove persistent Not really needed as we use nutcracker thus doesn't make a connection to redis on each request. [19:45:46] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:45:47] [02mw-config] 07paladox opened pull request 03#3578: Redis: Remove persistent - 13https://git.io/JLalc [19:46:00] [02mw-config] 07paladox closed pull request 03#3578: Redis: Remove persistent - 13https://git.io/JLalc [19:46:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLall [19:46:03] [02miraheze/mw-config] 07paladox 03151fdea - Redis: Remove persistent (#3578) Not really needed as we use nutcracker thus doesn't make a connection to redis on each request. [19:46:04] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:46:06] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:48:23] miraheze/mw-config/paladox-patch-1/616b2de - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/750722587 [19:50:51] RECOVERY - madzebrascience.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'madzebrascience.wiki' will expire on Fri 29 Jan 2021 13:43:32 GMT +0000. [19:52:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLa8O [19:52:18] [02miraheze/mw-config] 07paladox 03dc5eab5 - Switch $wgMessageCacheType and $wgLanguageConverterCacheType back to using the db [19:55:57] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.23, 7.20, 6.44 [19:57:53] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.34, 6.49, 6.27 [20:00:19] Something is wrong with miraheze formatting [20:00:32] https://meta.miraheze.org/wiki/Meta:Administrators%27_noticeboard#Oppose_4 bottom number is 4 when it should be 2 [20:00:32] [ Meta:Administrators' noticeboard - Miraheze Meta ] - meta.miraheze.org [20:01:22] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.60, 6.92, 6.47 [20:03:07] Naleksuh, I'm not sure what's wrong [20:03:20] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.27, 7.63, 6.64 [20:04:19] @Naleksuh I don't think it's a Miraheze specific issue. I'll try and fix it [20:04:20] dmehus, possibly archive your 70+ conversation talk page..... [20:04:38] also, my talk page's been sitting there for 10 days........ [20:04:52] @blackwidowmovie0 It's on my to-do list. I have to review and make sure I've replied to everyone first [20:05:23] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.96, 7.94, 6.95 [20:05:30] @blackwidowmovie0 I largely answered you; there is little else to be said hence why I have triaged it as low priority. I will get to it in the next week or so. [20:05:32] dmehus : It works fine on test [20:06:30] @Naleksuh I don't see the issue. All four opposes are numbered correctly. [20:06:47] Naleksuh: I don't see an issue [20:07:23] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.45, 7.61, 6.94 [20:08:02] #2 has been struck, but [[w:WP:LISTGAP]] convention is to preserve the numerical formatting for struck votes. Closer can manually assess that as only three opposes. Otherwise, we strike the numerical digit, and that breaks the gaps in the page, which are sub-optimal and undesirable [20:08:26] https://usercontent.irccloud-cdn.com/file/wYsZkF9A/image.png [20:08:29] Hm [20:08:31] I will try fix it more [20:09:16] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.34, 7.81, 7.13 [20:09:34] Oh I found it [20:09:41] You can't indent the first comment there is [20:11:17] Oh, #3 was indented, yeah. [20:11:27] but weird I didn't see the issue on the wiki page [20:11:36] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.39, 6.70, 6.26 [20:12:34] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.58, 7.44, 6.73 [20:13:04] @Naleksuh why did you change them all to indentations? Numerical was fine [20:13:36] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.92, 6.08, 6.06 [20:13:54] Currently looking at WP:LISTGAP [20:13:54] Cause in my experience generally indenting is used [20:14:04] Now Waldo's oppose is not correctly numbered as number #1. [20:14:30] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.47, 6.73, 6.56 [20:14:38] That was my intended goal [20:14:50] Although it does seem he has added to it since I last checked [20:15:09] So it may be better to have it as 1 [20:15:18] Oh, no not on English Wikipedia RfAs and Wikimedia Meta permissions requests, generally, afaik. We've always used either # or * lists [20:15:20] yeah [20:15:22] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.40, 6.54, 6.66 [20:15:28] you want to self revert or me? [20:15:48] Okay now it should be good [20:16:59] Yeah, I think generally struck votes are fine. Closing bureaucrat can ignore the numerical value adjacent to it. It's not perfect, but I think it's reasonable [20:17:04] Naleksuh, could you please point me to the specific revision or section? It could just be an indentation issue. [20:17:38] dmehus : Generally stuff is indented but not stricken in my experience...some people use both [20:17:50] r4356th: Its fixed now [20:17:59] Ah. [20:18:02] Cool. [20:18:49] @Naleksuh Actually, I see what we could do, and that's to insert a colon following my struck vote. That should drop the numerical value and indent it slightly? [20:19:03] I already did that [20:19:07] oh ok [20:22:18] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.29, 7.53, 6.88 [20:24:15] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.50, 7.06, 6.78 [20:26:12] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.46, 6.79, 6.71 [20:26:59] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.66, 6.09, 6.64 [20:28:27] RECOVERY - wiki.macc.nyc - reverse DNS on sslhost is OK: rDNS OK - wiki.macc.nyc reverse DNS resolves to cp6.miraheze.org [20:30:09] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.33, 7.05, 6.82 [20:32:06] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.46, 6.69, 6.71 [20:34:13] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.05, 7.63, 6.80 [20:34:52] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.88, 7.94, 7.22 [20:36:12] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.40, 7.52, 6.87 [20:36:49] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.78, 7.74, 7.25 [20:40:10] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.98, 6.69, 6.67 [20:40:58] lots of mw4, mw5, and mw7 issues ^ [20:44:36] @Reception123 Requesting enable of Phab [20:44:38] account [20:44:41] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.74, 6.08, 6.69 [20:45:40] `2020-11-19: [20:45:40] 19:13 Reception123: disabled BlackWidowMovieEditor on phab` [20:45:54] `2020-11-19: 9:13 Reception123: disabled BlackWidowMovieEditor on phab` [21:03:46] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 9.50, 7.43, 6.48 [21:05:43] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 5.68, 6.89, 6.41 [21:07:41] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.16, 6.64, 6.38 [21:35:22] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.36, 6.89, 6.51 [21:37:22] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.92, 6.32, 6.36