[00:48:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.29, 6.27, 5.66 [00:52:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.33, 6.07, 5.74 [02:14:23] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [02:40:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYEQK [02:40:11] [02miraheze/services] 07MirahezeSSLBot 03bef3647 - BOT: Updating services config for wikis [06:30:43] PROBLEM - gp.ct777.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gp.ct777.cf' expires in 15 day(s) (Fri 16 Apr 2021 06:21:27 GMT +0000). [06:31:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYu63 [06:31:33] [02miraheze/ssl] 07MirahezeSSLBot 0371ea9e9 - Bot: Update SSL cert for gp.ct777.cf [07:06:07] RECOVERY - gp.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'gp.ct777.cf' will expire on Tue 29 Jun 2021 05:31:27 GMT +0000. [08:25:08] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [08:30:18] PROBLEM - www.erikapedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.erikapedia.com reverse DNS resolves to host.chariothost.com [08:33:09] PROBLEM - erikapedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - erikapedia.com reverse DNS resolves to host.chariothost.com [08:58:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JYzlt [08:58:38] [02miraheze/ssl] 07Reception123 03fa62342 - add wiki.fbpml.org cert [09:00:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JYzlQ [09:00:38] [02miraheze/ssl] 07Reception123 034b2a81c - add zh.gyaanipedia.com cert [09:33:26] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [09:34:58] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (3 critical updates). [09:34:59] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [09:35:11] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (3 critical updates). [09:35:18] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (3 critical updates). [09:35:36] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (3 critical updates). [09:36:36] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (3 critical updates). [09:36:55] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (3 critical updates). [09:37:23] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (3 critical updates). [09:37:51] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [09:38:45] PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [09:38:54] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [09:42:59] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [09:44:25] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [09:45:17] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [09:46:10] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (3 critical updates). [09:50:41] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (3 critical updates). [09:50:53] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 72 packages available for upgrade (3 critical updates). [09:51:03] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 108 packages available for upgrade (3 critical updates). [09:52:21] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (3 critical updates). [09:52:23] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [09:52:35] PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [09:52:37] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (3 critical updates). [09:53:15] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 72 packages available for upgrade (3 critical updates). [09:53:50] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 2.39, 3.53, 2.06 [09:54:40] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (3 critical updates). [09:55:05] PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (3 critical updates). [09:55:10] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [09:56:06] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.35, 2.26, 1.79 [09:56:17] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [09:56:19] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (3 critical updates). [09:57:09] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (3 critical updates). [09:58:37] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (3 critical updates). [10:02:07] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [11:27:26] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [12:51:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.63, 6.24, 5.07 [12:56:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.63, 6.12, 5.31 [13:06:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.79, 7.11, 6.08 [13:10:11] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.43, 6.78, 6.20 [13:18:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.77, 6.18 [13:22:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.06, 7.18, 6.46 [13:24:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.14, 6.04, 6.13 [13:25:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.35, 6.57, 6.33 [13:33:01] RhinosF1: https://meta.miraheze.org/w/index.php?diff=169486&oldid=119696&rcid=608963 good job [13:33:04] [ Difference between revisions of "User:RhinosF1/global.js" - Miraheze Meta ] - meta.miraheze.org [13:33:31] SPF|Cloud: need to fix my status.js for updating when I'm online now [13:33:33] looking now how I can make it show the database cluster as well [13:33:58] https://usercontent.irccloud-cdn.com/file/s8iCK1df/1617197630.JPG [13:34:06] It's blocking the top of the page [13:34:15] Might just remove that seen as MirahezeBot can control it [13:35:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.27, 7.06, 6.59 [13:37:20] !log running test backup on metawiki [13:38:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.99, 6.29, 6.36 [13:39:45] RhinosF1, never tried loading it on mobile, but on desktop, it loads perfectly for me. I did note you and SPF|Cloud copied the actual script into your global.js file. We do have https://dev.miraheze.org/wiki/MediaWiki:BackendInformation.js that can be loaded into your global.js file with `mw.loader.load`, if you wanted [13:39:47] [ MediaWiki:BackendInformation.js - Miraheze Developers Wiki ] - dev.miraheze.org [13:40:36] dmehus: when I first started with that script it refused to work using mw.loader.load iirc [13:41:44] hrm, weird, it loads fine for me and Universal_Omega. It used to be VarnishXID.js, but the file name changed [13:41:56] huh [13:42:05] I'll look when at my pc [13:42:08] I can fiddle better then [13:42:23] did I just make a consistent(!) backup of metawiki even though metawiki was in read-write mode? [13:42:55] If you did that sounds good [13:42:58] https://meta.miraheze.org/w/index.php?title=User:Southparkfan/Test&curid=29310&action=history edited metawiki every few seconds, yet no database locking at all [13:43:00] [ Revision history of "User:Southparkfan/Test" - Miraheze Meta ] - meta.miraheze.org [13:43:16] this is awesome [13:43:25] It is [14:04:29] ha, I saw an entry in the text table that was not attached to a revision / content entry [14:05:01] but it's used for https://meta.miraheze.org/wiki/Special:AbuseLog/276318, the abuse filter stores wikitext in the text table [14:05:03] [ Abuse filter log - Miraheze Meta ] - meta.miraheze.org [14:05:04] mystery solved [14:05:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.91, 6.99, 5.98 [14:06:09] and since that abuse log entry is present in the abuse_filter_log table + it is the last entry in the log table, it looks like this backup is consistent [14:06:52] awesome [14:06:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.65, 6.99, 6.40 [14:09:03] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.76, 6.45, 6.26 [14:09:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.39, 6.03 [14:11:41] !log performing extra backups for metawiki (testing for T5877) [14:16:16] PROBLEM - linkwiki.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'linkwiki.cf' expires in 15 day(s) (Fri 16 Apr 2021 14:11:42 GMT +0000). [14:17:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYgpT [14:17:01] [02miraheze/ssl] 07MirahezeSSLBot 03d08b08b - Bot: Update SSL cert for linkwiki.cf [14:21:17] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.slimelab.net' expires in 15 day(s) (Fri 16 Apr 2021 14:12:59 GMT +0000). [14:26:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 7.97, 6.69 [14:27:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.82, 7.79, 6.70 [14:27:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 7.06, 6.27 [14:27:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.69, 7.73, 6.91 [14:28:41] RhinosF1: https://phabricator.miraheze.org/T5877#139982 [14:28:42] [ ⚓ T5877 Revise MariaDB backup strategy ] - phabricator.miraheze.org [14:29:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 7.51, 6.68 [14:29:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JY2eN [14:29:14] [02miraheze/ssl] 07MirahezeSSLBot 034a3b9f7 - Bot: Update SSL cert for wiki.slimelab.net [14:29:19] looks good eh? [14:29:23] SPF|Cloud: ye [14:29:35] :) [14:29:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.05, 8.26, 7.03 [14:29:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.14, 6.52, 6.19 [14:30:42] 1350 GB space at RamNode costs about $22/mo, which is a fair price for database backups [14:30:49] Yeah [14:31:36] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [14:31:45] if one compressed backup of three clusters is 150G (haven't tested yet), you could store eight of them (technically, 1350 / 150 = 9, but then the disk would be full :) ) [14:32:15] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.38, 8.06, 7.20 [14:33:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.34, 7.43, 6.82 [14:33:35] using a replica, you could create incremental backups, which is even more efficient, but a replica with SSDs is a lot more expensive [14:33:53] Yeah SSDs have long been an issue [14:35:27] let me see what SSD instances @ ovh cost [14:35:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.18, 7.11, 6.79 [14:37:09] RECOVERY - linkwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'linkwiki.cf' will expire on Tue 29 Jun 2021 13:16:54 GMT +0000. [14:38:31] an instance with 7 GB RAM and 800 GB SSD is ~$106/mo [14:38:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.90, 7.51, 7.39 [14:38:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.10, 7.83, 7.50 [14:39:17] whereas two RamNode KVMs with 8 GB RAM and 2650 GB HDD cost $96/mo [14:39:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.42, 7.72, 7.11 [14:39:50] yes, that's per VM, so over 5 TB total [14:43:00] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.27, 5.91, 6.78 [14:44:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.77, 7.72, 7.24 [14:47:40] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.62, 6.95, 6.95 [14:49:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.36, 6.79, 6.87 [14:50:23] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.59, 8.26, 7.54 [14:52:00] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.08, 6.17, 6.64 [14:52:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.97, 8.10, 6.94 [14:53:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.30, 5.87, 6.63 [14:54:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.53, 7.81, 7.55 [14:54:40] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.23, 6.79, 6.61 [15:00:55] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 6.55, 6.49 [15:03:13] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.07, 6.68 [15:05:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.74, 7.26, 6.82 [15:07:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.55, 5.85, 6.61 [15:07:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.35, 6.96 [15:11:29] RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Tue 29 Jun 2021 13:29:07 GMT +0000. [15:11:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.99, 6.42, 6.65 [15:14:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.48, 6.28, 6.69 [15:18:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 6.83, 6.46 [15:24:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.47, 7.31, 6.99 [15:29:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.02, 6.27, 6.65 [15:29:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.52, 7.01, 6.73 [15:31:55] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.76, 6.23, 6.50 [15:36:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.30, 7.16, 6.88 [15:38:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.36, 6.35, 6.61 [15:49:20] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.59, 8.90, 7.40 [15:51:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.26, 7.29, 6.60 [15:52:43] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 7.09, 6.59 [15:53:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 7.76, 6.89 [15:53:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.58, 7.92, 7.37 [15:54:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.24, 6.69 [15:56:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.89, 6.68, 6.55 [15:58:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.74, 6.04, 6.77 [15:59:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.95, 6.27, 6.53 [16:05:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.34, 7.25, 6.77 [16:06:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.98, 7.07, 6.56 [16:07:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 7.40, 7.08 [16:09:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 7.38, 6.92 [16:10:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.59, 6.34, 6.44 [16:11:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.70, 7.38, 6.98 [16:14:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.96, 7.90, 7.22 [16:15:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.67, 5.78, 6.49 [16:16:12] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.51, 6.85, 6.91 [16:18:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.78, 6.52, 6.78 [16:29:49] RECOVERY - db11 APT on db11 is OK: APT OK: 69 packages available for upgrade (0 critical updates). [16:33:36] RECOVERY - cp12 APT on cp12 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:33:46] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 105 packages available for upgrade (0 critical updates). [16:33:46] !log installing security patches per https://linuxsecurity.com/advisories/debian/debian-dsa-4881-1-curl-security-update-06-30-31 [16:33:47] [ Debian: DSA-4881-1: curl security update. ] - linuxsecurity.com [16:33:53] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [16:33:58] RECOVERY - mon2 APT on mon2 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [16:34:09] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [16:34:11] RECOVERY - mem2 APT on mem2 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [16:34:13] RECOVERY - services4 APT on services4 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:34:15] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [16:34:22] RECOVERY - test4 APT on test4 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [16:34:42] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [16:34:46] RECOVERY - ns2 APT on ns2 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [16:34:47] RECOVERY - cp3 APT on cp3 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:34:49] RECOVERY - cp11 APT on cp11 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:34:51] RECOVERY - mem1 APT on mem1 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [16:34:51] RECOVERY - mail2 APT on mail2 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:34:58] RECOVERY - db13 APT on db13 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [16:34:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.92, 7.36, 6.47 [16:35:00] RECOVERY - services3 APT on services3 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:35:03] RECOVERY - mw10 APT on mw10 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [16:35:04] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:35:04] RECOVERY - test3 APT on test3 is OK: APT OK: 32 packages available for upgrade (0 critical updates). [16:35:10] RECOVERY - db12 APT on db12 is OK: APT OK: 69 packages available for upgrade (0 critical updates). [16:35:10] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [16:35:12] RECOVERY - ns1 APT on ns1 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [16:35:19] no log bot? [16:35:29] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [16:35:36] RECOVERY - cp10 APT on cp10 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [16:35:41] RECOVERY - mw11 APT on mw11 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [16:35:54] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [16:35:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.50, 7.66, 6.91 [16:35:56] RECOVERY - mw8 APT on mw8 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [16:36:01] RECOVERY - mw9 APT on mw9 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [16:36:10] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [16:36:12] RECOVERY - phab2 APT on phab2 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [16:36:21] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [16:36:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.55, 7.87, 7.06 [16:36:55] SPF|Cloud, you're right, MirahezeLogbot didn't log your log entry. That's strange [16:37:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.33, 6.51, 6.27 [16:37:08] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [16:37:08] restarted logbot [16:37:18] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [16:37:19] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [16:37:31] !log installing security patches per https://linuxsecurity.com/advisories/debian/debian-dsa-4881-1-curl-security-update-06-30-31 [16:37:31] [ Debian: DSA-4881-1: curl security update. ] - linuxsecurity.com [16:37:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:51] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [16:37:51] Should we also log "restarted logbot" too, SPF|Cloud? [16:38:01] !log restarted adminlogbot [16:38:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:06] be patient :-) [16:38:08] ty [16:38:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 6.96, 6.75 [16:38:25] doing multiple things at once [16:38:40] SPF|Cloud, wasn't sure if you forgot or not. [16:38:49] nope [16:38:49] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [16:38:54] looks like dbbackup1 has died [16:38:55] wut? [16:39:13] oh [16:39:26] that doesn't sound good :( [16:40:02] !log reboot dbbackup1 [16:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:25] probably caused by my networking adventures earlier [16:41:01] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.52, 6.59, 6.76 [16:41:14] fortunately, the hands-on work is still fun here [16:41:27] ah [16:42:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.02, 6.17, 6.51 [16:43:14] JohnLewis: I remember you suggested when we run jobs we set a memory limit, what do you think a good value for that limit would be? [16:43:33] looks like dbbackup1 came back online [16:44:14] RECOVERY - Host dbbackup1 is UP: PING OK - Packet loss = 0%, RTA = 97.28 ms [16:44:16] PROBLEM - dbbackup1 MariaDB c2 on dbbackup1 is CRITICAL: Can't connect to MySQL server on 'dbbackup1.miraheze.org' (111) [16:44:16] PROBLEM - dbbackup1 APT on dbbackup1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:44:16] PROBLEM - dbbackup1 MariaDB c4 on dbbackup1 is CRITICAL: Can't connect to MySQL server on 'dbbackup1.miraheze.org' (111) [16:44:26] PROBLEM - dbbackup1 NTP time on dbbackup1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:44:27] RECOVERY - dbbackup1 NTP time on dbbackup1 is OK: NTP OK: Offset 0.004104107618 secs [16:44:35] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 0.54, 0.33, 0.13 [16:44:49] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is UNKNOWN: NRPE: Unable to read output [16:44:55] I was wondering why all mediawiki servers have apt failues [16:45:01] PROBLEM - dbbackup1 Disk Space on dbbackup1 is CRITICAL: connect to address 107.161.20.211 port 5666: Connection refusedconnect to host 107.161.20.211 port 5666: Connection refused [16:45:01] RECOVERY - dbbackup1 Disk Space on dbbackup1 is OK: DISK OK - free space: / 102829 MB (16% inode=98%); [16:45:06] E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it? [16:45:15] probably caused by me installing security patches [16:48:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.27, 6.70, 6.35 [16:50:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.24, 7.13, 6.71 [16:51:00] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [16:51:16] RECOVERY - dbbackup1 MariaDB c2 on dbbackup1 is OK: Uptime: 272 Threads: 7 Questions: 12 Slow queries: 1 Opens: 16 Flush tables: 1 Open tables: 10 Queries per second avg: 0.044 [16:51:31] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 88.55 ms [16:52:15] !log start replica mariadb@c2 dbbackup1 [16:52:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.12, 6.75 [16:52:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.87, 7.88, 7.05 [16:55:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.12, 7.88, 6.96 [16:55:15] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:55:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.56, 7.52, 6.92 [16:55:39] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:55:54] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:56:10] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:57:08] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:57:11] Reception123: check the code of jobrunner, I’d say docs but you’ve not written them yet ;) [16:57:28] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:57:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.48, 7.00, 6.55 [16:57:58] thanks! they seem to now have been run [16:59:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.52, 6.89, 6.57 [16:59:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 7.74, 7.28 [17:01:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.24, 7.79, 7.25 [17:01:55] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.02, 9.02, 7.80 [17:03:14] talking about jobrunner, 55% of the graylog message sent over the past 48 hours come from the jobrunner process [17:03:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.48, 6.75, 6.60 [17:04:33] interesting, though not terribly surprising, I suspect. Could be interesting data analysis to analyze what jobrunner-related error messages are occurring the most [17:04:48] <30>1 2021-03-31T17:04:04+00:00 jobrunner3.miraheze.org jobrunner 567 - - [19cde399025ae6eb5f85fdd6] [no req] MWException from line 129 of /srv/mediawiki/w/includes/MagicWord.php: Error: invalid magic word 'getshortdesc' [17:05:44] ah, interesting, I wonder if that's because of wikis trying to transclude that magic word for the ShortDescription extension (which is still pending security review, iirc) [17:06:00] Reception123: did you rebuild l10n cache after installing ShortDescription? [17:07:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.41, 6.94 [17:09:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.93, 7.87, 7.82 [17:09:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.35, 7.62, 7.76 [17:10:03] Reception123: mem limit for CW is in puppet when done by the queue so just use that [17:11:08] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.97, 6.46, 6.78 [17:13:00] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.73, 7.78, 7.74 [17:13:50] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.02, 7.72, 7.15 [17:15:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 6.98, 7.45 [17:15:06] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.71, 7.95, 7.29 [17:15:15] !log rebuild LC on jobrunner3 for en, does not fix exceptions [17:15:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:15:47] SPF|Cloud: yes, and even after for unrelated updates [17:15:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.91, 7.70, 7.21 [17:16:05] last time was yesterday actually [17:16:12] https://phabricator.miraheze.org/T7070 [17:16:12] [ ⚓ T7070 MWException from line 129 of /srv/mediawiki/w/includes/MagicWord.php: Error: invalid magic word 'getshortdesc' ] - phabricator.miraheze.org [17:16:46] hmm, if it's not LC it could be upstream [17:17:07] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.19, 7.51, 7.23 [17:17:27] the extension file is recognised by mergeMessageFileList [17:19:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.33, 5.63, 6.67 [17:19:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.61, 7.45, 7.17 [17:21:41] Reception123: could you look, I'm dealing with a security issue [17:22:22] Reception123: look at the line from TreeAndMenu in the trace [17:23:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.10, 6.98, 7.13 [17:25:13] I can have a look quickly, but will have to go do something IRL soon [17:26:28] RhinosF1: https://gitlab.com/organicdesign/TreeAndMenu/-/blob/master/TreeAndMenu_body.php#L27 [17:26:28] [ TreeAndMenu_body.php · master · Organic Design / TreeAndMenu · GitLab ] - gitlab.com [17:26:52] That's not helpful [17:26:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.48, 6.05, 6.69 [17:29:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.82, 5.76, 6.59 [17:29:41] !log restart php-fpm on mediawiki servers (depool, restart & repool) [17:29:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:46] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.21, 6.00, 6.66 [17:33:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.12, 8.52, 7.52 [17:33:41] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8 [17:33:42] PROBLEM - wiki.etn.link - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.etn.link could not be found [17:33:42] PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minkyu.kim could not be found [17:33:43] PROBLEM - stablestate.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stablestate.org could not be found [17:33:45] PROBLEM - wiki.counterculturelabs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.counterculturelabs.org could not be found [17:33:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.58, 9.18, 7.88 [17:33:51] PROBLEM - wikibeats.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikibeats.net could not be found [17:35:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 2.47, 6.44, 6.90 [17:35:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 2.76, 6.77, 7.15 [17:35:58] php-fpm restarted [17:36:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.21, 6.74, 6.27 [17:37:08] PROBLEM - wiki.invinciblewellbeing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.invinciblewellbeing.com could not be found [17:37:09] PROBLEM - wiki.nevillepedia.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nevillepedia.eu could not be found [17:37:10] PROBLEM - miraheze.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ml could not be found [17:37:12] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [17:37:14] PROBLEM - www.bluevies.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.bluevies.org could not be found [17:37:18] PROBLEM - www.tradwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.tradwiki.org could not be found [17:37:30] PROBLEM - heavyironmodding.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for heavyironmodding.org could not be found [17:37:35] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mr.gyaanipedia.com could not be found [17:37:36] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [17:37:39] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lefrenchmelee.fr could not be found [17:37:45] PROBLEM - programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for programming.red could not be found [17:37:50] !log restart php-fpm on mon2, mail2, phab2, test3, jobrunner4 for security update [17:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:52] !log restart phd for security updates [17:38:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.88, 5.89, 6.01 [17:39:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.34, 7.32, 7.12 [17:40:32] RECOVERY - wiki.counterculturelabs.org - reverse DNS on sslhost is OK: rDNS OK - wiki.counterculturelabs.org reverse DNS resolves to cp11.miraheze.org [17:40:32] RECOVERY - wikibeats.net - reverse DNS on sslhost is OK: rDNS OK - wikibeats.net reverse DNS resolves to cp10.miraheze.org [17:40:35] RECOVERY - wiki.etn.link - reverse DNS on sslhost is OK: rDNS OK - wiki.etn.link reverse DNS resolves to cp10.miraheze.org [17:40:35] RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: rDNS OK - wiki.minkyu.kim reverse DNS resolves to cp10.miraheze.org [17:40:41] RECOVERY - stablestate.org - reverse DNS on sslhost is OK: rDNS OK - stablestate.org reverse DNS resolves to cp10.miraheze.org [17:40:47] !log restart jobchron, jobrunner (jobrunners) and mongod (graylog2) [17:40:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.66, 8.22, 7.66 [17:41:54] I'm off [17:42:04] see you tomorrow, I guess [17:42:26] ... almost off [17:43:15] !log rebuilding LC on jobrunner[34] in a screen for https://phabricator.miraheze.org/T7070 [17:43:16] [ ⚓ T7070 MWException from line 129 of /srv/mediawiki/w/includes/MagicWord.php: Error: invalid magic word 'getshortdesc' ] - phabricator.miraheze.org [17:43:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:50] RECOVERY - miraheze.ml - reverse DNS on sslhost is OK: rDNS OK - miraheze.ml reverse DNS resolves to cp11.miraheze.org [17:43:58] RECOVERY - wiki.invinciblewellbeing.com - reverse DNS on sslhost is OK: rDNS OK - wiki.invinciblewellbeing.com reverse DNS resolves to cp10.miraheze.org [17:44:09] RECOVERY - wiki.nevillepedia.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.nevillepedia.eu reverse DNS resolves to cp11.miraheze.org [17:44:10] RECOVERY - www.tradwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.tradwiki.org reverse DNS resolves to cp11.miraheze.org [17:44:11] RECOVERY - www.bluevies.org - reverse DNS on sslhost is OK: rDNS OK - www.bluevies.org reverse DNS resolves to cp10.miraheze.org [17:44:11] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: rDNS OK - www.zenbuddhism.info reverse DNS resolves to cp10.miraheze.org [17:44:16] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - meta.gyaanipedia.com reverse DNS resolves to cp11.miraheze.org [17:44:25] RECOVERY - mr.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - mr.gyaanipedia.com reverse DNS resolves to cp11.miraheze.org [17:44:29] RECOVERY - heavyironmodding.org - reverse DNS on sslhost is OK: rDNS OK - heavyironmodding.org reverse DNS resolves to cp10.miraheze.org [17:44:36] RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: rDNS OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp10.miraheze.org [17:44:41] RECOVERY - programming.red - reverse DNS on sslhost is OK: rDNS OK - programming.red reverse DNS resolves to cp10.miraheze.org [17:54:41] fixed [17:54:42] bye :) [19:17:49] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.75, 3.69, 3.08 [19:21:46] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.80, 3.05, 2.95 [19:27:43] paladox: any idea regarding https://phabricator.miraheze.org/T7068 ? [19:27:44] [ ⚓ T7068 DataDump "queued" forever ] - phabricator.miraheze.org [19:27:54] UO asked me and i do not [19:27:59] you will need to debug to find out [19:39:32] PROBLEM - wiki.scrungecraft.gg - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.scrungecraft.gg' expires in 15 day(s) (Fri 16 Apr 2021 19:35:02 GMT +0000). [19:43:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYaIh [19:43:20] [02miraheze/ssl] 07MirahezeSSLBot 03d94b671 - Bot: Update SSL cert for wiki.scrungecraft.gg [20:04:52] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.16, 3.41, 2.12 [20:06:22] RECOVERY - wiki.scrungecraft.gg - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scrungecraft.gg' will expire on Tue 29 Jun 2021 18:43:12 GMT +0000. [20:06:52] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.41, 2.96, 2.13 [20:15:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.58, 7.23, 7.91 [20:17:59] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.93, 8.10 [20:21:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.91, 7.62, 7.95 [20:21:59] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 7.49, 7.89 [20:29:58] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.88, 7.95, 7.86 [20:31:06] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.57, 7.95, 7.78 [20:42:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.07, 7.46, 8.00 [20:43:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.21, 7.42, 7.88 [20:49:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 7.23, 7.55 [20:54:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 8.00, 7.73 [20:58:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.69, 7.96, 7.83 [21:00:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.65, 8.03, 7.88 [21:07:03]