[00:34:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.02, 6.81, 4.86 [00:34:56] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.02, 6.46, 4.60 [00:35:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.64, 6.10, 4.22 [00:36:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 7.03, 4.93 [00:38:46] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 6.20, 4.88 [00:39:13] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [00:39:13] PROBLEM - wiki.candela.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.candela.digital could not be found [00:39:14] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for housing.wiki could not be found [00:39:14] PROBLEM - www.christipedia.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.christipedia.nl could not be found [00:39:20] PROBLEM - portalsofphereon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for portalsofphereon.com could not be found [00:39:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.25, 5.69, 4.53 [00:40:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.89, 7.67, 6.00 [00:42:49] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.02, 6.74, 5.95 [00:42:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.28, 6.68, 5.84 [00:46:07] RECOVERY - vitriol.spoon.army - reverse DNS on sslhost is OK: rDNS OK - vitriol.spoon.army reverse DNS resolves to cp10.miraheze.org [00:46:07] RECOVERY - wiki.candela.digital - reverse DNS on sslhost is OK: rDNS OK - wiki.candela.digital reverse DNS resolves to cp10.miraheze.org [00:46:09] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: rDNS OK - housing.wiki reverse DNS resolves to cp11.miraheze.org [00:46:09] RECOVERY - www.christipedia.nl - reverse DNS on sslhost is OK: rDNS OK - www.christipedia.nl reverse DNS resolves to cp10.miraheze.org [00:46:19] RECOVERY - portalsofphereon.com - reverse DNS on sslhost is OK: rDNS OK - portalsofphereon.com reverse DNS resolves to cp10.miraheze.org [00:57:48] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [03:58:58] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [04:00:59] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.60, 6.26, 4.63 [04:04:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 7.67, 5.67 [04:06:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.71, 6.71, 5.57 [04:15:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JONMw [04:15:12] [02miraheze/services] 07MirahezeSSLBot 033f996fa - BOT: Updating services config for wikis [04:53:16] !log continue wikibackups for top 15 wikis [04:53:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:06] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 6.70, 4.13, 1.72 [05:06:06] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.07, 2.83, 1.53 [05:37:16] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JONbJ [05:37:18] [02miraheze/dns] 07Reception123 03cc72ab6 - add inourownwords.online zone [05:38:09] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JONbI [05:38:11] [02miraheze/dns] 07Reception123 03228946f - add tot.wiki zone [05:41:53] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JONb9 [05:41:54] [02miraheze/ssl] 07Reception123 0327ab61c - add tot.wiki cert [09:12:31] !log disabled puppet on mw* [09:12:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:13:18] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±2] 13https://git.io/JOADZ [09:13:19] [02miraheze/mediawiki] 07Reception123 03ea676b9 - Switch AutoCreatePage to Universal Omega fork (T7156) [09:13:27] [02mw-config] 07Reception123 closed pull request 03#3852: Convert AutoCreatePage to ExtensionRegistry - 13https://git.io/JOSQk [09:13:28] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JOADc [09:13:30] [02miraheze/mw-config] 07Universal-Omega 03db02892 - Convert AutoCreatePage to ExtensionRegistry (#3852) [09:13:31] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [09:14:38] miraheze/mw-config - Reception123 the build passed. [09:15:09] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: fork, last run 11 minutes ago with 0 failures [09:15:15] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: fork, last run 11 minutes ago with 0 failures [09:15:40] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: fork, last run 11 minutes ago with 0 failures [09:16:36] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: fork, last run 13 minutes ago with 0 failures [09:17:14] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:17:34] !log cd /srv/mediawiki/w/extensions && sudo -u www-data git pull ; sudo -u www-data git submodule sync ; sudo -u www-data git submodule update && sudo puppet agent --enable && sudo puppet agent -t on mw*/jbr* (individually) [09:17:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:17:40] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [09:18:14] !log sudo puppet agent -tv && sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr* [09:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:18:36] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:19:10] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:25:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.64, 18.55, 13.05 [09:27:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.85, 15.94, 12.77 [10:07:42] RECOVERY - mw10 APT on mw10 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [10:08:07] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [10:08:56] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [10:09:06] !log install security patches for php-pear [10:09:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:09:29] RECOVERY - mw9 APT on mw9 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [10:09:29] RECOVERY - mw8 APT on mw8 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [10:09:33] RECOVERY - mw11 APT on mw11 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [10:09:40] RECOVERY - mon2 APT on mon2 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [10:09:42] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [10:10:45] RECOVERY - mail2 APT on mail2 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [10:10:49] !log resize bacula2 [10:10:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:27:32] !log not sure when bacula2 resize will be finished, downtimed in icinga for the next four hours [10:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:38:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:1bbd::4/cpweb, 2607:5300:205:200::1c30/cpweb [10:40:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:45:40] !log install security patch for php-pear on test3 [10:45:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:45:58] afk :) [10:47:46] RECOVERY - test3 APT on test3 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [10:51:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 6.15, 4.36 [10:53:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.40, 5.62, 4.43 [11:43:13] PROBLEM - mw9 Check Gluster Clients on mw9 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [11:44:05] PROBLEM - mw8 Check Gluster Clients on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [11:46:13] !log sudo umount /mnt/mediawiki-static on mw8/9 [11:46:16] RECOVERY - mw8 Check Gluster Clients on mw8 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [11:46:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:47:45] RECOVERY - mw9 Check Gluster Clients on mw9 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [11:53:19] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 1.43, 4.77, 2.86 [11:55:31] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.53, 3.19, 2.52 [13:16:29] dmehus: it would be a bad thing if I didn't know how to write a good implementation plan [13:40:41] SPF|Cloud, true, heh [13:59:03] PROBLEM - biblestrength.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [14:05:56] PROBLEM - puppet3 Disk Space on puppet3 is WARNING: DISK WARNING - free space: / 2605 MB (10% inode=92%); [14:15:48] [02mediawiki] 07R4356th opened pull request 03#1526: Update DarkMode - 13https://git.io/JOpjv [14:24:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.28, 6.77, 5.46 [14:24:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.87, 6.63, 5.54 [14:26:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.16, 7.20, 5.89 [14:26:39] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: Diff, 31032 files, 38.84GB, 2021-03-21 01:17:00 (5.4 weeks ago) [14:26:39] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CRITICAL: Full, 238729 files, 84.61GB, 2021-03-10 22:42:00 (6.8 weeks ago) [14:26:39] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Full, 5531 files, 22.29MB, 2021-03-14 00:05:00 (6.4 weeks ago) [14:26:39] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: no terminated jobs [14:26:39] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CRITICAL: Full, 87218 files, 3.984GB, 2021-03-10 23:10:00 (6.8 weeks ago) [14:26:59] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CRITICAL: Full, 87218 files, 3.984GB, 2021-03-10 23:10:00 (6.8 weeks ago) [14:26:59] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: no terminated jobs [14:26:59] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Full, 5531 files, 22.29MB, 2021-03-14 00:05:00 (6.4 weeks ago) [14:26:59] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: Diff, 31032 files, 38.84GB, 2021-03-21 01:17:00 (5.4 weeks ago) [14:26:59] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CRITICAL: Full, 238729 files, 84.61GB, 2021-03-10 22:42:00 (6.8 weeks ago) [14:27:39] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:28:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.74, 7.26, 5.86 [14:29:00] PROBLEM - biblestrength.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - biblestrength.org reverse DNS resolves to 30.99.98.34.bc.googleusercontent.com [14:30:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.22, 7.20, 6.20 [14:30:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.53, 7.37, 6.30 [14:30:36] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.33, 6.16, 5.62 [14:32:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.85, 6.77, 6.16 [14:32:30] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.97, 6.49, 6.10 [14:52:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 2.90, 2.48 [14:54:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.87, 2.43, 2.35 [15:02:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.67, 6.85, 6.07 [15:03:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.59, 8.00, 6.06 [15:04:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.01, 6.60, 5.78 [15:04:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.66, 6.49, 6.04 [15:05:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.75, 7.20, 6.00 [15:06:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.29, 5.69, 5.54 [15:07:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.27, 7.63, 6.28 [15:08:47] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.97, 7.39, 6.49 [15:09:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.36, 6.45, 5.95 [15:09:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.93, 6.85, 6.18 [15:10:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.89, 6.42, 6.23 [15:11:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.29, 6.78, 6.23 [16:21:43] [02mediawiki] 07Universal-Omega commented on pull request 03#1526: Update DarkMode - 13https://git.io/JOjk3 [16:26:46] [02mediawiki] 07R4356th commented on pull request 03#1526: Update DarkMode - 13https://git.io/JOjL8 [16:28:30] [02mediawiki] 07Universal-Omega commented on pull request 03#1526: Update DarkMode - 13https://git.io/JOjLd [16:29:10] [02mediawiki] 07Universal-Omega edited a comment on pull request 03#1526: Update DarkMode - 13https://git.io/JOjLd [16:29:38] [02mediawiki] 07R4356th closed pull request 03#1526: Update DarkMode - 13https://git.io/JOpjv [16:30:10] [02mediawiki] 07R4356th commented on pull request 03#1526: Update DarkMode - 13https://git.io/JOjtz [16:30:30] [02mediawiki] 07R4356th edited a comment on pull request 03#1526: Update DarkMode - 13https://git.io/JOjtz [16:31:09] [02mediawiki] 07Universal-Omega commented on pull request 03#1526: Update DarkMode - 13https://git.io/JOjtb [16:47:06] [02puppet] 07paladox closed pull request 03#1741: gluster: Remove gluster.[pem|key|ca] file - 13https://git.io/JOsAD [16:47:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JOjGB [16:47:09] [02miraheze/puppet] 07paladox 03fb2ded5 - gluster: Remove gluster.[pem|key|ca] file (#1741) [16:47:10] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [16:47:12] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [17:21:50] [02mw-config] 07Universal-Omega commented on pull request 03#3856: set wgCentralAuthGlobalPasswordPolicies for sensitive global groups (… - 13https://git.io/JOju4 [17:22:08] [02mw-config] 07Universal-Omega deleted a comment on pull request 03#3856: set wgCentralAuthGlobalPasswordPolicies for sensitive global groups (… - 13https://git.io/JOju4 [17:28:25] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [17:28:29] PROBLEM - permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for permanentfuturelab.wiki could not be found [17:35:22] RECOVERY - permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - permanentfuturelab.wiki reverse DNS resolves to cp10.miraheze.org [17:35:25] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: rDNS OK - wiki.nowchess.org reverse DNS resolves to cp10.miraheze.org [18:12:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.49, 7.08, 5.64 [18:14:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.98, 6.34, 5.56 [18:21:08] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-4 [+0/-0/±1] 13https://git.io/JOj9G [18:21:09] [02miraheze/mw-config] 07Reception123 0389bfbf8 - fix [18:21:11] [02mw-config] 07Reception123 synchronize pull request 03#3856: set wgCentralAuthGlobalPasswordPolicies for sensitive global groups (… - 13https://git.io/JOFDs [18:22:15] miraheze/mw-config - Reception123 the build has errored. [18:22:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-4 [+0/-0/±1] 13https://git.io/JOj9K [18:22:24] [02miraheze/mw-config] 07Reception123 03a368c1c - syntax [18:22:25] [02mw-config] 07Reception123 synchronize pull request 03#3856: set wgCentralAuthGlobalPasswordPolicies for sensitive global groups (… - 13https://git.io/JOFDs [18:23:25] miraheze/mw-config - Reception123 the build passed. [18:26:52] [02mw-config] 07Reception123 closed pull request 03#3856: set wgCentralAuthGlobalPasswordPolicies for sensitive global groups (… - 13https://git.io/JOFDs [18:26:54] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOjHP [18:26:55] [02miraheze/mw-config] 07Reception123 03dc939fc - set wgCentralAuthGlobalPasswordPolicies for sensitive global groups (… (#3856) [18:26:57] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [18:27:37] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-4 - 13https://git.io/vbvb3 [18:27:38] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-4 [18:28:15] miraheze/mw-config - Reception123 the build passed. [18:29:24] Reception123: ^ there is no "default" global group for that policy to apply [18:31:07] Majavah: ah, ok. Will remove that then, thanks. Basically the mix-up was that this setting was very different from the core $wgPasswordPolicies and both 'policies' and 'checks' had to be removed and when testing I think I only tried removing those individually. And then the other issue was that while testing separately the 'bad' settings were overriding my other tests [18:33:33] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOj75 [18:33:33] [02miraheze/mw-config] 07Reception123 03ea62b67 - rm 'default' group from wgCentralAuthGlobalPasswordPolicies [18:33:33] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [18:34:54] miraheze/mw-config - Reception123 the build passed. [18:48:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.27, 5.92, 5.10 [18:50:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.16, 5.77, 5.16 [19:07:09] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [19:11:10] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:11:35] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [20:05:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.94, 6.62, 5.17 [20:07:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.45, 5.76, 5.04 [20:21:55] Reception123, has there been any update on what memory size to allocate to CreateWiki with the updated model? [20:22:44] not really sure what we could try, the current one is at 1500MB which is quite high [20:23:02] Reception123, ah, is there any way we can check to see how much memory it needs? [20:23:35] if not, maybe we could try bumping it by 250MB and seeing if it errors? [20:23:41] i.e., the trial and error method? [20:24:08] oh if you mean 2500 that's way too much [20:24:16] no [20:24:19] I mean 250MB [20:24:22] not 2500MB [20:24:32] I mean like going from 1.5 GB to 1.75 GB [20:24:38] ah, you mean +250MB [20:24:41] yeah [20:25:02] and if it still errors, we know it needs more, and have to rethink our approach I guess? [20:25:04] yeah, I mean I think we could give it a go and see if that fixes it [20:25:11] JohnLewis: RhinosF1 ^ thoughts? [20:25:22] dmehus: yeah not sure what we'd do then, as we can't allocate way more memory to it [20:25:35] but there is https://phabricator.miraheze.org/T7139 [20:25:35] [ ⚓ T7139 MediaWiki Capacity Proposal ] - phabricator.miraheze.org [20:25:42] Reception123, ack, yeah [20:27:04] Well rather than changing and hoping, the best method would be changing it and running the job to see [20:27:13] Which is what I did to get the 1.5 number [20:34:05] oh yeah, that's all I meant, not changing it permanently, but just running the job with an increased allocation and see if it works [20:39:00] Though if it requires more memory than we have, our only option is to not use AI at all [20:40:58] oh :( [20:41:37] will we free up any RAM, though, from the jobrunner reorganization task? [20:41:47] maybe we could allocate some additional memory from that [20:44:59] Yes [20:45:11] But that takes resources [20:45:54] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 7.41, 9.00, 4.41 [20:51:50] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.19, 2.98, 3.12 [21:40:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3e2c [21:40:13] [02miraheze/services] 07MirahezeSSLBot 0387dbc81 - BOT: Updating services config for wikis [22:15:37] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [23:17:14] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 77.09 ms [23:25:06] PROBLEM - techwiki.techboyg5blog.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for techwiki.techboyg5blog.com could not be found