[00:03:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.61, 3.34, 2.77 [00:09:25] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.57, 3.24, 2.96 [00:56:06] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 2.77, 7.83, 4.38 [00:58:06] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 0.57, 5.33, 3.88 [01:00:06] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 0.20, 3.59, 3.42 [01:01:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.43, 3.28, 2.93 [01:03:25] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.07, 3.22, 2.95 [02:13:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.23, 3.48, 3.07 [02:15:25] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.27, 3.37, 3.08 [02:49:33] PROBLEM - nonciclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonciclopedia.org could not be found [02:49:34] PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found [02:49:34] PROBLEM - www.programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.programming.red could not be found [02:49:34] PROBLEM - www.christipedia.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.christipedia.nl could not be found [02:49:35] PROBLEM - icclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for icclopedia.org could not be found [02:56:27] RECOVERY - nonciclopedia.org - reverse DNS on sslhost is OK: rDNS OK - nonciclopedia.org reverse DNS resolves to cp7.miraheze.org [02:56:28] RECOVERY - www.programming.red - reverse DNS on sslhost is OK: rDNS OK - www.programming.red reverse DNS resolves to cp7.miraheze.org [02:56:28] RECOVERY - miraheze.tk - reverse DNS on sslhost is OK: rDNS OK - miraheze.tk reverse DNS resolves to cp6.miraheze.org [02:56:29] RECOVERY - icclopedia.org - reverse DNS on sslhost is OK: rDNS OK - icclopedia.org reverse DNS resolves to cp7.miraheze.org [02:56:30] RECOVERY - www.christipedia.nl - reverse DNS on sslhost is OK: rDNS OK - www.christipedia.nl reverse DNS resolves to cp6.miraheze.org [03:07:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.90, 4.10, 2.56 [06:25:12] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtfNG [06:25:14] [02miraheze/mediawiki] 07Reception123 03b9172fe - Update Cosmos [06:25:46] Thanks @Reception123! [06:40:18] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:40:27] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:41:13] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:41:47] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:41:59] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:04:27] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:04:28] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:12] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:48] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:58] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:41:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.94, 3.38, 3.98 [09:59:26] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.59, 3.70, 3.66 [10:05:26] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.45, 3.81, 3.76 [10:07:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.12, 3.89, 3.79 [10:09:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.58, 3.76, 3.76 [10:13:26] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.21, 3.86, 3.78 [10:19:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.80, 3.98, 3.88 [10:21:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.63, 4.23, 3.99 [10:35:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.94, 3.54, 3.93 [10:41:25] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 1.96, 2.41, 3.29 [11:33:05] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JtJYj [11:33:07] [02miraheze/dns] 07Reception123 033f363c8 - add sportsanalytics.wiki zone [11:45:41] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JtJOQ [11:45:42] [02miraheze/ssl] 07Reception123 03c711943 - add sportsanalytics.wiki cert [15:12:02] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JtJE9 [15:12:03] [02miraheze/CreateWiki] 07translatewiki 03e0fee0e - Localisation updates from https://translatewiki.net. [15:12:04] [ Main page - translatewiki.net ] - translatewiki.net [15:12:05] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+8/-0/±3] 13https://git.io/JtJEH [15:12:06] [02miraheze/landing] 07translatewiki 037e52d9d - Localisation updates from https://translatewiki.net. [15:12:07] [ Main page - translatewiki.net ] - translatewiki.net [15:12:08] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtJEQ [15:12:09] [02miraheze/ManageWiki] 07translatewiki 035f0d9d1 - Localisation updates from https://translatewiki.net. [15:12:10] [ Main page - translatewiki.net ] - translatewiki.net [15:12:11] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtJE7 [15:12:12] [02miraheze/MirahezeMagic] 07translatewiki 038370104 - Localisation updates from https://translatewiki.net. [15:12:13] [ Main page - translatewiki.net ] - translatewiki.net [15:13:12] miraheze/CreateWiki - translatewiki the build passed. [15:13:18] miraheze/landing - translatewiki the build passed. [15:13:22] miraheze/MirahezeMagic - translatewiki the build passed. [15:13:25] miraheze/ManageWiki - translatewiki the build passed. [15:52:34] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 9.55, 5.89, 3.67 [15:54:29] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 5.51, 5.07, 3.60 [15:56:24] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 3.40, 4.44, 3.54 [16:04:15] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 32.09, 21.76, 17.35 [16:06:19] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.11, 20.72, 17.50 [16:11:31] PROBLEM - kk.uncyc.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kk.uncyc.tk could not be found [16:12:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.52, 17.83, 17.26 [16:18:11] RECOVERY - kk.uncyc.tk - reverse DNS on sslhost is OK: rDNS OK - kk.uncyc.tk reverse DNS resolves to cp6.miraheze.org [16:45:35] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 7.38, 6.66, 4.47 [16:47:29] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 5.34, 5.81, 4.40 [16:48:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.29, 20.62, 18.24 [16:49:25] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 4.03, 4.90, 4.21 [16:50:15] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.58, 19.32, 18.07 [17:38:13] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.40, 21.02, 18.40 [17:38:49] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 5.14, 5.32, 3.90 [17:40:49] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 2.59, 4.51, 3.79 [17:42:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.26, 19.88, 18.60 [17:50:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.72, 20.70, 18.91 [17:52:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.04, 18.66, 18.38 [17:58:37] [02CreateWiki] 07JohnFLewis synchronize pull request 03#179: Machine Learning AI from live requests - 13https://git.io/JLQpk [17:59:31] [02CreateWiki] 07JohnFLewis closed pull request 03#179: Machine Learning AI from live requests - 13https://git.io/JLQpk [17:59:33] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/JtJ6V [17:59:34] [02miraheze/CreateWiki] 07OwenBaines 0352637fe - Machine Learning AI from live requests (#179) * add approval score to requests * script to produce persistent model file * extension.json update * add composer.json * explode strings * Utilise Pipeline over straight forward Classifier * use Pipeline element * Change SVM model parameters * Trigger tryAutoCreate when approving/declining a request * config typo and [17:59:34] variable typo Co-authored-by: John F. Lewis [17:59:45] miraheze/CreateWiki - JohnFLewis the build passed. [18:00:07] [02CreateWiki] 07JohnFLewis closed pull request 03#173: Add file_exists check for creating database/deleted.json - 13https://git.io/JkofM [18:00:16] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [18:00:28] [02CreateWiki] 07JohnFLewis closed pull request 03#185: Add check in generateDatabaseList for checking if $allWikis returns non object - 13https://git.io/JLpS8 [18:00:36] miraheze/CreateWiki - JohnFLewis the build passed. [18:12:56] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtJi8 [18:12:58] [02miraheze/mediawiki] 07JohnFLewis 0343f6df1 - CW [18:16:59] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtJiD [18:17:00] [02miraheze/puppet] 07JohnFLewis 03f610748 - CW composer [18:37:28] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.48, 18.39, 17.19 [18:39:23] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 19.15, 18.18, 17.23 [18:41:39] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:41:40] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:41:42] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:41:54] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:42:05] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:42:41] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:54:29] Hello Kieran! If you have any questions, feel free to ask and someone should answer soon. [18:54:33] Hi [18:54:37] Is this irc active? [18:55:30] I was wondering if there’s a way to set edits to require approval on my page? So I can pervent vandalism [18:57:01] Well, Discord links to IRC, so yeah... [18:57:43] As for your question, I can't really answer it, so hopefully someone else can provide you the answer (although, you could use the abuse filter to block all edits from non-approved users) [19:01:14] .tell Kieran you could try [[mw:Extension:ApprovedRevs]], potentially, though the edits will still exist; they just won't be the current revision until approved. Alternativel, you could enable [[mw:Extension:Moderation]], with edits not live until moderated. You could even add the automoderated user rights to `autoconfirmed`, potentially, so that once a user has mode x number of approved edits, they're no longer moderated [19:01:14] dmehus: I'll pass that on when Kieran is around. [19:01:52] .help [19:01:53] dmehus: I've published a list of my commands at: https://sopel.bots.miraheze.wiki/help_prod.html [19:02:10] .help link [19:02:10] dmehus: Log a link in the meeing log. See [meetbot module usage]({% link _usage/meetbot-module.md %}) [19:02:11] e.g. .link http://example.com [19:03:40] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:04:08] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:04:40] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:05:41] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:05:42] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:05:51] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:07:49] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtJ1r [19:07:50] [02miraheze/CreateWiki] 07JohnFLewis 0389e6861 - Resolve exceptions when running on servers, fix model assignment [19:08:17] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtJ1i [19:08:19] [02miraheze/mediawiki] 07JohnFLewis 03ed87c2e - CW [19:09:48] miraheze/CreateWiki - JohnFLewis the build passed. [19:11:24] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtJ17 [19:11:25] [02miraheze/mw-config] 07JohnFLewis 033d86a66 - add persistent model file location [19:12:29] miraheze/mw-config - JohnFLewis the build passed. [19:21:55] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtJMl [19:21:56] [02miraheze/mediawiki] 07paladox 032a0a1d9 - Update Mermaid [19:24:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtJMw [19:24:24] [02miraheze/puppet] 07paladox 033d7c913 - mediawiki::composer: Remove Mermaid [19:39:38] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:41:05] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [19:42:15] PROBLEM - meta.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.nocyclo.tk could not be found [19:42:48] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:47:35] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled] [19:48:58] RECOVERY - meta.nocyclo.tk - reverse DNS on sslhost is OK: rDNS OK - meta.nocyclo.tk reverse DNS resolves to cp6.miraheze.org [20:04:40] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:11:54] Hey [20:12:29] Just checking something [20:13:46] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:37:56] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 4.29, 5.24, 3.26 [20:39:50] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 1.20, 3.84, 2.98 [20:57:56] !log reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki thegamerwiki /home/reception/johnthegamer_pages_full.xml [20:57:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:55] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 10.97, 7.44, 5.07 [21:42:59] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.41, 7.28, 5.19 [21:43:53] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 6.13, 7.05, 5.22 [21:44:59] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.31, 6.19, 5.06 [22:01:45] [02mw-config] 07The-Voidwalker opened pull request 03#3661: add wgARENamespaces to ManageWikiNamespaces - 13https://git.io/JtJds [22:02:49] miraheze/mw-config - The-Voidwalker the build passed. [22:42:48] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.48, 3.09, 2.33 [22:45:10] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JtJFb [22:45:12] [02miraheze/CreateWiki] 07JohnFLewis 030fe6a3c - Move AI tasks to JobRunner as memory intensive in a web request [22:46:16] miraheze/CreateWiki - JohnFLewis the build passed. [22:46:35] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JtJFA [22:46:36] [02miraheze/CreateWiki] 07JohnFLewis 036e17673 - Move AI tasks to JobRunner as memory intensive in a web request [22:47:02] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtJFh [22:47:04] [02miraheze/mediawiki] 07JohnFLewis 037d1709e - CW [22:47:41] miraheze/CreateWiki - JohnFLewis the build passed. [22:56:36] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 1.69, 3.70, 3.57 [22:57:19] [02mw-config] 07dmehus commented on pull request 03#3661: add wgARENamespaces to ManageWikiNamespaces - 13https://git.io/JtJbz [22:58:34] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 0.90, 2.78, 3.25 [23:11:37] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:11:42] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:11:53] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:12:09] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:12:41] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:33:53] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:35:35] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:35:40] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:36:04] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:36:07] !log start c4 replication on dbbackup1 [23:36:09] *waits for puppet to recover on mw4 and jobrunner1* [23:36:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:40] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:36:46] There we go. :) [23:36:49] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 4.45, 7.17, 4.02 [23:37:57] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 7.75, 4.20, 2.67 [23:38:49] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 1.50, 5.20, 3.68 [23:39:28] urgh, this backup server can't handle our load [23:40:49] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 1.22, 3.87, 3.37 [23:40:55] @SPF|Cloud Can we up the resources on the backup server, and what would that cost? [23:41:06] !log dbbackup1's mydumper performance test: dumped all c4 databases (mydumper running with one thread) to .sql.gz (compressed logical backup). Started at 15:24:53, finished at 21:17:28. mariadb data directory is 202G, dump directory is 33G [23:41:08] What's it need...more cores, or more RAM, or neither? [23:41:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:42] I'd start with a faster disk and a service provider that doesn't mind performance impact [23:43:42] I must say compressing 202G of databases into just 33G of sql files is impressive, a reduction of ~84% [23:46:53] approximately ten months ago, when migrating our database servers to OVH, I thought "I know miraheze is large and busy, but not large and busy enough to outgrow slow disks, we'll be fine", and I was wrong [23:49:02] @SPF|Cloud Ah, okay, that makes sense, and lol re: your story :) [23:49:23] ten months later, while setting up the backup server, I am wrong agian [23:49:37] the irony [23:50:37] you know a server is slow if it starts building up lag [23:56:00] heh