[00:04:54] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.65, 6.74, 4.10
[00:08:53] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.43, 6.26, 4.62
[00:44:20] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.08, 6.69, 6.49
[00:46:17] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.49, 6.23, 6.34
[00:53:43] <icinga-miraheze>	 PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.51, 3.29, 2.79
[00:55:43] <icinga-miraheze>	 RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.00, 3.19, 2.81
[01:11:24] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%);
[01:44:28] <icinga-miraheze>	 PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.33, 3.41, 3.12
[01:46:22] <icinga-miraheze>	 RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.07, 3.33, 3.13
[01:52:07] <icinga-miraheze>	 PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.62, 3.41, 3.19
[01:54:02] <icinga-miraheze>	 RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 2.55, 3.09, 3.09
[04:30:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjdQC
[04:30:13] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 036d25f2c - BOT: Updating services config for wikis
[06:25:24] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2906 MB (12% inode=94%);
[09:20:11] <Not-f817>	 [02MirahezeMagic] 07RhinosF1 opened pull request 03#90: Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[09:20:30] <Not-f817>	 [02MirahezeMagic] 07RhinosF1 edited pull request 03#90: [WIP - DO NOT MERGE] Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[09:20:59] <Not-f817>	 [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#90: [WIP - DO NOT MERGE] Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[09:41:25] <Not-f817>	 [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#90: [WIP - DO NOT MERGE] Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[10:07:32] <Not-f817>	 [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#90: [WIP - DO NOT MERGE] Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[10:07:47] <RhinosF1>	 Reception123: merge pls ^ that resolves it
[10:11:00] <Reception123>	 RhinosF1: you should change the title ;)
[10:11:19] <Not-f817>	 [02MirahezeMagic] 07RhinosF1 edited pull request 03#90: Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[10:11:22] <RhinosF1>	 Reception123: ^
[10:12:08] <Reception123>	 RhinosF1: ok, and could you also get rid of the extra end line?
[10:12:32] <Not-f817>	 [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#90: Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[10:12:49] <RhinosF1>	 Reception123: done ^ that's been there since you created the file
[10:13:16] <Reception123>	 my bad :P
[10:13:37] <Not-f817>	 [02MirahezeMagic] 07Reception123 closed pull request 03#90: Batch 4 i18n T4370 - 13https://git.io/fjd5Q
[10:13:38] <Not-f817>	 [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjdF6
[10:13:40] <Not-f817>	 [02miraheze/MirahezeMagic] 07RhinosF1 039808067 - Batch 4 i18n T4370 (#90) * add more * add visual editor config * Update en.json * this reverts last commit * restored + fixed syntax + 2 more * Update en.json * add more ~40% * Update en.json * add more >50% * per MWSettings * per https://github.com/miraheze/mw-config/pull/2721 * +1 * +up to links * localisation settings * per MWSettings * maps + 1 lang setting
[10:13:40] <Not-f817>	 * good catch github formatting, i should check copy-pastes * a lot more now ~71% * format fixes * Update en.json * add notification related settings to i18n T4370 * +restricted settings * Add styling settings this resolves T4370 * remove whitespace
[10:13:41] <ZppixBot>	 [ consistency by RhinosF1 · Pull Request #2721 · miraheze/mw-config · GitHub ] - github.com
[10:42:18] <Not-f817>	 [02mediawiki] 07RhinosF1 opened pull request 03#121: Updating MirahezeMagic - 13https://git.io/fjdbN
[10:42:27] <RhinosF1>	 Reception123: worked thx
[10:42:42] <Reception123>	 great :) I'll run puppet and then LC
[10:42:55] <RhinosF1>	 Reception123: yey, I'm going to note them down
[10:58:01] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki loginwiki on mw* 
[10:58:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[10:58:18] <RhinosF1>	 Reception123: you have to merge the PR first
[10:58:30] <RhinosF1>	 don't you?
[10:58:48] <Reception123>	 :P the merged one above confused me and I thought I did
[10:59:24] <Not-f817>	 [02mediawiki] 07Reception123 closed pull request 03#121: Updating MirahezeMagic - 13https://git.io/fjdbN
[10:59:26] <Not-f817>	 [02miraheze/mediawiki] 07Reception123 pushed 032 commits to 03REL1_33 [+0/-0/±2] 13https://git.io/fjdNJ
[10:59:27] <Not-f817>	 [02miraheze/mediawiki] 07RhinosF1 039719bf7 - Updating MirahezeMagic (T4370)
[10:59:29] <Not-f817>	 [02miraheze/mediawiki] 07Reception123 03737c489 - Merge pull request #121 from RhinosF1/REL1_33 Updating MirahezeMagic
[11:27:24] <Reception123>	 !log moved import for osaindexwiki to test1 due to redis issues
[11:27:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[12:10:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjdN5
[12:10:13] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 036dfc17a - BOT: Updating services config for wikis
[15:05:00] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is CRITICAL: DISK CRITICAL - free space: / 9159 MB (5% inode=98%);
[15:14:00] <Not-f817>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[15:14:02] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjFvl
[15:14:03] <Not-f817>	 [02miraheze/puppet] 07paladox 03c593bf8 - lizardfs: Add chunkserver_mount_removal variable This variable will allow us to remove chunkservers.
[15:14:05] <Not-f817>	 [02puppet] 07paladox opened pull request 03#1068: lizardfs: Add chunkserver_mount_removal variable - 13https://git.io/fjFv8
[15:15:39] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.02, 6.88, 5.86
[15:15:49] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjFv0
[15:15:51] <Not-f817>	 [02miraheze/puppet] 07paladox 03fdcaa16 - Update mfshdd.cfg.erb
[15:15:52] <Not-f817>	 [02puppet] 07paladox synchronize pull request 03#1068: lizardfs: Add chunkserver_mount_removal variable - 13https://git.io/fjFv8
[15:16:34] <Not-f817>	 [02puppet] 07paladox closed pull request 03#1068: lizardfs: Add chunkserver_mount_removal variable - 13https://git.io/fjFv8
[15:16:35] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjFv2
[15:16:37] <Not-f817>	 [02miraheze/puppet] 07paladox 037659708 - lizardfs: Add chunkserver_mount_removal variable (#1068) * lizardfs: Add chunkserver_mount_removal variable This variable will allow us to remove chunkservers. * Update mfshdd.cfg.erb
[15:16:38] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[15:16:40] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[15:17:39] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.86, 6.54, 5.86
[15:20:44] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFvr
[15:20:45] <Not-f817>	 [02miraheze/puppet] 07paladox 03e4e93dd - lizardfs: Do not reload lizardfs-chunkserver automatically if config changes Should be done manually.
[15:23:39] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.91, 7.01, 6.29
[15:25:01] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is WARNING: DISK WARNING - free space: / 9413 MB (6% inode=98%);
[15:25:39] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.16, 6.71, 6.26
[15:32:02] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFv9
[15:32:04] <Not-f817>	 [02miraheze/puppet] 07paladox 031049587 - lizardfs: Do not automatically reload the lizardfs-master service A manual restart should be done.
[15:59:42] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFf4
[15:59:44] <Not-f817>	 [02miraheze/puppet] 07paladox 039219432 - lizardfs: Remove cp5 from web
[16:01:08] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFfE
[16:01:10] <Not-f817>	 [02miraheze/puppet] 07paladox 034b712ca - Update groups.conf
[16:01:24] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFfu
[16:01:26] <Not-f817>	 [02miraheze/puppet] 07paladox 033e84b8d - Update groups.conf
[16:01:53] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFfz
[16:01:54] <Not-f817>	 [02miraheze/puppet] 07paladox 0346ffee2 - mediawiki: Remove cp5
[16:03:01] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is CRITICAL: DISK CRITICAL - free space: / 9211 MB (5% inode=98%);
[16:03:39] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.22, 6.84, 6.28
[16:03:43] <paladox>	 !log delete cp5 from firewall on mw*
[16:03:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:05:04] <icinga-miraheze>	 PROBLEM - misc1 Check correctness of the icinga configuration on misc1 is CRITICAL: Icinga configuration contains errors
[16:05:16] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[icinga2]
[16:05:42] <paladox>	 hmm
[16:06:15] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFfw
[16:06:16] <Not-f817>	 [02miraheze/puppet] 07paladox 03c6c1205 - icinga2: Fix syntax
[16:06:30] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS
[16:06:32] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4
[16:06:33] <Not-f817>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[16:06:35] <Not-f817>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[16:07:39] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.64, 6.72, 6.35
[16:11:41] <paladox>	 !log redis-cli --scan --pattern *cirrus* | xargs redis-cli  del 
[16:11:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:12:28] <paladox>	 !log redis-cli --scan --pattern *CookieWarning\Decisions:cookieWarningIpLookupCache* | xargs redis-cli  del 
[16:12:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:14:11] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown)
[16:14:31] <icinga-miraheze>	 PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown)
[16:14:40] <icinga-miraheze>	 PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown)
[16:14:41] <icinga-miraheze>	 PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown)
[16:14:41] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 381 failures. Last run 2 minutes ago with 381 failures. Failed resources (up to 3 shown)
[16:14:47] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 2 minutes ago with 17 failures. Failed resources (up to 3 shown)
[16:15:00] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 375 failures. Last run 2 minutes ago with 375 failures. Failed resources (up to 3 shown)
[16:15:16] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 3 minutes ago with 365 failures. Failed resources (up to 3 shown)
[16:15:18] <icinga-miraheze>	 PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown)
[16:15:24] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown)
[16:15:26] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 371 failures. Last run 3 minutes ago with 371 failures. Failed resources (up to 3 shown)
[16:15:28] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown)
[16:15:29] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 3 minutes ago with 23 failures. Failed resources (up to 3 shown)
[16:15:32] <icinga-miraheze>	 PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown)
[16:15:33] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 25 failures. Last run 3 minutes ago with 25 failures. Failed resources (up to 3 shown)
[16:15:51] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 372 failures. Last run 3 minutes ago with 372 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org],File[wildcard.miraheze.org_private],File[/etc/mathoid/config.yaml],File[/etc/apt/trusted.gpg.d/php.gpg]
[16:16:41] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 298 failures. Last run 3 minutes ago with 298 failures. Failed resources (up to 3 shown): File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common]
[16:17:23] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 176 failures. Last run 3 minutes ago with 176 failures. Failed resources (up to 3 shown): File[/etc/default/stunnel4],File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate]
[16:21:58] <paladox>	 hmm
[16:22:04] <paladox>	 ah, gc
[16:23:24] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[16:23:31] <icinga-miraheze>	 RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[16:24:02] <icinga-miraheze>	 RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[16:24:06] <icinga-miraheze>	 RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[16:24:07] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[16:24:22] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[16:24:24] <icinga-miraheze>	 RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[16:25:00] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is WARNING: DISK WARNING - free space: / 9524 MB (6% inode=98%);
[16:25:06] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[16:25:06] <icinga-miraheze>	 RECOVERY - misc1 Check correctness of the icinga configuration on misc1 is OK: Icinga configuration is correct
[16:25:15] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:25:16] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:25:18] <icinga-miraheze>	 RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:25:28] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:25:29] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:25:32] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:25:35] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:25:40] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[16:26:02] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:26:29] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:26:45] <paladox>	 !log restarting redis
[16:27:13] <paladox>	 !log redis-cli --scan --pattern *:cookieWarningIpLookupCache* | xargs -n 1 -d '\n' redis-cli  del
[16:27:22] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:28:01] <paladox>	 !log restarting redis
[16:28:05] <paladox>	 !log redis-cli --scan --pattern *:cookieWarningIpLookupCache* | xargs -n 1 -d '\n' redis-cli  del
[16:28:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:28:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:28:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:29:13] <paladox>	 !log restart logbot - misc1
[16:29:14] <paladox>	 !log restarting redis
[16:29:16] <paladox>	 !log redis-cli --scan --pattern *:cookieWarningIpLookupCache* | xargs -n 1 -d '\n' redis-cli  del
[16:29:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:29:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:29:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:39:19] <paladox>	 !log running runJobs.php for all wikis on test1
[16:39:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:45:44] <paladox>	 !log sudo -u www-data php removeOrphanedEvents.php --wiki=americangirldollswiki --force
[16:45:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:15:08] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFJ7
[17:15:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03179fa4f - BOT: Updating services config for wikis
[19:46:55] <Hispano76_>	 Miraheze is very slow
[19:51:12] <RhinosF1>	 Hispano76_: looks fine to me
[19:51:55] <Hispano76_>	 ok
[19:58:37] <RhinosF1>	 Hispano76_: what internet speeds u get ?
[19:58:42] <RhinosF1>	 Try fast.com
[20:00:41] <Hispano76_>	 I regularly have between 5 mbps and 9 mbps
[20:01:05] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is CRITICAL: DISK CRITICAL - free space: / 9119 MB (5% inode=98%);
[20:01:13] <RhinosF1>	 Hispano76_: that's why it's slow - it's fine for me at about 60-70mbps on that test
[20:04:23] <Hispano76_>	 Wait, what country are you from? (I joke), I remember that always the cableras give between 5 and 15 mb. But in other countries I
[20:06:48] <RhinosF1>	 Hispano76_: England
[20:23:05] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is WARNING: DISK WARNING - free space: / 9298 MB (6% inode=98%);
[20:28:10] <Hispano76_>	 ping Reception123 
[21:22:24] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.08, 5.26, 3.85
[21:24:22] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.42, 5.86, 4.25
[21:25:54] <Hispano76>	  Miraheze is very slow
[21:26:47] <RhinosF1>	 Hispano76: will double check in a moment
[21:31:14] <RhinosF1>	 Hispano76: go to fast.com and confirm speed you are getting pls
[21:32:12] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.12, 1.61, 1.24
[21:33:31] <Hispano76>	 RhinosF1, hispanowiki https://prnt.sc/ou0rn5 
[21:33:31] <ZppixBot>	 [ Screenshot by Lightshot ] - prnt.sc
[21:34:11] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.31, 1.51, 1.25
[21:34:26] <Hispano76>	 Well it's weird, because in Wikivoyage I access in a second while in hispanowiki takes more than 1 minute loading page
[21:34:59] <RhinosF1>	 Hispano76: yeah, i can see it varying a bit with cache blocked.
[21:35:21] <Hispano76>	 hum https://prnt.sc/ou0se4
[21:35:21] <ZppixBot>	 [ Screenshot by Lightshot ] - prnt.sc
[21:35:43] <RhinosF1>	 Hispano76: well that won't help
[21:36:37] <Hispano76>	 It's funny, on another speed page, there's a much higher figure.
[21:36:48] <RhinosF1>	 Hispano76: eh, strange
[21:38:07] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.35, 1.82, 1.42
[21:38:54] <RhinosF1>	 Hispano76: seems a bit temperamental to me
[21:41:05] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is CRITICAL: DISK CRITICAL - free space: / 9131 MB (5% inode=98%);
[21:41:19] <RhinosF1>	 ah great
[21:41:28] <RhinosF1>	 Reception123: ^^^ ping pong
[21:41:43] <RhinosF1>	 paladox: ^
[21:43:14] <paladox>	 Hmm?
[21:43:58] <RhinosF1>	 paladox: also, Reception123 was going to change that figure again if storage got too close so keep an eye pls as well as load / access speed being poor
[21:44:24] <paladox>	 I’m on holiday now :)
[21:44:33] <RhinosF1>	 paladox: oh, yeah you are
[21:44:47] * RhinosF1 facepalms - you only told me yesterday
[21:45:28] <RhinosF1>	 I think storage is fine for a bit but if it gets close to around 1 GB Reception wanted to push a fix so err if no ones around we might need someone to merge
[21:45:47] <RhinosF1>	 i'll keep as much of an eye as I can but I have no access of course
[21:45:47] <paladox>	 He can only reduce it to 5G.
[21:45:57] <paladox>	 He won’t be able to reduce it any furthur
[21:46:28] <RhinosF1>	 okay, well I'll keep an eye and if we need to do that then someone might need to if he's not around. I can do the commit but not merge of course
[21:47:20] <paladox>	 Ok, I won’t be able to do anything. So John or Reception123 will have to
[21:47:36] <RhinosF1>	 who both aren't appearing so we might be stuck
[21:48:01] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.53, 1.90, 1.76
[21:48:02] <paladox>	 Yeh
[21:48:27] <RhinosF1>	 load is crazy and the speeds are a bit temperamental as well
[21:48:57] <RhinosF1>	 Hispano76: we need a sysadmin whose around unfortunately
[21:49:12] * RhinosF1 pokes Reception123
[21:49:52] <paladox>	 Test1 not a concern, mw3 would be, but we have mw 1 and 2
[21:49:53] <paladox>	 He’s in bed likley
[21:49:59] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.61, 1.44, 1.61
[21:50:39] <RhinosF1>	 paladox: it's bloody slow with cache off if you try at points
[21:51:16] <paladox>	 Oh
[21:51:23] <paladox>	 Mind opening a task please?
[21:51:29] <RhinosF1>	 paladox: I can do yes
[21:53:19] <RhinosF1>	 paladox: https://phabricator.miraheze.org/T4654
[21:53:20] <ZppixBot>	 [ ⚓ T4654 Slow loading speeds on some wikis ] - phabricator.miraheze.org
[21:53:55] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.14, 2.33, 1.97
[21:55:28] <paladox>	 Thanks
[21:55:46] <RhinosF1>	 paladox: np
[21:55:54] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.12, 1.93, 1.87
[21:55:57] * RhinosF1 wishes paladox a good holiday
[21:56:16] * RhinosF1 also whispers to icinga-miraheze to stop spamming us about test1
[21:57:13] <paladox>	 Thanks
[21:57:19] <RhinosF1>	 np
[21:57:52] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.40, 2.40, 2.04
[22:34:35] <Hispano76>	 because my wikis take too long to open.
[22:39:10] <paladox>	 Hmm, it loads fast so it may be cp2, but not really sure
[22:39:11] <paladox>	 *for Me 
[22:39:35] <Hispano76>	 hum
[23:01:05] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 14.08, 8.66, 5.99
[23:07:02] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.66, 7.11, 6.43
[23:09:00] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.33, 6.25, 6.20
[23:09:04] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.84, 1.03, 1.94
[23:14:59] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.63, 0.89, 1.59
[23:17:21] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:17:30] <icinga-miraheze>	 PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:17:30] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:17:31] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:17:37] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:18:04] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:18:32] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for test1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:18:36] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:18:52] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:19:17] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.00, 0.03, 0.06
[23:19:26] <icinga-miraheze>	 RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1519 bytes in 0.698 second response time
[23:19:29] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 23705 bytes in 0.689 second response time
[23:19:31] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: Status line output matched "502" - 309 bytes in 0.520 second response time
[23:19:32] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[23:20:02] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures
[23:20:27] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for test1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 23688 bytes in 0.686 second response time
[23:20:31] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 1%
[23:20:52] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 23705 bytes in 0.652 second response time
[23:44:38] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.95, 1.62, 1.35
[23:46:36] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.17, 1.46, 1.33
[23:54:30] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.47, 1.90, 1.52
[23:56:28] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.58, 1.75, 1.50
[23:58:27] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.35, 1.52, 1.44