[00:31:21] <icinga-miraheze>	 PROBLEM - test4 Puppet on test4 is WARNING: WARNING: Puppet is currently disabled, message: palaodx, last run 12 minutes ago with 0 failures
[01:40:30] <dmehus>	 paladox, I don't know where the icinga warning messages are stored, but if you get a chance, can you fix that typo in your name ^ as it says "palaodx" instead of "paladox"
[03:01:50] <icinga-miraheze>	 RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org
[03:36:54] <icinga-miraheze>	 PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu]
[04:04:52] <icinga-miraheze>	 RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:03:10] <icinga-miraheze>	 RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 33 packages available for upgrade (0 critical updates).
[06:05:41] <icinga-miraheze>	 RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 31 packages available for upgrade (0 critical updates).
[06:09:42] <icinga-miraheze>	 RECOVERY - mw9 APT on mw9 is OK: APT OK: 31 packages available for upgrade (0 critical updates).
[06:11:59] <icinga-miraheze>	 RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 105 packages available for upgrade (0 critical updates).
[06:12:59] <icinga-miraheze>	 RECOVERY - mon2 APT on mon2 is OK: APT OK: 28 packages available for upgrade (0 critical updates).
[06:19:49] <icinga-miraheze>	 RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 59 packages available for upgrade (0 critical updates).
[06:26:16] <icinga-miraheze>	 RECOVERY - mw10 APT on mw10 is OK: APT OK: 31 packages available for upgrade (0 critical updates).
[06:28:33] <icinga-miraheze>	 RECOVERY - services3 APT on services3 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[06:43:33] <icinga-miraheze>	 RECOVERY - services4 APT on services4 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[06:44:15] <icinga-miraheze>	 RECOVERY - test3 APT on test3 is OK: APT OK: 32 packages available for upgrade (0 critical updates).
[06:44:51] <icinga-miraheze>	 RECOVERY - mw11 APT on mw11 is OK: APT OK: 31 packages available for upgrade (0 critical updates).
[06:48:33] <icinga-miraheze>	 RECOVERY - mw8 APT on mw8 is OK: APT OK: 31 packages available for upgrade (0 critical updates).
[06:49:35] <icinga-miraheze>	 RECOVERY - mail2 APT on mail2 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[06:51:39] <icinga-miraheze>	 RECOVERY - phab2 APT on phab2 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[06:55:11] <icinga-miraheze>	 RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 59 packages available for upgrade (0 critical updates).
[07:47:06] <Reception123>	 !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --full --logs --uploads --output gzip:/home/reception/socdemwikiwiki02042021.xml --wiki socdemwikiwiki
[07:47:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:55:36] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 created branch 03Reception123-patch-2 13https://git.io/JYKXA
[07:55:38] <Not-818c>	 [02MirahezeMagic] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/fQRGX
[07:56:24] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JYK1U
[07:56:25] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 039dc0951 - Update MirahezeMagicHooks.php
[07:56:26] <icinga-miraheze>	 PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (2 critical updates).
[07:57:09] <icinga-miraheze>	 PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (2 critical updates).
[07:57:11] <icinga-miraheze>	 PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (2 critical updates).
[07:57:52] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYK1c
[07:57:53] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 03a571f97 - Update en.json
[07:58:28] <Not-818c>	 [02MirahezeMagic] 07Reception123 opened pull request 03#242: message on DataDump regarding it not working T7068 - 13https://git.io/JYK1R
[07:58:36] <Not-818c>	 [02MirahezeMagic] 07Reception123 closed pull request 03#242: message on DataDump regarding it not working T7068 - 13https://git.io/JYK1R
[07:58:37] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYK1E
[07:58:39] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 03a64c996 - Update MirahezeMagicHooks.php (#242)
[07:58:40] <Not-818c>	 [02MirahezeMagic] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/fQRGX
[07:58:42] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 deleted branch 03Reception123-patch-2
[07:58:48] <miraheze-github>	 miraheze/MirahezeMagic - Reception123 the build passed.
[07:58:52] <Not-818c>	 [02MirahezeMagic] 07Reception123 commented on commit 03a571f97f1039c18611f713aac50d04033d29c531 - 13https://git.io/JYK12
[07:59:33] <miraheze-github>	 miraheze/MirahezeMagic - Reception123 the build passed.
[07:59:39] <miraheze-github_>	 miraheze/MirahezeMagic - Reception123 the build passed.
[07:59:48] <icinga-miraheze>	 PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:00:29] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYK1d
[08:00:31] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 034652dc1 - Update MirahezeMagic
[08:03:07] <icinga-miraheze>	 PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 23 packages available for upgrade (2 critical updates).
[08:03:38] <icinga-miraheze>	 PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:05:14] <icinga-miraheze>	 PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates).
[08:05:40] <icinga-miraheze>	 PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates).
[08:06:16] <icinga-miraheze>	 PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates).
[08:07:27] <icinga-miraheze>	 PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 23 packages available for upgrade (2 critical updates).
[08:07:57] <icinga-miraheze>	 PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:09:20] <icinga-miraheze>	 PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 23 packages available for upgrade (2 critical updates).
[08:10:53] <icinga-miraheze>	 PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[08:10:55] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[08:11:44] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[08:12:01] <icinga-miraheze>	 PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:12:33] <icinga-miraheze>	 PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:12:53] <icinga-miraheze>	 PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:13:45] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki
[08:13:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:15:58] <icinga-miraheze>	 PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (2 critical updates).
[08:19:32] <icinga-miraheze>	 PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (2 critical updates).
[08:19:59] <icinga-miraheze>	 PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 107 packages available for upgrade (2 critical updates).
[08:20:57] <icinga-miraheze>	 PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (2 critical updates).
[08:21:14] <icinga-miraheze>	 PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (2 critical updates).
[08:21:21] <icinga-miraheze>	 PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (2 critical updates).
[08:21:42] <icinga-miraheze>	 PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 23 packages available for upgrade (2 critical updates).
[08:22:47] <icinga-miraheze>	 PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 23 packages available for upgrade (2 critical updates).
[08:22:59] <icinga-miraheze>	 PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[08:23:13] <icinga-miraheze>	 PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:24:41] <icinga-miraheze>	 PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (2 critical updates).
[08:25:16] <icinga-miraheze>	 PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (2 critical updates).
[08:25:34] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.60, 7.07, 5.52
[08:25:37] <icinga-miraheze>	 PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (2 critical updates).
[08:26:54] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.88, 6.43, 5.16
[08:27:16] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.60, 6.49, 5.12
[08:28:25] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.77, 7.14, 5.77
[08:29:17] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 6.26, 5.23
[08:30:20] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.65, 6.26, 5.61
[08:30:54] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.35, 5.65, 5.21
[08:31:36] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.58, 5.74, 5.53
[08:33:42] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.76, 18.19, 14.92
[08:34:17] <icinga-miraheze>	 PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates).
[08:34:29] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.39, 3.63, 2.42
[08:34:37] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates).
[08:34:54] <icinga-miraheze>	 RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:34:55] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:35:42] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.90, 16.14, 14.60
[08:35:44] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:35:45] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates).
[08:36:29] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.50, 2.60, 2.19
[08:36:46] <icinga-miraheze>	 PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates).
[09:07:49] <icinga-miraheze>	 PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com
[09:20:41] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 5.64, 4.66, 2.23
[09:24:40] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.56, 2.91, 2.13
[09:31:35] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.78, 6.57, 5.14
[09:32:15] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.95, 6.37, 4.95
[09:33:34] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.42, 7.55, 5.65
[09:34:12] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 7.28, 5.46
[09:34:50] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.23, 7.44, 5.74
[09:35:46] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.12, 7.43, 5.79
[09:36:49] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.02, 6.74, 5.71
[09:37:35] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.06, 7.27, 6.01
[09:37:40] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.08, 6.72, 5.73
[09:38:02] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.29, 6.42, 5.58
[09:39:34] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.70, 6.36, 5.83
[09:48:46] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.32, 7.46, 6.17
[09:49:06] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.75, 6.76, 5.97
[09:49:34] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.29, 7.71, 6.58
[09:49:35] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.85, 7.73, 6.20
[09:50:45] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.63, 7.55, 6.37
[09:51:00] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.51, 6.71, 6.05
[09:51:30] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.78, 7.15, 6.17
[09:51:34] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.89, 7.84, 6.78
[09:52:44] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 8.22, 6.77
[09:55:20] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.18, 7.67, 6.56
[09:56:42] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.09, 7.30, 6.76
[09:57:19] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.01, 6.29, 6.19
[09:57:34] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.38, 6.58, 6.64
[09:58:42] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.55, 6.59, 6.58
[10:00:40] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 2.10, 3.57, 2.13
[10:02:41] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.90, 4.13, 2.49
[10:04:41] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.98, 2.86, 2.22
[12:05:20] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.35, 6.28, 5.07
[12:07:21] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.67, 5.62, 4.98
[12:23:52] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 6.83, 6.08
[12:25:47] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.86, 6.28, 5.96
[13:35:21] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.01, 6.40, 5.31
[13:37:20] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.71, 5.81, 5.24
[14:07:51] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.222.25.132/cpweb
[14:09:15] <icinga-miraheze>	 PROBLEM - www.bluepageswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.bluepageswiki.org could not be found
[14:09:22] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.09, 4.65, 2.66
[14:09:40] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.02, 4.69, 2.96
[14:09:47] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[14:11:05] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.02, 20.06, 16.02
[14:11:19] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.14, 4.08, 2.70
[14:11:39] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.15, 3.93, 2.88
[14:13:02] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.45, 17.74, 15.67
[14:13:40] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.94, 3.27, 2.74
[14:16:07] <icinga-miraheze>	 RECOVERY - www.bluepageswiki.org - reverse DNS on sslhost is OK: rDNS OK - www.bluepageswiki.org reverse DNS resolves to cp11.miraheze.org
[14:34:34] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 6.54, 6.01
[14:35:19] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.87, 7.28, 5.80
[14:36:33] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.74, 6.57, 6.08
[14:37:20] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 7.06, 5.90
[14:45:21] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.67, 6.14, 6.00
[14:52:17] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.86, 6.66, 6.17
[14:54:14] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.27, 5.84, 5.93
[15:04:04] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.16, 6.52, 5.99
[15:05:45] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.88, 8.12, 6.76
[15:06:01] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.53, 5.98, 5.85
[15:06:09] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.96, 7.50, 6.41
[15:06:53] <icinga-miraheze>	 PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu]
[15:07:40] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.33, 7.44, 6.68
[15:08:07] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.42, 7.43, 6.52
[15:09:36] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.29, 6.71, 6.50
[15:09:49] <icinga-miraheze>	 RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org
[15:12:05] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.44, 6.44, 6.32
[15:17:20] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 6.77, 6.56
[15:21:21] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.88, 6.01, 6.32
[15:34:52] <icinga-miraheze>	 RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[15:35:20] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.48, 7.45, 6.70
[15:35:35] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.00, 7.42, 6.30
[15:36:01] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.13, 6.89, 6.26
[15:36:54] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.44, 7.04, 6.05
[15:37:35] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 7.37, 6.42
[15:38:00] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.29, 5.97, 6.00
[15:38:54] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.47, 6.54, 5.99
[15:39:21] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.47, 6.46
[15:40:46] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.58, 5.26, 2.70
[15:41:36] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.55, 6.75, 6.37
[15:46:49] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.56, 3.71, 3.08
[15:48:49] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.21, 2.53, 2.72
[15:50:09] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.82, 6.91, 6.63
[15:52:04] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.60, 6.66, 6.57
[16:02:29] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYivo
[16:02:30] <Not-818c>	 [02miraheze/MirahezeMagic] 07Reception123 0397ea1f7 - fix
[16:03:40] <miraheze-github>	 miraheze/MirahezeMagic - Reception123 the build passed.
[16:04:58] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.32, 6.78
[16:08:19] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.85, 6.97, 6.49
[16:08:59] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.87, 5.80, 6.32
[16:09:40] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYiJe
[16:09:41] <Not-818c>	 [02miraheze/mediawiki] 07Reception123 03d963b60 - Update MirahezeMagic
[16:10:18] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.97, 6.20, 6.26
[16:16:56] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki mw*/jbr*
[16:17:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:18:56] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:19:00] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.64, 7.80, 6.52
[16:20:05] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.04, 7.38, 6.68
[16:21:56] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 7.07, 6.59
[16:22:54] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 7.81, 6.84
[16:23:56] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.40, 6.78
[16:24:54] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.22, 7.88, 6.98
[16:26:50] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.23, 6.97, 6.70
[16:26:54] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.55, 7.62, 6.98
[16:28:06] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.28, 7.38, 6.87
[16:28:13] <icinga-miraheze>	 PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fbpml.org could not be found
[16:28:15] <icinga-miraheze>	 PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tep.wiki could not be found
[16:28:49] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.92, 6.79, 6.68
[16:28:55] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.21, 8.51, 7.35
[16:29:09] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 624.87 ms
[16:32:05] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.69, 7.73, 7.14
[16:32:41] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 7.71, 7.07
[16:34:57] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[16:35:06] <icinga-miraheze>	 RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fbpml.org reverse DNS resolves to cp10.miraheze.org
[16:35:08] <icinga-miraheze>	 RECOVERY - tep.wiki - reverse DNS on sslhost is OK: rDNS OK - tep.wiki reverse DNS resolves to cp10.miraheze.org
[16:36:31] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.10, 7.57, 7.21
[16:38:07] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.77, 7.29, 7.72
[16:38:08] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.04, 5.93, 6.60
[16:38:54] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.31, 6.96, 7.40
[16:40:23] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.04, 6.27, 6.75
[16:42:54] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.40, 5.59, 6.74
[16:45:57] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 2.97, 4.74, 6.42
[16:55:04] <icinga-miraheze>	 RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 246.48 ms
[17:16:39] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.56, 8.50, 4.62
[17:22:35] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.44, 3.19, 3.48
[17:26:33] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.12, 2.53, 3.17
[17:34:07] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 6.11, 5.34
[17:35:26] <SPF|Cloud>	 in short, using the Shell class, you can specify if restrictions have to be enabled -> if enabled, we want to use firejail, but DataDump calls a maintenance script in MediaWiki core
[17:35:44] <SPF|Cloud>	 (if I am correct)
[17:35:54] <SPF|Cloud>	 and if that is the case, there is no need for restrictions
[17:36:05] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.72, 5.85, 5.34
[17:38:03] <paladox>	 !log delete P400 object per request on phabricator
[17:38:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:39:01] <Reception123>	 paladox: if you have time could you please give https://meta.miraheze.org/wiki/Tech:Upgrading_MediaWiki (the slightly updated version) a quick check and make sure none of the steps are incorrect?
[17:39:03] <MirahezeBot>	 [ Tech:Upgrading MediaWiki - Miraheze Meta ] - meta.miraheze.org
[17:42:29] <Universal_Omega>	 SPF|Cloud: Oh. Then I guess I did something wrong when setting up firejail in the beginning. If that's the case, then we can do that. https://doc.wikimedia.org/mediawiki-core/master/php/classMediaWiki_1_1Shell_1_1Command.html#a5d7047e8d33f753fed310b26a5168c44 I think then, unless I am completely mistaken or misunderstanding? But thanks for the guidance in that.
[17:42:30] <MirahezeBot>	 [ MediaWiki: MediaWiki\Shell\Command Class Reference ] - doc.wikimedia.org
[17:43:50] <SPF|Cloud>	 RESTRICT_NONE sounds right
[18:02:45] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.55, 6.25, 5.14
[18:06:10] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 7.03, 5.60
[18:06:47] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.95, 5.73
[18:08:24] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYi4e
[18:08:26] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega 03b779551 - Fix T7068: Add 'useRestriction' option to 'generate'
[18:08:27] <Not-818c>	 [02DataDump] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fhhKV
[18:08:29] <Not-818c>	 [02DataDump] 07Universal-Omega opened pull request 03#21: Fix T7068: Add 'useRestriction' option to 'generate' - 13https://git.io/JYi4v
[18:09:30] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build has errored.
[18:10:00] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.06, 6.66, 5.78
[18:11:38] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYi42
[18:11:39] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega 037279abe - Update DataDumpGenerateJob.php
[18:11:41] <Not-818c>	 [02DataDump] 07Universal-Omega synchronize pull request 03#21: Fix T7068: Add 'useRestriction' option to 'generate' - 13https://git.io/JYi4v
[18:12:45] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build has errored.
[18:16:49] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.09, 7.68, 6.65
[18:17:35] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYiBz
[18:17:37] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega 033b33bb2 - Update DataDumpGenerateJob.php
[18:17:38] <Not-818c>	 [02DataDump] 07Universal-Omega synchronize pull request 03#21: Fix T7068: Add 'useRestriction' option to 'generate' - 13https://git.io/JYi4v
[18:18:35] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build passed.
[18:19:36] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.37, 8.29, 6.82
[18:25:39] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.44, 20.07, 16.30
[18:26:29] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.58, 5.06, 2.84
[18:27:40] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.58, 18.02, 16.05
[18:28:29] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.75, 3.46, 2.52
[18:29:23] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.60, 6.92, 7.05
[18:30:29] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.50, 2.43, 2.25
[18:30:43] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.03, 6.70, 7.50
[18:31:20] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.44, 6.03, 6.70
[18:35:22] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.58, 6.77, 6.86
[18:35:56] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 6.89, 5.91
[18:38:41] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.32, 5.81, 6.80
[18:39:21] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.37, 6.23, 6.64
[18:39:54] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.68, 7.94, 6.54
[18:41:51] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.80, 7.12, 6.40
[18:41:57] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYiEj
[18:41:59] <Not-818c>	 [02miraheze/DataDump] 07Universal-Omega 031963116 - Fix
[18:42:00] <Not-818c>	 [02DataDump] 07Universal-Omega synchronize pull request 03#21: Fix T7068: Add 'useRestriction' option to 'generate' - 13https://git.io/JYi4v
[18:43:05] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build passed.
[18:43:53] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.62, 6.36, 6.22
[18:52:38] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.21, 6.72, 6.61
[18:54:10] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 6.90, 6.67
[18:54:37] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.58, 6.20, 6.43
[18:56:05] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.27, 6.46
[19:04:35] <RhinosF1>	 !sre
[19:04:45] <Reception123>	 RhinosF1: yes?
[19:05:10] <RhinosF1>	 Reception123: wikimedia image issues again, things look slow but hopefully nothing will crash
[19:05:33] <RhinosF1>	 I have no clue if we have graphs of healthcheck response times
[19:05:50] <Reception123>	 ok, let's keep an eye on that then
[19:09:10] <RhinosF1>	 Reception123: looks like accidental DOS from a bot. Any clue how we can see response times in Grafana?
[19:09:13] <RhinosF1>	 paladox: ^
[19:09:21] <RhinosF1>	 Especially like average icinga is seeing
[19:09:44] <paladox>	 We don't really have any of those statistics
[19:10:23] <paladox>	 apart from https://grafana.miraheze.org/d/xtkCtBkiz/prometheus-blackbox-exporter-test-ferran-tufan?orgId=1 and even then it's only limited to wikis that doesn't exist
[19:10:24] <MirahezeBot>	 [ Grafana ] - grafana.miraheze.org
[19:10:31] <RhinosF1>	 paladox: are they possible?
[19:11:03] <paladox>	 i guess so. i would imagine so. But you would need to find a way to do it.
[19:11:53] <RhinosF1>	 I'll file a task
[19:14:42] <RhinosF1>	 https://phabricator.miraheze.org/T7087
[19:14:43] <MirahezeBot>	 [ ⚓ T7087 Add (rolling average) response time to grafana ] - phabricator.miraheze.org
[19:25:14] <JohnLewis>	 Why is that tagged as SRE (master project)?
[19:27:16] <RhinosF1>	 JohnLewis: because we'll need your help doing it and MediaWiki is in part our job
[19:27:16] <dmehus>	 JohnLewis, I assume you mean https://phabricator.miraheze.org/tag/site_reliability_engineering/ is the master project, right? I ask that only because I see that project tagged so infrequently
[19:27:17] <MirahezeBot>	 [ Site Reliability Engineering · Workboard ] - phabricator.miraheze.org
[19:27:32] <RhinosF1>	 dmehus: yes as oppose to MediaWiki/infra
[19:27:51] <JohnLewis>	 It doesn’t need to be tagged as Infra to ask someone in Infra for help 
[19:28:09] <JohnLewis>	 It’s only tagged as Infra when the primary responsibility is Infra
[19:28:50] <RhinosF1>	 Changed then
[19:29:00] <RhinosF1>	 But someone will need to help me get the stats
[19:29:59] <JohnLewis>	 I’m sure Paladox can assist, or Reception can as an SRE
[19:35:20] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 6.11, 5.33
[19:37:21] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.11, 5.19, 5.08
[20:06:17] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.36, 6.91, 5.99
[20:08:17] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.31, 6.00, 5.77
[20:08:27] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.19, 6.98, 6.38
[20:10:28] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.65, 6.70, 6.37
[20:44:45] <Not-818c>	 [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYiPx
[20:44:47] <Not-818c>	 [02miraheze/WikiDiscover] 07Universal-Omega 037710d63 - Fix creation date format
[20:45:46] <miraheze-github>	 miraheze/WikiDiscover - Universal-Omega the build passed.
[20:46:19] <Not-818c>	 [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYiXJ
[20:46:20] <Not-818c>	 [02miraheze/WikiDiscover] 07Universal-Omega 03dc538a5 - Formatting
[20:47:51] <miraheze-github>	 miraheze/WikiDiscover - Universal-Omega the build passed.
[21:03:19] <Not-818c>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYi1V
[21:03:20] <Not-818c>	 [02miraheze/mw-config] 07Universal-Omega 03d691f1a - Update docs relating to LocalExtensions/LocalSettings
[21:04:26] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[21:07:17] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 6.82, 6.00
[21:09:10] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.50, 6.10, 5.84
[21:53:41] <RhinosF1>	 SPF|Cloud: I'm going to sleep but why when I asked if we had a graph showing response times for health check a few hours ago was I told it didn't exist
[21:54:13] <SPF|Cloud>	 it does
[21:54:39] <SPF|Cloud>	 https://grafana.miraheze.org/d/xtkCtBkiz/prometheus-blackbox-exporter-test-ferran-tufan?orgId=1 it is not in 'production' stage, but it's there
[21:54:40] <MirahezeBot>	 [ Grafana ] - grafana.miraheze.org
[21:55:06] <RhinosF1>	 I wanted to see when wikimedia had their thumbor incident that we weren't close to depool times and how much it rose by
[21:55:19] <RhinosF1>	 20:10:23 <paladox> apart from https://grafana.miraheze.org/d/xtkCtBkiz/prometheus-blackbox-exporter-test-ferran-tufan?orgId=1 and even then it's only limited to wikis that doesn't exist
[21:55:20] <MirahezeBot>	 [ Grafana ] - grafana.miraheze.org
[21:55:27] <RhinosF1>	 That's only limited to wikis that don't exist
[21:55:36] <RhinosF1>	 Which I was confused by but yeah
[21:55:49] <SPF|Cloud>	 uh, no
[21:56:02] <SPF|Cloud>	 behind the scenes the Host is converted to meta.miraheze.org
[21:56:32] <SPF|Cloud>	 and X-Miraheze-Debug is used to make the exporter bypass the varnish cache
[21:56:44] <SPF|Cloud>	 otherwise you would get strange graphs :)
[21:58:24] <RhinosF1>	 https://grafana.miraheze.org/d/xtkCtBkiz/prometheus-blackbox-exporter-test-ferran-tufan?viewPanel=138&orgId=1&var-interval=10s&var-target=https:%2F%2Fcp10.miraheze.org%2Fwiki%2FMain_Page&var-target=https:%2F%2Fcp11.miraheze.org%2Fwiki%2FMain_Page&var-target=https:%2F%2Fcp12.miraheze.org%2Fwiki%2FMain_Page&var-target=https:%2F%2Fcp3.miraheze.org%2Fwiki%2FMain_Page&from=1617386400000&to=1617393600000
[21:58:25] <MirahezeBot>	 [ Grafana ] - grafana.miraheze.org
[21:58:30] <RhinosF1>	 That looks okay to me
[21:58:59] <RhinosF1>	 I don't see any rise around 8 pm so \o/
[22:21:42] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.93, 1.54, 1.13
[22:23:43] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.24, 1.41, 1.13
[22:30:09] <Not-818c>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYi5h
[22:30:10] <Not-818c>	 [02miraheze/services] 07MirahezeSSLBot 03b6d12be - BOT: Updating services config for wikis
[22:31:26] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.93, 2.88, 1.57
[22:33:25] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.70, 1.95, 1.38
[23:18:31] <SPF|Cloud>	 in hindsight, supporting a traffic growth of 267% with only 37% extra budget was not the most realistic idea
[23:24:45] <SPF|Cloud>	 night