[00:03:23] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 39.60, 37.64, 27.40
[00:19:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.18, 22.33, 23.91
[00:23:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.56, 22.68, 23.51
[00:27:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.31, 22.46, 23.30
[00:33:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.32, 23.72, 23.34
[00:35:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.86, 22.35, 22.86
[00:41:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.46, 22.87, 22.48
[00:45:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.54, 22.92, 22.79
[00:49:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.51, 22.80, 22.69
[00:57:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.68, 23.44, 23.18
[01:03:01] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.54, 23.00, 22.68
[01:07:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.13, 22.65, 22.61
[01:13:01] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.06, 23.79, 22.95
[01:13:07] <Zppix>	 paladox:  ^
[01:15:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.16, 21.15, 22.06
[01:23:01] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.75, 21.48, 21.46
[01:25:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.17, 20.08, 20.95
[01:31:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.60, 22.66, 21.65
[01:33:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.05, 23.40, 22.08
[01:44:32] <MH-Discord>	 <paladox> @Zppix ?
[01:45:17] <Zppix>	 @paladox  icinga keeps throwing load alerts for cloud2
[01:45:31] <MH-Discord>	 <paladox> Oh, known.
[01:47:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.00, 19.58, 20.39
[01:51:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.42, 21.91, 21.18
[02:11:20] <icinga-miraheze>	 PROBLEM - misc1 APT on misc1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates).
[02:21:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.71, 18.54, 20.18
[02:53:46] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.62, 21.42, 19.85
[02:55:58] <Not-edb>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJjJp
[02:56:00] <Not-edb>	 [02miraheze/puppet] 07paladox 031c56617 - jobrunner: Only enable crons on jobrunner1`
[02:56:01] <Not-edb>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[02:56:03] <Not-edb>	 [02puppet] 07paladox opened pull request 03#1483: jobrunner: Only enable crons on jobrunner1 - 13https://git.io/JJjJh
[02:56:37] <Not-edb>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJjUv
[02:56:38] <Not-edb>	 [02miraheze/puppet] 07paladox 0374671d5 - Update jobrunner1.yaml
[02:56:40] <Not-edb>	 [02puppet] 07paladox synchronize pull request 03#1483: jobrunner: Only enable crons on jobrunner1 - 13https://git.io/JJjJh
[02:57:33] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.13, 22.55, 20.50
[02:58:31] <Not-edb>	 [02puppet] 07paladox closed pull request 03#1483: jobrunner: Only enable crons on jobrunner1 - 13https://git.io/JJjJh
[02:58:33] <Not-edb>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJjUT
[02:58:34] <Not-edb>	 [02miraheze/puppet] 07paladox 03e59b7a6 - jobrunner: Only enable crons on jobrunner1 (#1483) * jobrunner: Only enable crons on jobrunner1` * Update jobrunner1.yaml
[02:58:36] <Not-edb>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[02:58:37] <Not-edb>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[03:01:22] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.89, 22.10, 20.81
[03:01:37] <Not-edb>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJjUO
[03:01:38] <Not-edb>	 [02miraheze/puppet] 07paladox 03e7015e1 - jobrunner: Reduce runners to 2 Per @Southparkfan, we are over utilising, so going to reduce some of the pressure by lowering the amount of runners.
[03:01:40] <Not-edb>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[03:01:41] <Not-edb>	 [02puppet] 07paladox opened pull request 03#1484: jobrunner: Reduce runners to 2 - 13https://git.io/JJjU3
[03:02:09] <Not-edb>	 [02puppet] 07paladox closed pull request 03#1484: jobrunner: Reduce runners to 2 - 13https://git.io/JJjU3
[03:02:10] <Not-edb>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjUs
[03:02:12] <Not-edb>	 [02miraheze/puppet] 07paladox 031dfc8ad - jobrunner: Reduce runners to 2 (#1484) Per @Southparkfan, we are over utilising, so going to reduce some of the pressure by lowering the amount of runners.
[03:02:13] <Not-edb>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[03:02:15] <Not-edb>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[03:09:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.66, 18.88, 20.13
[03:13:46] <paladox>	 !log reduce jobrunner2 cores by 1 (so 3 cores in total)
[03:13:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:23:42] <paladox>	 !log reduce jobrunner1 cores by 1 (so 3 cores in total)
[03:23:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:33:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.35, 22.77, 20.39
[03:35:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.56, 19.64, 19.50
[03:38:10] <paladox>	 !log root@jobrunner2:/home/paladox# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Cargo/sql/cargo_tables.patch.field_helper_tables.mssql.sql - T6070
[03:38:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:39:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.24, 20.77, 19.96
[03:41:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.06, 20.12, 19.81
[03:41:01] <Not-edb>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJjkU
[03:41:03] <Not-edb>	 [02miraheze/mediawiki] 07paladox 03a4706f4 - Update Cargo to 2.7
[03:41:39] <Not-edb>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJjkk
[03:41:41] <Not-edb>	 [02miraheze/mediawiki] 07paladox 03afdf74d - Update PageForms
[03:44:54] <Not-edb>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJjkn
[03:44:56] <Not-edb>	 [02miraheze/mediawiki] 07paladox 030770fd4 - Update Citizen
[03:45:01] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.29, 23.23, 20.99
[03:48:24] <paladox>	 !log rebuild lc on mw* and jobrunner*
[03:48:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:49:41] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[03:49:53] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[03:50:17] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[03:50:40] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[03:50:48] <icinga-miraheze>	 PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[03:51:05] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[03:51:21] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[03:52:47] <icinga-miraheze>	 RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[03:53:04] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[03:53:23] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[03:53:39] <icinga-miraheze>	 RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:53:40] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.50, 17.09, 11.45
[03:53:57] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:54:23] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:54:42] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:55:39] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 11.96, 14.88, 11.29
[04:03:18] <paladox>	 !log root@jobrunner2:/home/paladox# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Cargo/sql/cargo_tables.patch.field_helper_tables.sql
[04:03:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:03:29] <paladox>	 T6070
[04:03:33] <icinga-miraheze>	 PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.28, 6.81, 5.24
[04:03:33] <MirahezeBot>	 https://phabricator.miraheze.org/T6070 - Apply 2017! change to cargo table, authored by RhinosF1, assigned to None, Priority: Normal, Status: Open
[04:05:33] <icinga-miraheze>	 RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.74, 5.96, 5.12
[04:23:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.60, 19.77, 23.92
[04:31:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.26, 21.46, 22.68
[04:33:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.80, 20.90, 22.35
[04:41:07] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.42, 26.36, 23.69
[04:45:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.06, 23.18, 23.06
[04:59:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.43, 16.45, 19.46
[05:05:00] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.54, 20.09, 20.46
[05:09:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.27, 18.45, 19.77
[05:33:15] <icinga-miraheze>	 PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 5.38, 3.37, 1.48
[05:33:40] <icinga-miraheze>	 PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[05:35:38] <icinga-miraheze>	 RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[05:37:12] <icinga-miraheze>	 RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.05, 1.48, 1.49
[05:50:52] <Reception123>	 !log added discord webhook for ficreationwiki
[05:50:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:07:59] <Reception123>	 !log renamed gloriusdawnwiki to gloriousdawnwiki.
[06:08:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:15:35] <Not-edb>	 [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JJjq2
[06:15:37] <Not-edb>	 [02miraheze/ssl] 07Reception123 031261c3d - add wiki.dom64.xyz cert
[06:15:38] <Not-edb>	 [02miraheze/ssl] 07Reception123 036f61faf - add wiki.yunachannel.com cert
[06:15:50] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 55%
[06:17:49] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 31%
[06:22:39] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.73, 20.73, 19.65
[06:24:37] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.34, 18.24, 18.88
[06:26:34] <Reception123>	 !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki awesomegameswiki --r "Requested - T6068" /home/reception/awesomegamesdel.txt
[06:26:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:27:51] <Reception123>	 !log reception@jobrunner2:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki rottenwebsiteswiki --r "Requested - T6068" /home/reception/rottenwebsitesdel.txt
[06:27:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:43:03] <icinga-miraheze>	 RECOVERY - misc1 APT on misc1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[06:51:16] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.06, 21.89, 19.46
[06:53:14] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.84, 22.10, 19.85
[06:59:10] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.11, 20.26, 19.85
[07:43:43] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.20, 21.12, 19.78
[07:45:41] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.63, 19.16, 19.22
[08:02:07] <Reception123>	 !log MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = true WHERE id = '35'; (User resigned)
[08:02:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:24:16] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.16, 23.00, 21.03
[08:26:14] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.69, 20.23, 20.24
[08:34:07] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.77, 20.23, 19.89
[08:36:05] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.65, 19.30, 19.57
[09:23:13] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.79, 20.64, 18.42
[09:25:11] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.93, 18.95, 18.08
[09:33:04] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.02, 20.43, 18.99
[09:37:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.73, 17.63, 18.33
[10:11:20] <Reception123>	 !log reception@jobrunner1:/srv/mediawiki/w/extensions/CreateWiki/maintenance$ sudo -u www-data php deleteWikis.php --wiki loginwiki --delete Reception123
[10:11:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:15:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjCX
[10:15:15] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03f4b756a - BOT: Updating services config for wikis
[10:20:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjCx
[10:20:20] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03f49e82e - BOT: Updating services config for wikis
[10:21:24] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.20, 22.86, 20.07
[10:25:12] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjWO
[10:25:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03a0d3a89 - BOT: Updating services config for wikis
[10:25:20] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.29, 20.44, 19.98
[10:27:18] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.40, 20.10, 19.90
[10:30:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjW8
[10:30:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03429aa5a - BOT: Updating services config for wikis
[10:31:14] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 34.57, 25.64, 22.03
[10:35:11] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjWa
[10:35:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 032bcc2b0 - BOT: Updating services config for wikis
[10:35:14] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.17, 21.75, 21.27
[10:40:16] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjWM
[10:40:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0315904ce - BOT: Updating services config for wikis
[10:41:10] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.01, 23.59, 21.88
[10:43:08] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.63, 21.47, 21.35
[10:45:07] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.00, 23.23, 22.01
[10:45:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/services/compare/15904cedf5c7...708ea1f81849
[10:45:17] <MirahezeBot>	 [ Comparing 15904cedf5c7...708ea1f81849 · miraheze/services · GitHub ] - github.com
[10:45:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03708ea1f - BOT: Updating services config for wikis
[10:47:05] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.35, 22.86, 22.03
[10:49:05] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.10, 23.58, 22.40
[10:50:16] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjlI
[10:50:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0326ab99c - BOT: Updating services config for wikis
[10:54:59] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.13, 20.50, 21.76
[10:55:11] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjl3
[10:55:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03833bb18 - BOT: Updating services config for wikis
[10:56:56] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.90, 21.74, 22.03
[10:58:54] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.27, 19.94, 21.31
[11:00:53] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.98, 22.43, 22.02
[11:01:00] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjlB
[11:01:00] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 036aae921 - BOT: Updating services config for wikis
[11:05:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjl1
[11:05:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03f28ae56 - BOT: Updating services config for wikis
[11:07:56] <RhinosF1>	 Reception123: why so many wikis at once?
[11:10:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjlh
[11:10:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03a7d81f4 - BOT: Updating services config for wikis
[11:15:16] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/services/compare/a7d81f426b0f...70a522d353e0
[11:15:16] <MirahezeBot>	 [ Comparing a7d81f426b0f...70a522d353e0 · miraheze/services · GitHub ] - github.com
[11:15:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0370a522d - BOT: Updating services config for wikis
[11:20:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj8B
[11:20:20] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0343062ce - BOT: Updating services config for wikis
[11:21:21] <Reception123>	 RhinosF1: deletedwikis is big
[11:21:31] <RhinosF1>	 Reception123: how many?
[11:21:44] <RhinosF1>	 Can we please cross check some of that list
[11:22:37] <Reception123>	 RhinosF1: I mean there's not many ways of cross checking since they're deleted. And I told you, it's because the script didn't work for months at one period 
[11:23:08] <RhinosF1>	 Reception123: okay, I'm gonna leave it at blaming that on they're all deleted wiki
[11:23:10] <RhinosF1>	 And yeah
[11:23:22] <Reception123>	 RhinosF1: in any case, there's some XML backups
[11:23:54] <RhinosF1>	 Reception123: I meant compare the list of wikis being deleted to the ones being pulled from services
[11:24:05] * RhinosF1 hopes they do
[11:24:14] <Reception123>	 RhinosF1: oh
[11:25:12] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj82
[11:25:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 037a4745d - BOT: Updating services config for wikis
[11:30:16] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj8y
[11:30:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03ead8f19 - BOT: Updating services config for wikis
[11:35:12] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj8p
[11:35:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0342035aa - BOT: Updating services config for wikis
[11:38:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.16, 21.65, 23.79
[11:40:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj4k
[11:40:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03d9f9f08 - BOT: Updating services config for wikis
[11:40:23] <icinga-miraheze>	 RECOVERY - gluster2 Disk Space on gluster2 is OK: DISK OK - free space: / 36720 MB (11% inode=77%);
[11:42:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.23, 26.28, 25.03
[11:45:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj4s
[11:45:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 034a4d469 - BOT: Updating services config for wikis
[11:50:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj40
[11:50:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 035db14d4 - BOT: Updating services config for wikis
[11:55:15] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj4i
[11:55:16] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 031ecf8bb - BOT: Updating services config for wikis
[12:00:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj4S
[12:00:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03ac1c662 - BOT: Updating services config for wikis
[12:05:11] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjBe
[12:05:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03317b65e - BOT: Updating services config for wikis
[12:10:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjBO
[12:10:20] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03d1fd0c8 - BOT: Updating services config for wikis
[12:15:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/services/compare/d1fd0c8ee67f...9ba99fc82ac2
[12:15:17] <MirahezeBot>	 [ Comparing d1fd0c8ee67f...9ba99fc82ac2 · miraheze/services · GitHub ] - github.com
[12:15:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 039ba99fc - BOT: Updating services config for wikis
[12:16:10] <icinga-miraheze>	 RECOVERY - gluster1 Disk Space on gluster1 is OK: DISK OK - free space: / 75663 MB (11% inode=84%);
[12:16:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.53, 21.48, 23.87
[12:20:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjBa
[12:20:21] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0323545bb - BOT: Updating services config for wikis
[12:20:30] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.91, 22.41, 23.57
[12:25:12] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjBj
[12:25:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03fea00cb - BOT: Updating services config for wikis
[12:26:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.90, 20.57, 22.57
[12:28:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.12, 23.37, 23.41
[12:30:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjRK
[12:30:20] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03d6564a1 - BOT: Updating services config for wikis
[12:34:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.81, 21.52, 22.87
[12:35:11] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjR9
[12:35:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03f168437 - BOT: Updating services config for wikis
[12:40:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjRj
[12:40:20] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 031971517 - BOT: Updating services config for wikis
[12:45:12] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj0O
[12:45:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 033527788 - BOT: Updating services config for wikis
[12:50:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj0V
[12:50:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 034d80ac4 - BOT: Updating services config for wikis
[12:50:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.07, 21.83, 21.61
[12:52:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.38, 22.23, 21.77
[12:54:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.93, 22.31, 21.78
[12:55:11] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJj0A
[12:55:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0392e0897 - BOT: Updating services config for wikis
[13:00:20] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjEl
[13:00:21] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 036326f3a - BOT: Updating services config for wikis
[13:05:11] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjEX
[13:05:13] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 031c63529 - BOT: Updating services config for wikis
[13:06:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.16, 23.31, 23.63
[13:08:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 35.57, 28.08, 25.31
[13:10:17] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjEh
[13:10:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 032b28e3f - BOT: Updating services config for wikis
[13:12:13] <Zppix>	 !log initSiteStats (ran on jobrunner2) and rebuildRC (ran on jobrunner1) for ficreationwiki following import completion (T6067) 
[13:12:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:15:12] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjum
[13:15:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 030883fcc - BOT: Updating services config for wikis
[13:20:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjuy
[13:20:20] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03fb9b4ef - BOT: Updating services config for wikis
[13:25:12] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjzO
[13:25:14] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03d6c02b0 - BOT: Updating services config for wikis
[13:28:13] <Reception123>	 !log dropped all wikis listed at https://phabricator.miraheze.org/P344
[13:28:14] <MirahezeBot>	 [ ✎ P344 Deleted wikis 21/08/2020 ] - phabricator.miraheze.org
[13:28:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:30:16] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjz6
[13:30:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 0332a4d7d - BOT: Updating services config for wikis
[13:30:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.55, 21.62, 23.80
[13:31:20] <dmehus>	 ❤️ [[phab:P344]]. Thanks, @Reception123 ;)
[13:31:50] <dmehus>	 Oh, right, linkie doesn't support interwiki links yet...have to upstream that request :(
[13:32:03] <Reception123>	 heh :)
[13:32:09] <RhinosF1>	 *linkie on
[13:32:10] <Reception123>	 dmehus: no problem :) that was a lot of wikis
[13:32:52] <dmehus>	 @Reception123 Yeah, @RhinosF1 was saying the deletion code was broken, so this will be a much bigger batch than normal, I suspect, now that it's fixed.
[13:32:59] <RhinosF1>	 dmehus: should work will just like to meta
[13:33:19] <RhinosF1>	 .mh phab P344
[13:33:19] <MirahezeBot>	 https://phab.miraheze.org/wiki/P344
[13:33:35] <RhinosF1>	 .mh phab:P344
[13:33:35] <MirahezeBot>	 https://meta.miraheze.org/wiki/phab:P344
[13:33:53] <dmehus>	 @RhinosF1, okay, let me try again. Ah, you used a @MirahezeBot command. I will use that instead. That's better anyway, since, I'm guessing linkie uses WMF's interwiki table
[13:34:17] <RhinosF1>	 dmehus: I wonder how we can build the Interwiki map. I guess that's another thing for json caching
[13:34:21] <RhinosF1>	 Create a task
[13:34:28] <RhinosF1>	 We'd have to save a map
[13:35:13] <dmehus>	 @RhinosF1 okay, could we set up a crontab and have it resync from Meta's global interwiki table maybe?
[13:35:31] <dmehus>	 s/maybe?/maybe every week or so?
[13:35:31] <MirahezeBot>	 dmehus meant to say: @RhinosF1 okay, could we set up a crontab and have it resync from Meta's global interwiki table maybe every week or so?
[13:36:59] <RhinosF1>	 dmehus: no crontab. It'll be a command like .reloadinterwiki that updates an on disk json map from hopefully an API. After that then we'd have to use the json cache infra to reload the stored data in the bots memory
[13:38:07] <dmehus>	 @RhinosF1, ah, okay, by "hopefully an API," do you mean subject to the interwiki table being something that can be parsed by the mediawiki API?
[13:38:20] <RhinosF1>	 dmehus: yes
[13:38:28] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.07, 14.78, 19.85
[13:38:51] <dmehus>	 @RhinosF1, ah, okay, that makes sense. Simpler than running a crontab and any opped/voiced user could, presumably, refresh that, too.
[13:39:06] <RhinosF1>	 Any bot admin
[13:39:15] <RhinosF1>	 Or maybe op in here
[13:39:17] <RhinosF1>	 Depends
[13:40:11] <dmehus>	 @RhinosF1, ah, okay, yeah, so it would only need to be done once, in one channel in which @MirahezeBot is in? It wouldn't need to be refreshed in every channel, since the same bot operates across all channels. Assuming that's the case, that seems fine.
[13:40:23] <RhinosF1>	 Yep
[13:40:45] <RhinosF1>	 dmehus: defining the access list needs to be fairly trusted
[13:40:57] <dmehus>	 yeah
[13:40:59] <RhinosF1>	 Because breakages could corrupt on disk data
[13:41:25] <RhinosF1>	 But using getchanops() isn't advised as it's not public
[13:41:40] <dmehus>	 @RhinosF1 ah, yeah, and we're indirectly granting write access to the server by updating that json map, too.
[13:41:52] <RhinosF1>	 Yes we are
[13:42:00] <RhinosF1>	 But it should be safe in theory
[13:42:08] <dmehus>	 ah
[13:52:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.65, 21.73, 20.59
[13:56:28] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.68, 19.18, 19.97
[14:02:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.57, 24.03, 21.65
[14:04:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.61, 23.07, 21.62
[14:05:46] <RhinosF1>	 dmehus: https://meta.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=interwikimap exists
[14:05:46] <MirahezeBot>	 [ MediaWiki API result - Miraheze Meta ] - meta.miraheze.org
[14:10:03] <dmehus>	 @RhinosF1, ah, that's great. I figured it might, but only because I noticed a page on Meta Wikimedia (https://meta.wikimedia.org/) where they specifically call it an "interwiki map." Will file a task later this morning on Bots Phab. :)
[14:10:04] <MirahezeBot>	 [ Meta, a Wikimedia project coordination wiki ] - meta.wikimedia.org
[14:10:29] <RhinosF1>	 dmehus: ack
[14:18:28] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.11, 19.31, 20.25
[14:18:51] <RhinosF1>	 .status mhmeta On
[14:18:53] <MirahezeBot>	 RhinosF1: ⧼abusefilter-warning-userpage⧽
[14:19:01] <RhinosF1>	 Huh, I guess login is broke
[14:23:59] * hispano76 greetings
[14:24:27] <Sario>	 Hello hispano76 
[14:25:20] <dmehus>	 hi @hispano76 and hi @sario
[14:25:22] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.50, 20.07, 20.61
[14:27:20] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.49, 18.52, 20.00
[14:30:18] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjV0
[14:30:19] <Not-edb>	 [02miraheze/services] 07MirahezeSSLBot 03086e8d4 - BOT: Updating services config for wikis
[14:42:03] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.29, 20.18, 19.79
[14:44:02] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.19, 17.09, 18.71
[14:44:40] <RhinosF1>	 .status mhtest test
[14:44:42] <MirahezeBot>	 RhinosF1: '''Your IP address has been blocked on all wikis.''' The block was made by [https://meta.miraheze.org/wiki/User:Reception123 Reception123] (meta.miraheze.org). The reason given is ''Belongs to MirahezeBots. Requested by owner.''. * Start of block: 14:42, 21 August 2020 * Expiration of block: 14:42, 21 August 2021 You can contact [https://meta.miraheze.org/wiki/User:Reception123 Reception123] to discuss the block. You cannot use the "E
[14:45:01] <RhinosF1>	 That's more helpful than locally blocked message but yet
[14:45:03] <RhinosF1>	 Yey
[14:52:39] <dmehus>	 agreed
[14:54:03] <RhinosF1>	 dmehus: it can't edit logged out now
[14:57:33] <dmehus>	 @RhinosF1, ah, that's good. Was it easier to block it than update the code to prevent it from editing if it wasn't logged in?
[14:57:43] <dmehus>	 s/block it/block its IP
[14:57:44] <MirahezeBot>	 dmehus meant to say: @RhinosF1, ah, that's good. Was it easier to block its IP than update the code to prevent it from editing if it wasn't logged in?
[14:58:25] <RhinosF1>	 dmehus: yes much easier
[14:58:37] <RhinosF1>	 Seen as we don't even read the reply to the login request
[14:59:00] <RhinosF1>	 I can fix the config later if no one else does to make it work again
[14:59:30] <dmehus>	 @RhinosF1, ah, that makes sense, because it doesn't really know "when" it is logged in, just like it can't know what posts or pages it has already "read"
[15:01:00] <RhinosF1>	 dmehus: I mean it could if we improve the code but the login code is taken from upstreams examples
[15:01:14] <RhinosF1>	 But yeah it doesn't guard against a failed login
[15:01:20] <dmehus>	 @RhinosF1 ah
[15:03:40] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.74, 19.90, 19.22
[15:05:38] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.53, 20.08, 19.39
[15:21:39] <dmehus>	 @RhinosF1, I just noticed I did ask about the GUP migration possibility back in July actually...forgot about that. https://meta.miraheze.org/wiki/User_talk:MirahezeBots#Migrate_this_user_page_to_Login_wiki? :P
[15:21:40] <MirahezeBot>	 [ User talk:MirahezeBots - Miraheze Meta ] - meta.miraheze.org
[15:26:30] <Not-edb>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjoj
[15:26:32] <Not-edb>	 [02miraheze/CreateWiki] 07translatewiki 037e6e481 - Localisation updates from https://translatewiki.net.
[15:26:33] <MirahezeBot>	 [ Main page - translatewiki.net ] - translatewiki.net
[15:26:33] <Not-edb>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjKe
[15:26:35] <Not-edb>	 [02miraheze/MirahezeMagic] 07translatewiki 0375eae8a - Localisation updates from https://translatewiki.net.
[15:26:36] <MirahezeBot>	 [ Main page - translatewiki.net ] - translatewiki.net
[15:26:36] <Not-edb>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjKv
[15:26:38] <Not-edb>	 [02miraheze/ManageWiki] 07translatewiki 0363683da - Localisation updates from https://translatewiki.net.
[15:26:39] <MirahezeBot>	 [ Main page - translatewiki.net ] - translatewiki.net
[15:26:39] <Not-edb>	 [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjKf
[15:26:41] <Not-edb>	 [02miraheze/WikiDiscover] 07translatewiki 030ca56a6 - Localisation updates from https://translatewiki.net.
[15:26:42] <MirahezeBot>	 [ Main page - translatewiki.net ] - translatewiki.net
[15:27:56] <icinga-miraheze>	 RECOVERY - bacula2 Bacula Static 1 on bacula2 is OK: OK: Full, 11404431 files, 584.4GB, 2020-08-21 15:26:00 (1.9 minutes ago)
[16:10:49] <RhinosF1>	 dmehus: just do it
[16:17:08] <dmehus>	 @RhinosF1, okay, thanks (assuming you're meaning any common sense, useful changes to MirahezeBots' GUP)
[16:25:48] <Not-edb>	 [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_34 [+0/-0/±4] 13https://git.io/JJjPN
[16:25:49] <Not-edb>	 [02miraheze/mediawiki] 07Pix1234 03af3c52e - Update CreateWiki/ManageWiki/MirahezeMagic/WikiDiscover
[16:30:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.64, 21.40, 18.21
[16:35:12] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:35:31] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:36:06] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:36:40] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:36:59] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:37:01] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:38:58] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:39:32] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[16:40:08] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:42:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.17, 23.62, 21.79
[16:46:28] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.12, 16.64, 19.33
[16:52:58] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[16:57:09] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:31:05] <icinga-miraheze>	 PROBLEM - bacula2 Disk Space on bacula2 is WARNING: DISK WARNING - free space: / 103794 MB (10% inode=99%);
[17:32:31] <icinga-miraheze>	 PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 53%
[17:36:25] <icinga-miraheze>	 RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 24%
[17:36:53] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 34.98, 23.00, 19.01
[17:38:51] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.01, 22.29, 19.27
[17:40:50] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.52, 20.09, 18.84
[17:44:45] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.23, 22.16, 20.13
[17:56:05] <icinga-miraheze>	 PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 47%
[17:58:02] <icinga-miraheze>	 RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 13%
[18:00:34] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.50, 17.16, 19.31
[19:04:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.49, 17.39, 15.23
[19:06:28] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.98, 15.27, 14.69
[19:33:04] <icinga-miraheze>	 PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: DISK CRITICAL - free space: / 56432 MB (5% inode=99%);
[21:28:16] <icinga-miraheze>	 PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 145s
[21:59:52] <dmehus>	 @System Administrators, is jobrunner stuck? It's been ~25 minutes since @Voidwalker started a global rename.
[22:00:23] <dmehus>	 Usually there's one or two attached accounts by that timeframe.
[23:35:11] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJjx7
[23:35:13] <Not>	 [02miraheze/services] 07MirahezeSSLBot 03fd51d2a - BOT: Updating services config for wikis