[00:15:11] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUUAf
[00:15:13] <Not>	 [02miraheze/services] 07MirahezeSSLBot 033fc4e76 - BOT: Updating services config for wikis
[01:01:09] <icinga-miraheze>	 PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[01:47:51] <icinga-miraheze>	 PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[04:05:10] <icinga-miraheze>	 PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[04:25:43] <icinga-miraheze>	 PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[04:42:07] <icinga-miraheze>	 PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[04:55:10] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUTfG
[04:55:12] <Not>	 [02miraheze/services] 07MirahezeSSLBot 03dd07adc - BOT: Updating services config for wikis
[05:16:30] <icinga-miraheze>	 PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[05:34:57] <Reception123>	 !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php initSiteStats.php --wiki marionetworkwiki --update
[05:35:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:03:56] <icinga-miraheze>	 RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:05:03] <icinga-miraheze>	 RECOVERY - mw5 APT on mw5 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:19:09] <icinga-miraheze>	 RECOVERY - mw6 APT on mw6 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:24:04] <icinga-miraheze>	 RECOVERY - mw7 APT on mw7 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:45:51] <icinga-miraheze>	 RECOVERY - phab1 APT on phab1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[06:46:38] <icinga-miraheze>	 RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[07:06:16] <icinga-miraheze>	 PROBLEM - usagihimepedia.ddns.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for usagihimepedia.ddns.net could not be found
[07:50:01] <icinga-miraheze>	 PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[08:10:00] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[08:10:55] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[08:15:59] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[08:16:54] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:37:43] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.48, 20.52, 16.59
[08:41:43] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.70, 18.03, 16.66
[08:50:15] <icinga-miraheze>	 PROBLEM - housing.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'housing.wiki' expires in 15 day(s) (Fri 11 Sep 2020 08:46:20 GMT +0000).
[08:57:24] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUTYQ
[08:57:26] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot 03a9ccda7 - Bot: Update SSL cert for housing.wiki
[09:03:34] <icinga-miraheze>	 RECOVERY - housing.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'housing.wiki' will expire on Tue 24 Nov 2020 07:57:17 GMT +0000.
[09:24:48] <icinga-miraheze>	 PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pubwiki.lab.co.pl' expires in 15 day(s) (Fri 11 Sep 2020 09:20:38 GMT +0000).
[09:28:30] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUT37
[09:28:32] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot 03b961450 - Bot: Update SSL cert for pubwiki.lab.co.pl
[09:31:42] <icinga-miraheze>	 RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Tue 24 Nov 2020 08:28:22 GMT +0000.
[09:37:05] <icinga-miraheze>	 PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'index.pdcommunity.ir' expires in 15 day(s) (Fri 11 Sep 2020 09:28:57 GMT +0000).
[09:38:11] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUTsG
[09:38:13] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot 03f450968 - Bot: Update SSL cert for index.pdcommunity.ir
[09:44:01] <icinga-miraheze>	 RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Tue 24 Nov 2020 08:38:04 GMT +0000.
[10:42:00] <icinga-miraheze>	 RECOVERY - gluster2 Disk Space on gluster2 is OK: DISK OK - free space: / 38139 MB (11% inode=77%);
[10:42:16] <icinga-miraheze>	 RECOVERY - gluster1 Disk Space on gluster1 is OK: DISK OK - free space: / 77818 MB (11% inode=84%);
[10:51:42] <icinga-miraheze>	 PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates).
[11:03:01] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.31, 22.79, 18.94
[11:06:16] <icinga-miraheze>	 PROBLEM - gluster1 Disk Space on gluster1 is WARNING: DISK WARNING - free space: / 74778 MB (10% inode=84%);
[11:06:32] <MirahezeBot>	 Hello SistemaActivado! If you have any questions, feel free to ask and someone should answer soon.
[11:06:41] <SistemaActivado>	 I need Kubernetes.
[11:06:46] <SistemaActivado>	 I need Miraheze Toolforge.
[11:10:35] <icinga-miraheze>	 PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[11:16:34] <icinga-miraheze>	 RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[11:22:16] <icinga-miraheze>	 RECOVERY - gluster1 Disk Space on gluster1 is OK: DISK OK - free space: / 77806 MB (11% inode=84%);
[11:27:48] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 12.02, 19.08, 22.79
[11:33:43] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.66, 20.51, 21.99
[11:35:11] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUTCO
[11:35:12] <Not>	 [02miraheze/services] 07MirahezeSSLBot 0316de2be - BOT: Updating services config for wikis
[11:35:42] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.04, 21.92, 22.40
[11:35:43] <Not>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JUTCn
[11:35:45] <Not>	 [02miraheze/ssl] 07Reception123 03e503b8e - add wiki.usagihime.ml cert
[11:36:17] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[11:40:55] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[11:44:25] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.27, 21.41, 21.28
[11:46:19] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.88, 21.93, 21.49
[11:46:39] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:49:24] <Not>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JUTWU
[11:49:25] <Not>	 [02miraheze/ssl] 07Reception123 038c116b5 - rm usagihimepedia.ddns.net cert Custom domain replaced, not pointing
[11:50:08] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.10, 18.62, 20.20
[11:50:18] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUTWL
[11:50:20] <Not>	 [02miraheze/services] 07MirahezeSSLBot 03ec3741a - BOT: Updating services config for wikis
[11:56:40] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:14:35] <Reception123>	 !log ran GDPR script
[12:14:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:54] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.52, 19.20, 16.57
[12:25:52] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.13, 18.74, 16.71
[12:29:49] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[12:35:49] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[12:45:34] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.71, 18.53, 17.05
[12:47:33] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.31, 18.36, 17.18
[12:58:04] <SigmaBotMH>	 Alert to Miraheze Staff: It looks like the MirahezeRC bot has stopped! Recent Changes are no longer available from IRC.
[13:24:31] <MH-Discord>	 <RhinosF1> @Site Reliability Engineers ^
[13:36:01] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.10, 21.69, 18.34
[13:37:59] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.58, 18.74, 17.71
[13:45:00] <MH-Discord>	 <Reception123> oh fun
[13:45:14] <Reception123>	 looks back
[13:50:56] <MH-Discord>	 <RhinosF1> Thanks!
[14:04:04] <Reception123>	 !log reception@jobrunner1:/srv/mediawiki/w/extensions/MatomoAnalytics/maintenance$ sudo -u www-data php modifyMatomo.php --wiki metawiki
[14:04:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:14:28] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.88, 23.22, 21.04
[14:20:21] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUTXT
[14:20:22] <Not>	 [02miraheze/services] 07MirahezeSSLBot 0343d9b3f - BOT: Updating services config for wikis
[14:20:24] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.96, 20.01, 20.31
[14:34:06] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.38, 19.50, 19.03
[14:38:02] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.63, 22.56, 20.50
[14:43:57] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.46, 19.01, 19.75
[14:55:12] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUThU
[14:55:14] <Not>	 [02miraheze/services] 07MirahezeSSLBot 03f249873 - BOT: Updating services config for wikis
[14:59:20] <icinga-miraheze>	 PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 136s
[15:00:16] <icinga-miraheze>	 PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (3 critical updates).
[15:09:50] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 22.68, 26.65, 21.97
[15:11:40] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --wiki aoc2wiki /home/reception/Wikimedia_-_Infobox_templates_relating_to_country%2C_game%2C_government%2C_and_settlement.xml
[15:11:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:31:25] <icinga-miraheze>	 PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates).
[15:49:38] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.05, 17.43, 22.95
[15:51:39] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 39.93, 24.61, 24.80
[16:19:14] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:27:13] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:30:34] <icinga-miraheze>	 PROBLEM - jobrunner2 Current Load on jobrunner2 is CRITICAL: CRITICAL - load average: 9.78, 5.98, 3.70
[16:36:33] <icinga-miraheze>	 PROBLEM - jobrunner2 Current Load on jobrunner2 is WARNING: WARNING - load average: 3.12, 5.82, 4.56
[16:38:51] <icinga-miraheze>	 RECOVERY - jobrunner2 Current Load on jobrunner2 is OK: OK - load average: 3.10, 4.74, 4.32
[16:39:03] <icinga-miraheze>	 PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 59%
[16:39:54] <icinga-miraheze>	 PROBLEM - rupowerrangersfanon.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rupowerrangersfanon.tk could not be found
[16:40:06] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 12.44, 8.60, 5.21
[16:40:16] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[16:40:59] <icinga-miraheze>	 RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 5%
[16:41:44] <icinga-miraheze>	 RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 2s
[16:42:00] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.08, 6.82, 4.96
[16:43:56] <icinga-miraheze>	 RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.96, 5.40, 4.68
[16:46:03] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[16:46:45] <icinga-miraheze>	 RECOVERY - rupowerrangersfanon.tk - reverse DNS on sslhost is OK: rDNS OK - rupowerrangersfanon.tk reverse DNS resolves to cp6.miraheze.org
[16:55:51] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.94, 17.51, 22.90
[17:03:43] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.22, 16.86, 20.32
[17:07:38] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.26, 20.09, 21.03
[17:11:34] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.93, 18.74, 20.32
[17:13:08] <icinga-miraheze>	 PROBLEM - cp9 APT on cp9 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates).
[17:51:50] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 79%
[17:53:49] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 7%
[17:59:49] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 44%
[18:01:49] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 9%
[18:23:38] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.89, 19.25, 16.40
[18:29:34] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.72, 18.72, 17.34
[19:49:47] <MirahezeBot>	 Hello Guest47! If you have any questions, feel free to ask and someone should answer soon.
[20:16:07] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.40, 23.19, 18.89
[20:18:06] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.99, 19.33, 18.01
[20:29:50] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[20:43:41] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.99, 21.59, 20.10
[20:45:41] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.02, 19.64, 19.56
[20:45:48] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[21:07:17] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.79, 21.43, 19.64
[21:09:16] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.95, 18.45, 18.78
[22:35:39] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.21, 18.12, 14.84
[22:37:38] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.66, 15.32, 14.19