[03:02:04] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%);
[06:23:56] <Not-def6>	 [02mw-config] 07Reception123 closed pull request 03#2748: add ipv6 address for cookiewarn exemption - 13https://git.io/fjx7v
[06:24:16] <Not-def6>	 [02mw-config] 07Reception123 commented on pull request 03#2726: remove cologne blue - 13https://git.io/JeB0z
[06:26:25] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3279 MB (13% inode=94%);
[08:21:29] <Not-def6>	 [02mw-config] 07RhinosF1 commented on pull request 03#2726: remove cologne blue - 13https://git.io/JeBEw
[09:13:41] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki degreesoflewditywiki /home/reception/degreesoflewdity_pages_full.xml
[09:13:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[11:45:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBzK
[11:45:12] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03d2b4984 - BOT: Updating services config for wikis
[12:13:25] <Not-def6>	 [02mw-config] 07paladox deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3
[12:13:27] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03Reception123-patch-2
[12:27:11] <Not-def6>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JeBgv
[12:27:13] <Not-def6>	 [02miraheze/ssl] 07Reception123 0321c614c - add mobiledomain wiki.m.isina.ir
[12:39:59] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.96, 7.13, 5.80
[12:40:13] <RhinosF1>	 Welcome apap
[12:40:20] <apap>	 Congrats on fixing the outages! :)
[12:40:22] <ZppixBot>	 apap: 2019-10-17 - 19:35:51UTC <RhinosF1> tell apap if you need a new protection level file a Phabricator task with more information
[12:40:45] <RhinosF1>	 apap: look at paladox
[12:41:06] <apap>	 im staring at paladox
[12:41:23] <apap>	 great job :))
[12:42:05] <paladox>	 :)
[12:42:08] <RhinosF1>	 :)
[12:42:30] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.92, 6.43, 5.74
[12:48:16] <Not-def6>	 [02ssl] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vxP9L
[12:48:17] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeBgl
[12:48:19] <Not-def6>	 [02miraheze/ssl] 07paladox 03746cf53 - Fix renewing eerstelijnszones.be
[12:48:21] <Not-def6>	 [02ssl] 07paladox opened pull request 03#228: Fix renewing eerstelijnszones.be - 13https://git.io/JeBg8
[12:48:38] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeBg4
[12:48:39] <Not-def6>	 [02miraheze/ssl] 07paladox 0336cf357 - Update certs.yaml
[12:48:41] <Not-def6>	 [02ssl] 07paladox synchronize pull request 03#228: Fix renewing eerstelijnszones.be - 13https://git.io/JeBg8
[12:48:47] <Not-def6>	 [02ssl] 07paladox closed pull request 03#228: Fix renewing eerstelijnszones.be - 13https://git.io/JeBg8
[12:48:48] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JeBgB
[12:48:50] <Not-def6>	 [02miraheze/ssl] 07paladox 03dd039b1 - Fix renewing eerstelijnszones.be (#228) * Fix renewing eerstelijnszones.be * Update certs.yaml
[12:49:30] <Not-def6>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-2
[12:49:32] <Not-def6>	 [02ssl] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vxP9L
[12:50:41] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBgR
[12:50:42] <Not-def6>	 [02miraheze/ssl] 07paladox 0373d61cf - Renew eerstelijnszones.be ssl cert
[12:52:06] <Not-def6>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBg0
[12:52:08] <Not-def6>	 [02miraheze/dns] 07paladox 030d2f5b3 - Add acme challenge to orain dns
[12:52:34] <icinga-miraheze>	 RECOVERY - eerstelijnszones.be - LetsEncrypt on sslhost is OK: OK - Certificate 'eerstelijnszones.be' will expire on Fri 17 Jan 2020 11:50:22 AM GMT +0000.
[12:54:20] <icinga-miraheze>	 RECOVERY - www.eerstelijnszones.be - LetsEncrypt on sslhost is OK: OK - Certificate 'eerstelijnszones.be' will expire on Fri 17 Jan 2020 11:50:22 AM GMT +0000.
[12:54:58] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBgE
[12:54:59] <Not-def6>	 [02miraheze/ssl] 07paladox 0313bfeb3 - Renew orain cert
[12:55:36] <Not-def6>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBgu
[12:55:38] <Not-def6>	 [02miraheze/dns] 07paladox 0371f8ae7 - Update orain.org
[12:56:26] <Not-def6>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBgz
[12:56:27] <Not-def6>	 [02miraheze/dns] 07paladox 0309c0b62 - Add acme challenge to miraheze.wiki
[12:57:48] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBgg
[12:57:49] <Not-def6>	 [02miraheze/ssl] 07paladox 03aa9618b - Renew miraheze.wiki
[12:58:56] <Not-def6>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBg2
[12:58:58] <Not-def6>	 [02miraheze/dns] 07paladox 030218fc7 - Add acme challenge for miraheze.com
[13:00:11] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBga
[13:00:12] <Not-def6>	 [02miraheze/ssl] 07paladox 0342c0beb - Renew miraheze.com
[13:01:50] <Not-def6>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBgo
[13:01:51] <Not-def6>	 [02miraheze/dns] 07paladox 03d7c0436 - Add acme challenge for m.miraheze.org
[13:02:50] <icinga-miraheze>	 RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 17 Jan 2020 11:54:38 AM GMT +0000.
[13:03:44] <icinga-miraheze>	 RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 17 Jan 2020 11:57:22 AM GMT +0000.
[13:03:44] <icinga-miraheze>	 RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Fri 17 Jan 2020 11:59:53 AM GMT +0000.
[13:03:47] <icinga-miraheze>	 RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 17 Jan 2020 11:54:38 AM GMT +0000.
[13:04:04] <icinga-miraheze>	 RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 17 Jan 2020 11:54:38 AM GMT +0000.
[13:04:34] <Not-def6>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBgP
[13:04:36] <Not-def6>	 [02miraheze/ssl] 07paladox 03b3f6e68 - Renew m.miraheze.org
[13:04:52] <icinga-miraheze>	 RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 17 Jan 2020 11:54:38 AM GMT +0000.
[13:14:51] <icinga-miraheze>	 RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Fri 17 Jan 2020 12:04:11 PM GMT +0000.
[13:24:47] <Not-def6>	 [02mw-config] 07PlavorSeol commented on pull request 03#2726: remove cologne blue - 13https://git.io/JeBgF
[13:36:20] <Not-def6>	 [02mw-config] 07RhinosF1 commented on pull request 03#2726: remove cologne blue - 13https://git.io/JeBgj
[13:36:22] <Not-def6>	 [02mw-config] 07RhinosF1 closed pull request 03#2726: remove cologne blue - 13https://git.io/fj5cB
[13:36:32] <RhinosF1>	 Reception123: ^
[13:38:10] <Reception123>	 ok
[13:46:33] <paladox>	 !log upgrade icinga2 to 2.11.1 on misc1
[13:47:03] <paladox>	 !log upgrade icingaweb2 to 2.7.3 on misc1
[13:47:45] <apap>	 is mira down because of upgrades?
[13:47:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:48:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:48:49] <paladox>	 !log upgrade grafana to 6.4.3
[13:48:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:49:08] <paladox>	 apap nope, seems that we had a brief 503, but things should be more stable then they were in the past.
[13:49:30] <apap>	 ah alright
[13:51:23] <Not-def6>	 [02mw-config] 07PlavorSeol opened pull request 03#2778: Remove unnecessary configurations of $wgRightsIcon - 13https://git.io/JeB24
[13:54:57] <RhinosF1>	 Reception123: ^ looks good to me but laptop is upstairs so are you okay in case anything goes wrong somehow to respond
[13:55:48] <Not-def6>	 [02mw-config] 07paladox closed pull request 03#2778: Remove unnecessary configurations of $wgRightsIcon - 13https://git.io/JeB24
[13:55:49] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB2z
[13:55:51] <Not-def6>	 [02miraheze/mw-config] 07PlavorSeol 03cb6396b - $wgRightsIcon (#2778)
[13:55:54] <Reception123>	 RhinosF1: yup, just a removal
[13:58:32] <RhinosF1>	 Reception123: it is
[13:59:15] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB2V
[13:59:16] <Not-def6>	 [02miraheze/puppet] 07paladox 03daef45d - Update mediawiki.pp
[13:59:49] <Reception123>	 paladox: 503
[14:00:01] <paladox>	 again?
[14:00:31] <Reception123>	 just now yes on Meta
[14:01:01] <Reception123>	 now it's back
[14:01:54] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB2r
[14:01:56] <Not-def6>	 [02miraheze/puppet] 07paladox 03453c3e6 - Lower NR_OF_HDD_WORKERS_PER_NETWORK_WORKER to 3
[14:04:46] <paladox>	 !log restart lizardfs-chunkserver on lizardfs[45]
[14:06:15] <RhinosF1>	 paladox: were still down
[14:06:24] <paladox>	 yes i am aware
[14:06:27] <paladox>	 see Reception123 
[14:06:43] <paladox>	 !log reboot misc4
[14:06:48] <RhinosF1>	 paladox: reception said it was back but it isn't
[14:07:06] <paladox>	 yes, but it's going on and off and i've been monitoring.
[14:08:14] <RhinosF1>	 Ok
[14:10:07] <paladox>	 ok, i dunno why things are failing.
[14:10:16] <paladox>	 i see:
[14:10:17] <paladox>	 script_filename = /srv/mediawiki/w/index.php
[14:10:17] <paladox>	 [0x00007f86c001d210] fastcgi_finish_request() /srv/mediawiki/w/includes/MediaWiki.php:745
[14:10:50] <RhinosF1>	 paladox: in which log?
[14:10:56] <paladox>	 the php slowlog
[14:11:16] <RhinosF1>	 paladox: on?
[14:11:25] <RhinosF1>	 Full file path if I have access
[14:11:35] <paladox>	 seems it is a nginx issue
[14:11:46] <paladox>	 php7.2-fpm-www-slowlog.log in /var/log
[14:12:13] <paladox>	 !log restarted nginx on mw[123]
[14:12:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:13:06] <RhinosF1>	 paladox: file can not be opened
[14:13:17] <paladox>	 Yup, it's owned by root
[14:13:23] <RhinosF1>	 paladox: oh
[14:13:39] <paladox>	 anyways things are fixed now!
[14:13:48] <paladox>	 php-fpm is not the cause
[14:13:50] <paladox>	 nginx waas
[14:13:52] <paladox>	 *was
[14:14:01] <RhinosF1>	 Ah
[14:19:13] <Not-def6>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[14:19:14] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2H
[14:19:16] <Not-def6>	 [02miraheze/puppet] 07paladox 03ce2a20d - mediawiki: Fix timeout related config
[14:19:17] <Not-def6>	 [02puppet] 07paladox opened pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:21:16] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB25
[14:21:18] <Not-def6>	 [02miraheze/puppet] 07paladox 031253df7 - Update fpm.pp
[14:21:19] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:21:46] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2b
[14:21:47] <Not-def6>	 [02miraheze/puppet] 07paladox 039d3a75c - Update mediawiki-includes.conf.erb
[14:21:49] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:22:26] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2N
[14:22:27] <Not-def6>	 [02miraheze/puppet] 07paladox 03f6321b4 - Update php.pp
[14:22:29] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:22:40] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2A
[14:22:42] <Not-def6>	 [02miraheze/puppet] 07paladox 0302b6459 - Update fpm.pp
[14:22:43] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:22:59] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2x
[14:23:01] <Not-def6>	 [02miraheze/puppet] 07paladox 03cec9422 - Update php.pp
[14:23:02] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:23:17] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2p
[14:23:18] <Not-def6>	 [02miraheze/puppet] 07paladox 033421377 - Update init.pp
[14:23:20] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:23:37] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2h
[14:23:39] <Not-def6>	 [02miraheze/puppet] 07paladox 03f3c6749 - Update php_fpm.pp
[14:23:40] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:24:33] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:24:35] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeB2j
[14:24:36] <Not-def6>	 [02miraheze/puppet] 07paladox 03ee168a5 - Update mediawiki.conf
[14:24:46] <Not-def6>	 [02puppet] 07paladox closed pull request 03#1108: mediawiki: Fix timeout related config - 13https://git.io/JeB2Q
[14:24:47] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JeBae
[14:24:49] <Not-def6>	 [02miraheze/puppet] 07paladox 03f6507e4 - mediawiki: Fix timeout related config (#1108) * mediawiki: Fix timeout related config * Update fpm.pp * Update mediawiki-includes.conf.erb * Update php.pp * Update fpm.pp * Update php.pp * Update init.pp * Update php_fpm.pp * Update mediawiki.conf
[14:24:50] <Not-def6>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[14:24:52] <Not-def6>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[14:30:20] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JeBaf
[14:30:21] <Not-def6>	 [02miraheze/puppet] 07paladox 03299d015 - Revert "mediawiki: Set nginx::worker_processes to auto (#1107)" This reverts commit 3f1e9e316ba5c4aa10ea462cf3e667f0f3e0c567.
[14:32:40] <paladox>	 !log upgrade php on misc1
[14:32:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:35:32] <paladox>	 !log upgrade openssl on misc1
[14:35:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:36:23] <paladox>	 !log upgrade puppet-agent puppetdb puppetdb-termini puppetserver on puppet1
[14:36:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:38:09] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.2-apcu],Package[php7.2-redis]
[14:44:46] <icinga-miraheze>	 PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 15 seconds ago with 6 failures. Failed resources (up to 3 shown)
[14:49:01] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 4 minutes ago with 9 failures. Failed resources (up to 3 shown)
[14:49:33] <paladox>	 !log upgrade puppet-agent on db[45] misc[13]
[14:49:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:50:07] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:50:17] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 5 minutes ago with 12 failures. Failed resources (up to 3 shown)
[14:50:52] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 372 failures. Last run 5 minutes ago with 372 failures. Failed resources (up to 3 shown)
[14:51:14] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBan
[14:51:15] <Not-def6>	 [02miraheze/puppet] 07paladox 03823bdcc - Update roundcubemail to 1.4-rc2
[14:51:18] <icinga-miraheze>	 RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:51:44] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 360 failures. Last run 5 minutes ago with 360 failures. Failed resources (up to 3 shown)
[14:51:50] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 6 minutes ago with 10 failures. Failed resources (up to 3 shown)
[14:51:51] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 361 failures. Last run 6 minutes ago with 361 failures. Failed resources (up to 3 shown)
[14:51:52] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 7 minutes ago with 8 failures. Failed resources (up to 3 shown)
[14:52:14] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[14:52:52] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 375 failures. Last run 7 minutes ago with 375 failures. Failed resources (up to 3 shown)
[14:53:24] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 369 failures. Last run 7 minutes ago with 369 failures. Failed resources (up to 3 shown)
[14:53:31] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[14:53:34] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 332 failures. Last run 7 minutes ago with 332 failures. Failed resources (up to 3 shown): File[publictestwiki.com],File[publictestwiki.com_private],File[wiki.valentinaproject.org],File[wiki.valentinaproject.org_private]
[14:54:50] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[14:54:51] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:54:51] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:55:10] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBaW
[14:55:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03b69b9de - BOT: Updating services config for wikis
[14:55:12] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:55:43] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:56:03] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:56:07] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:56:15] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[14:56:37] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:57:16] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[15:06:08] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBa2
[15:06:09] <Not-def6>	 [02miraheze/puppet] 07paladox 03d72c66d - Revert "Update roundcubemail to 1.4-rc2" This reverts commit 823bdcc3c5f30d3257a4815fe1688b8e4dbc6ab8.
[15:30:11] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.12, 7.77, 6.23
[15:30:22] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.39, 7.82, 5.43
[15:32:52] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.58, 7.33, 6.30
[15:37:47] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.51, 6.54, 6.30
[15:37:59] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.17, 7.72, 6.40
[15:46:08] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.25, 8.57, 7.21
[16:01:42] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.51, 7.69, 7.75
[16:04:19] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.86, 9.14, 8.30
[16:06:49] <paladox>	 !log upgrade puppet-agent on mw2
[16:06:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:09:13] <paladox>	 !log depool mw2
[16:09:15] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.13, 7.35, 7.80
[16:09:16] <paladox>	 !log reboot mw2
[16:09:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:09:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:12:26] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 0.90, 0.23, 0.07
[16:22:07] <paladox>	 !log repool mw2
[16:22:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:28:53] <paladox>	 !log upgrade puppet-agent on mw3
[16:28:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:35:10] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBwx
[17:35:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 034a48514 - BOT: Updating services config for wikis
[19:56:34] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.40, 7.17, 5.96
[20:00:14] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.22, 6.97, 5.25
[20:08:07] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.52, 6.73, 5.92
[20:12:22] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.67, 6.69, 6.57
[20:16:27] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.09, 8.22, 6.78
[20:26:41] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 4.83, 6.94, 6.99
[20:31:43] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.95, 7.10, 6.96
[20:33:48] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.21, 6.00, 4.47
[20:36:20] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.21, 5.89, 4.63
[20:36:52] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.61, 7.08, 7.01
[20:39:22] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.75, 6.49, 6.80
[20:41:40] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.38, 8.15, 6.02
[20:44:07] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.19, 7.97, 6.27
[20:46:45] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.62, 8.39, 6.67
[20:51:24] <hispano76_>	 RhinosF1 hi
[20:54:16] <ZppixBot>	 Hello Mirnohom! If you have any questions, feel free to ask and someone should answer soon.
[20:54:25] <Mirnohom>	 hi
[20:54:31] <Mirnohom>	 anyone there
[20:55:01] <Mirnohom>	 irc is just oldschool discord lol
[20:59:41] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.34, 7.81, 7.58
[21:00:45] <Voidwalker>	 hi
[21:02:07] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.95, 8.93, 8.04
[21:02:23] <Mirnohom>	 yo void
[21:02:47] <Voidwalker>	 what's up?
[21:03:02] <Mirnohom>	 oh nothing, just watching some youtube videos
[21:06:57] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.15, 7.78, 7.84
[21:07:48] <RhinosF1>	 Hi hispano76_
[21:08:00] <Mirnohom>	 what are these errors?
[21:10:39] <RhinosF1>	 Mirnohom: nothing to worry about, just indicates to us that the server is doing at lot at once
[21:10:45] <Mirnohom>	 oh ok
[21:16:35] <RhinosF1>	 Mirnohom: we'll keep an eye on it
[21:19:43] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.51, 4.42, 6.24