[00:08:01] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/fNtoY
[00:08:02] <Not-683e>	 [02miraheze/puppet] 07paladox 033888538 - Revert "Set fallocate_reserve to 11%" This reverts commit 9b9044f9c1ef3bd67bddbeb7122ccbc16b7b67e2.
[00:08:04] <Not-683e>	 [02miraheze/puppet] 07paladox 0393c4e4a - Revert "Set fallocate_reserve to 11%" This reverts commit edcd480843c55d6bddcff714447756c69e32a7a5.
[00:08:05] <Not-683e>	 [02miraheze/puppet] 07paladox 030a99b58 - Revert "Set fallocate_reserve to 11%" This reverts commit cec27c3b17be5f933c8336de4dd7a627da10dcb7.
[00:11:04] <icinga-miraheze>	 PROBLEM - Puppet on swift2 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 13 seconds ago with 20 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-9102],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9100]  
[00:11:24] <icinga-miraheze>	 PROBLEM - Puppet on swift1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 2 minutes ago with 20 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-9102],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9100]  
[00:11:39] <paladox>	 !log reboot swift* OOM
[00:11:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:12:42] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 1.66, 0.63, 0.22  
[00:13:22] <icinga-miraheze>	 RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures  
[00:15:04] <icinga-miraheze>	 RECOVERY - Puppet on swift2 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures  
[00:26:23] <MacFan4000>	 .mangle
[00:26:34] <ZppixBot>	 MacFan4000: Mirage delays "Discover - 2 fast dummies are fine, are in US dollars, lasted 58 minutes, finally collapsed, 0"
[00:42:32] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.72, 1.16, 0.71  
[00:44:32] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.13, 1.00, 0.69  
[02:16:18] <NeoBogan>	 what are you still doing up? isn't it past your bedtime, paladox?
[02:16:39] <paladox>	 NeoBogan: I am over 20 so nope
[02:16:46] <paladox>	 03:16am
[02:16:57] <NeoBogan>	 isn't it past your curfew?
[02:17:01] <paladox>	 I’ve by mistake stayed up till 5am
[02:17:13] <paladox>	 NeoBogan: I doint have a curfew!
[03:23:01] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNtMe
[03:23:02] <Not-683e>	 [02miraheze/mw-config] 07paladox 0356993a2 - Update Sitenotice.php
[05:32:32] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2706  
[07:30:34] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[09:12:34] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw1  
[09:14:34] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy  
[12:38:33] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw1  
[12:40:33] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy  
[12:40:35] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy  
[12:43:07] <Wiki-1776>	 hi
[16:28:33] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2786  
[16:47:07] <MH-Discord>	 <Shenobi> Not sure where to go for this but images are down on our wiki, I saw the warning but just wondering if there is an estimate for when it should be fixed.
[16:50:09] <MH-Discord>	 <Shenobi> This is happening on the Beta Purple wiki for the record
[16:51:53] <Voidwalker>	 paladox ^
[16:52:01] <paladox>	 hmm
[16:52:06] <paladox>	 Shenobi which wiki?
[16:56:29] <Voidwalker>	 I'm guessing https://betapurple.miraheze.org/wiki/Beta_Purple_Wiki
[16:56:29] <ZppixBot>	 Title: [ Permission error - Beta Purple Wiki ] - betapurple.miraheze.org
[16:56:32] <paladox>	 @Shenobi fixed it now
[16:56:37] <paladox>	 Voidwalker thanks! Fixed
[16:56:41] <MH-Discord>	 <Shenobi> Heeeeeeey thanks
[16:56:56] <MH-Discord>	 <Shenobi> 👍
[16:57:02] <MH-Discord>	 <Shenobi> That’s what I call customer support
[16:57:07] <paladox>	 your welcome :)
[17:02:07] <Voidwalker>	 https://meta.miraheze.org/wiki/Community_noticeboard#Problems_with_images_and_thumbnails also needs looking at
[17:02:08] <ZppixBot>	 Title: [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org
[17:04:51] <paladox>	 Voidwalker fixed!
[17:06:41] <Voidwalker>	 neat
[18:07:22] <MacFan4000>	 !log macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki vsfanwiki /home/macfan/dump.xml in a screen
[18:07:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:20:24] <icinga-miraheze>	 PROBLEM - oecumene.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate oecumene.org expires in 15 day(s) (Sun 29 Jul 2018 06:17:28 PM GMT +0000).  
[18:20:34] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNqQT
[18:20:36] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 037f1e0ee - Bot: Update SSL cert for oecumene.org
[18:22:24] <icinga-miraheze>	 RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate oecumene.org will expire on Thu 11 Oct 2018 05:20:29 PM GMT +0000.  
[18:26:24] <icinga-miraheze>	 PROBLEM - www.wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate www.wikiescola.com.br expires in 15 day(s) (Sun 29 Jul 2018 06:23:21 PM GMT +0000).  
[18:26:35] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNqQi
[18:26:37] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 03e945de0 - Bot: Update SSL cert for www.wikiescola.com.br
[18:32:25] <icinga-miraheze>	 RECOVERY - www.wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate www.wikiescola.com.br will expire on Thu 11 Oct 2018 05:26:30 PM GMT +0000.  
[18:32:45] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[19:03:25] <icinga-miraheze>	 PROBLEM - Puppet on swift1 is WARNING: WARNING: Puppet last ran 1 hour ago  
[19:03:35] <paladox>	 huh
[19:07:25] <icinga-miraheze>	 PROBLEM - Puppet on swift1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 3 seconds ago with 20 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-9102],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9100]  
[19:11:25] <icinga-miraheze>	 RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures  
[19:15:25] <paladox>	 !log rebooted swift1 to reduce ram.
[19:15:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:53:36] <icinga-miraheze>	 PROBLEM - Disk Space on cp5 is WARNING: DISK WARNING - free space: / 4815 MB (20% inode=95%):  
[19:55:38] <JohnLewis>	 Hush you 
[19:56:26] <icinga-miraheze>	 PROBLEM - wiki.dobots.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.dobots.nl expires in 15 day(s) (Sun 29 Jul 2018 07:52:38 PM GMT +0000).  
[19:56:36] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNqpL
[19:56:37] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 031ee3e40 - Bot: Update SSL cert for wiki.dobots.nl
[20:02:26] <icinga-miraheze>	 RECOVERY - wiki.dobots.nl - LetsEncrypt on sslhost is OK: OK - Certificate wiki.dobots.nl will expire on Thu 11 Oct 2018 06:56:30 PM GMT +0000.  
[20:11:37] <icinga-miraheze>	 RECOVERY - Disk Space on cp5 is OK: DISK OK - free space: / 5096 MB (21% inode=95%):  
[20:18:46] <paladox>	 JohnLewis huh?
[20:32:13] <MacFan4000>	 FYI I don’t and probably won’t have access until Sunday. I
[20:33:03] <MacFan4000>	 if someone wants to keep an eye on my Import on mw1, that would be great :)
[21:10:46] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down.  mw1  
[21:10:48] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw1  
[21:11:47] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw1  
[21:12:47] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy  
[21:17:47] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy  
[21:24:45] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down.  mw1  
[21:25:07] <paladox>	 hmm why is mw1 going down so much
[21:42:35] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmkD
[21:42:37] <Not-683e>	 [02miraheze/puppet] 07paladox 039b67b2a - Update mediawiki.conf
[21:43:18] <paladox>	 no recovery for cp4
[21:50:17] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fNmIT
[21:50:18] <Not-683e>	 [02miraheze/puppet] 07paladox 03ef4fce1 - optimising php-fpm
[21:50:20] <Not-683e>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[21:50:21] <Not-683e>	 [02puppet] 07paladox opened pull request 03#775: optimising php-fpm - 13https://git.io/fNmIk
[21:51:08] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#775: optimising php-fpm - 13https://git.io/fNmIk
[21:51:09] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fNmII
[21:51:11] <Not-683e>	 [02miraheze/puppet] 07paladox 03b53c86b - Update www-7.2.conf
[22:00:00] <PuppyKun>	 paladox: because import on it ;)?
[22:00:01] <PuppyKun>	 no idea
[22:00:14] <paladox>	 PuppyKun huh?
[22:00:16] <paladox>	 import?
[22:00:45] <PuppyKun>	 mac*fan literally just said to keep an eye on the import on mw1
[22:00:52] <PuppyKun>	 im assuming its on a job or somethinh
[22:00:52] <paladox>	 oh
[22:01:01] <paladox>	 PuppyKun i doin't think it's the import
[22:01:08] <paladox>	 it should not be using php-fpm
[22:01:11] <paladox>	 on the command line
[22:03:35] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#775: optimising php-fpm - 13https://git.io/fNmIk
[22:03:37] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fNmLO
[22:03:38] <Not-683e>	 [02miraheze/puppet] 07paladox 038bbd83b - Update www-7.2.conf
[22:04:46] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmLC
[22:04:48] <Not-683e>	 [02miraheze/puppet] 07paladox 03a6bc589 - Set request_terminate_timeout to 300s We already set the max execution time in php.ini as 300 we have to do it in two places. php-fpm and php.ini.
[22:07:09] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmLg
[22:07:10] <Not-683e>	 [02miraheze/puppet] 07paladox 03149753b - mediawiki: Set fastcgi_read_timeout to 300 in nginx conf
[22:08:03] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmLw
[22:08:04] <Not-683e>	 [02miraheze/puppet] 07paladox 038e0a649 - Update mediawiki-includes.conf.erb
[23:11:27] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.140 second response time  
[23:13:27] <icinga-miraheze>	 RECOVERY - MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 30101 bytes in 0.288 second response time  
[23:22:45] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmOY
[23:22:47] <Not-683e>	 [02miraheze/puppet] 07paladox 03004df22 - Update container-server.conf.erb
[23:22:56] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNmOO
[23:22:57] <Not-683e>	 [02miraheze/puppet] 07paladox 036453856 - Update object-server.conf.erb
[23:55:25] <icinga-miraheze>	 RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures