[00:03:14] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:46] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:58] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:00] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:04] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:12] <icinga-miraheze>	 PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:38] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:11:12] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures  
[00:11:58] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:00] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures  
[00:12:04] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:12] <icinga-miraheze>	 RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:34] <mutante>	 lots of lizards. i like it
[00:12:38] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:46] <mutante>	 lizards are cool
[00:13:39] <paladox>	 lol
[00:13:46] <paladox>	 mutante that's our file storage :P
[00:13:46] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures  
[00:14:29] <mutante>	 yea :) i learned that recently. you said it works better than NFS :)
[00:14:46] <paladox>	 yep :)
[00:15:20] <mutante>	 offers random page of the day: and then disappears:   https://en.wikipedia.org/wiki/Lizard_King  https://en.wikipedia.org/wiki/Jim_Morrison
[00:15:21] <ZppixBot>	 [WIKIPEDIA] Jim Morrison | "James Douglas Morrison (December 8, 1943 – July 3, 1971), better known as Jim Morrison, was an American singer, songwriter and poet, best remembered as the lead vocalist of the rock band The Doors. Due to his poetic lyrics, distinctive voice, wild personality, performances, and the dramatic circumstances..."
[00:15:29] <mutante>	 ^ The Lizard King :)
[00:15:42] <paladox>	 lol
[04:24:13] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 368  
[04:26:13] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[08:20:15] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fx8bg
[08:20:17] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 03832e59c - BOT: Updating services config for wikis
[14:13:19] <icinga-miraheze>	 PROBLEM - wiki.staraves-no.cz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.staraves-no.cz' expires in 15 day(s) (Sun 28 Oct 2018 02:09:45 PM GMT +0000).  
[14:13:29] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fx4zM
[14:13:31] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot 03b46f1ce - Bot: Update SSL cert for wiki.staraves-no.cz
[14:16:19] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5676  
[14:21:17] <icinga-miraheze>	 RECOVERY - wiki.staraves-no.cz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.staraves-no.cz' will expire on Thu 10 Jan 2019 01:13:24 PM GMT +0000.  
[15:04:10] <icinga-miraheze>	 PROBLEM - infectowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'infectowiki.com' expires in 15 day(s) (Sun 28 Oct 2018 03:01:34 PM GMT +0000).  
[15:04:22] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fx4r7
[15:04:23] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot 0351ac96f - Bot: Update SSL cert for infectowiki.com
[15:12:10] <icinga-miraheze>	 RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Thu 10 Jan 2019 02:04:16 PM GMT +0000.  
[16:06:00] <MacFan4000>	 Uh, I’m getting 503s
[16:06:20] <MacFan4000>	 And back
[18:14:54] <paladox>	 !log upgrading icinga2 to 2.10 and grafana
[18:14:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:30:47] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: <Terminated by signal 15 (Terminated).>  
[18:32:00] <Not-2d3>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[18:32:02] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBe8
[18:32:03] <Not-2d3>	 [02miraheze/puppet] 07paladox 03ce56c05 - Support icinga2 2.10
[18:32:05] <Not-2d3>	 [02puppet] 07paladox opened pull request 03#879: Support icinga2 2.10 - 13https://git.io/fxBeR
[18:32:37] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBeg
[18:32:38] <Not-2d3>	 [02miraheze/puppet] 07paladox 0390e7755 - Update commands.conf
[18:32:40] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#879: Support icinga2 2.10 - 13https://git.io/fxBeR
[18:33:02] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxBew
[18:33:03] <Not-2d3>	 [02miraheze/puppet] 07paladox 03351d354 - Update commands.conf
[18:33:05] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#879: Support icinga2 2.10 - 13https://git.io/fxBeR
[18:33:23] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1568 files, 1.760MB, 2018-09-30 00:08:00 (1.8 weeks ago)  
[18:33:29] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxBeD
[18:33:31] <Not-2d3>	 [02miraheze/puppet] 07paladox 0358e581b - Support icinga2 2.10 (#879) * Support icinga2 2.10 * Update commands.conf * Update commands.conf
[18:33:32] <Not-2d3>	 [02puppet] 07paladox closed pull request 03#879: Support icinga2 2.10 - 13https://git.io/fxBeR
[18:33:34] <Not-2d3>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[18:33:35] <Not-2d3>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[18:48:14] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[19:04:55] <Not-2d3>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxBJX
[19:04:56] <Not-2d3>	 [02miraheze/dns] 07paladox 0363afd28 - Depool cp5 for kernel upgrade
[19:06:29] <paladox>	 !log upgrade cp5 kernel to 4.18 (performance/security reasons)
[19:06:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:09:08] <icinga-miraheze>	 PROBLEM - cp5 SSH on cp5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:09:12] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[19:09:20] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[19:11:02] <icinga-miraheze>	 RECOVERY - cp5 SSH on cp5 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0)   
[19:11:12] <icinga-miraheze>	 RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 1%  
[19:11:25] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online  
[19:11:27] <Not-2d3>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxBU4
[19:11:29] <Not-2d3>	 [02miraheze/dns] 07paladox 03293d1f4 - Revert "Depool cp5 for kernel upgrade" This reverts commit 63afd289ebecbcf7adade762b13965955c78addf.
[19:38:16] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5723  
[23:16:53] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40297 MB (10% inode=94%);  
[23:19:35] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 46430 MB (12% inode=94%);  
[23:19:45] <paladox>	 !log PURGE BINARY LOGS BEFORE '2018-10-13 00:18:00'; on db4
[23:19:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:50:15] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs