[00:16:15] <icinga-wm>	 PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 67 statistics  
[00:20:15] <icinga-wm>	 RECOVERY - MySQL Processlist on db1051 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics  
[00:38:29] <grrrit-wm>	 (03CR) 10Akosiaris: "(1 comment)" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85219 (owner: 10Ottomata)
[00:39:33] <grrrit-wm>	 (03PS3) 10Akosiaris: Not including consumer.properties and producer.properties in /etc/kafka. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85219 (owner: 10Ottomata)
[00:42:19] <grrrit-wm>	 (03CR) 10Akosiaris: [C: 032 V: 032] Not including consumer.properties and producer.properties in /etc/kafka. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85219 (owner: 10Ottomata)
[00:50:17] <grrrit-wm>	 (03PS1) 10Akosiaris: Typo fix MIRROR_CONFFILES_EXAMPLES [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85514 
[00:50:47] <grrrit-wm>	 (03CR) 10Akosiaris: [C: 032 V: 032] Typo fix MIRROR_CONFFILES_EXAMPLES [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85514 (owner: 10Akosiaris)
[02:07:46] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf17) at Sun Sep 22 02:07:46 UTC 2013
[02:07:51] <morebots>	 Logged the message, Master
[02:14:05] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf18) at Sun Sep 22 02:14:05 UTC 2013
[02:14:09] <morebots>	 Logged the message, Master
[02:27:32] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 22 02:27:32 UTC 2013
[02:27:36] <morebots>	 Logged the message, Master
[03:02:53] <icinga-wm>	 PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100%  
[03:03:33] <icinga-wm>	 RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.64 ms  
[03:06:13] <icinga-wm>	 PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused  
[03:07:07] <icinga-wm>	 RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.227 second response time  
[04:18:16] <icinga-wm>	 PROBLEM - Host ms-be4 is DOWN: PING CRITICAL - Packet loss = 100%  
[04:37:06] <icinga-wm>	 RECOVERY - Host ms-be4 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms  
[04:45:48] <grrrit-wm>	 (03CR) 10Rschen7754: [C: 031] Allow crats on outreachwiki to revoke translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85513 (owner: 10TTO)
[04:58:33] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.110 second response time  
[05:30:17] <grrrit-wm>	 (03PS1) 10TTO: Change ptwiktionary logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85516 
[05:56:02] <grrrit-wm>	 (03CR) 10Nemo bis: [C: 031] "Consensus good, already protected." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85516 (owner: 10TTO)
[07:07:27] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search1003 is CRITICAL: Connection timed out  
[07:08:27] <icinga-wm>	 RECOVERY - search indices - check lucene status page on search1003 is OK: HTTP OK: HTTP/1.1 200 OK - 269 bytes in 0.001 second response time  
[09:44:23] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[09:46:13] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[13:04:06] <icinga-wm>	 PROBLEM - RAID on analytics1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[13:05:46] <icinga-wm>	 PROBLEM - DPKG on analytics1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[13:06:06] <icinga-wm>	 PROBLEM - Disk space on analytics1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[13:15:12] <icinga-wm>	 PROBLEM - SSH on analytics1021 is CRITICAL: Server answer:  
[13:21:22] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[13:24:22] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[13:35:42] <icinga-wm>	 RECOVERY - DPKG on analytics1021 is OK: All packages OK  
[13:35:52] <icinga-wm>	 RECOVERY - Disk space on analytics1021 is OK: DISK OK  
[13:36:12] <icinga-wm>	 RECOVERY - SSH on analytics1021 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[14:17:13] <icinga-wm>	 RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000  
[14:20:23] <icinga-wm>	 PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[14:38:23] <icinga-wm>	 RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000  
[14:41:33] <icinga-wm>	 PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[16:49:30] <icinga-wm>	 PROBLEM - Disk space on vanadium is CRITICAL: DISK CRITICAL - free space: / 4275 MB (3% inode=95%):  
[17:00:30] <icinga-wm>	 RECOVERY - Disk space on vanadium is OK: DISK OK  
[17:40:45] <icinga-wm>	 RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000  
[17:43:55] <icinga-wm>	 PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[18:39:39] <icinga-wm>	 RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000  
[18:42:49] <icinga-wm>	 PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[18:53:09] <icinga-wm>	 PROBLEM - MySQL Processlist on db1043 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 67 statistics  
[18:55:09] <icinga-wm>	 RECOVERY - MySQL Processlist on db1043 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 7 statistics  
[19:43:18] <icinga-wm>	 RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.551 second response time  
[21:57:40] <icinga-wm>	 PROBLEM - SSH on lvs6 is CRITICAL: Server answer:  
[22:00:10] <icinga-wm>	 PROBLEM - SSH on lvs1001 is CRITICAL: Server answer:  
[22:01:10] <icinga-wm>	 RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[22:01:40] <icinga-wm>	 RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[22:04:40] <icinga-wm>	 PROBLEM - SSH on lvs6 is CRITICAL: Server answer:  
[22:07:10] <icinga-wm>	 PROBLEM - SSH on lvs1001 is CRITICAL: Server answer:  
[22:08:13] <icinga-wm>	 RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[22:10:43] <icinga-wm>	 RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[22:11:13] <icinga-wm>	 PROBLEM - SSH on lvs1001 is CRITICAL: Server answer:  
[22:15:13] <icinga-wm>	 RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[22:15:16] <grrrit-wm>	 (03PS1) 10Yuvipanda: Replace snarky comment [operations/puppet] - 10https://gerrit.wikimedia.org/r/85629 
[22:17:07] <grrrit-wm>	 (03CR) 10Legoktm: [C: 04-1] "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/85629 (owner: 10Yuvipanda)
[22:19:24] <grrrit-wm>	 (03PS1) 10Yuvipanda: Ensure that a matching version of JDK is present for the JRE [operations/puppet] - 10https://gerrit.wikimedia.org/r/85630 
[22:20:51] <grrrit-wm>	 (03PS2) 10Yuvipanda: Replace snarky comment [operations/puppet] - 10https://gerrit.wikimedia.org/r/85629 
[22:20:52] <YuviPanda>	 legoktm: ^
[22:20:56] <YuviPanda>	 legoktm: plz +1
[22:20:58] <YuviPanda>	 :P
[22:21:02] <legoktm>	 heheh
[22:21:57] <legoktm>	 heh
[22:22:06] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] Replace snarky comment [operations/puppet] - 10https://gerrit.wikimedia.org/r/85629 (owner: 10Yuvipanda)
[22:22:18] <YuviPanda>	 legoktm: is mc popular on TS at all?
[22:22:28] <YuviPanda>	 i've only read about it as a historical curiosity
[22:22:28] <legoktm>	 I didn't know what it was until I googled it
[22:22:32] <YuviPanda>	 hehe :P
[22:22:37] <Elsie>	 It's an editor, isn't it?
[22:22:38] <YuviPanda>	 it was Miguel's baby, IRIC
[22:22:57] <legoktm>	 https://wiki.toolserver.org/w/index.php?title=Special:Search&search=mc&fulltext=Search&ns0=1&ns1=1&ns2=1&ns3=1&ns4=1&ns5=1&ns6=1&ns7=1&ns8=1&ns9=1&ns10=1&ns11=1&ns12=1&ns13=1&ns14=1&ns15=1&ns100=1&ns101=1&redirs=0
[22:23:03] <YuviPanda>	 Elsie: it is a 'file manager'
[22:23:08] <Elsie>	 Oh, right.
[22:23:18] <Elsie>	 https://en.wikipedia.org/wiki/Midnight_Commander
[22:23:34] <Elsie>	 It's like an (S)FTP client, kind of.
[22:24:03] <YuviPanda>	 well, you login to the server and then run it *there*
[22:24:18] <Elsie>	 Right.
[22:24:28] <YuviPanda>	 hmm, yeah, 'kind of'
[22:24:31] <YuviPanda>	 without the copying bits :D
[22:24:49] <Elsie>	 It's like Finder.app, kind of.
[22:25:05] <grrrit-wm>	 (03PS1) 10Yuvipanda: It is apparently not that popular, perhaps [operations/puppet] - 10https://gerrit.wikimedia.org/r/85631 
[22:25:07] <YuviPanda>	 I remember using something similar looking in DOS
[22:25:26] <YuviPanda>	 press f8 on boot during win98 to drop into DOS :D
[22:25:44] <YuviPanda>	 legoktm: can you verify that the bug is gone now?
[22:25:50] <YuviPanda>	 legoktm: i'm writign up an explanation on the bug now
[22:32:18] <YuviPanda>	 whaaat
[22:32:21] <YuviPanda>	 morebots is alive here
[22:32:21] <morebots>	 I am a logbot running on tools-exec-07.
[22:32:21] <morebots>	 Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log.
[22:32:21] <morebots>	 To log a message, type !log <msg>.
[22:32:26] <YuviPanda>	 just not in -labs?!
[22:53:35] <legoktm>	 YuviPanda: -labs has 'labs-morebots'
[22:54:07] <YuviPanda>	 legoktm: but why didn't it log?
[22:54:13] <YuviPanda>	 when I did !log something
[22:54:19] <legoktm>	 its not there
[22:55:16] <Elsie>	 it's
[22:56:01] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours