[00:51:50] <icinga-wm>	 PROBLEM - Ubuntu mirror in sync with upstream on carbon is CRITICAL: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 12 hours old.  
[02:19:45] <logmsgbot>	 !log LocalisationUpdate completed (1.24wmf1) at 2014-04-27 02:19:43+00:00
[02:19:55] <morebots>	 Logged the message, Master
[02:25:24] <grrrit-wm>	 (03PS3) 10Ori.livneh: miscellaneous improvements for diamond module [operations/puppet] - 10https://gerrit.wikimedia.org/r/129075 
[02:28:34] <logmsgbot>	 !log LocalisationUpdate completed (1.24wmf2) at 2014-04-27 02:28:32+00:00
[02:28:41] <morebots>	 Logged the message, Master
[03:08:07] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 27 03:08:01 UTC 2014 (duration 8m 0s)
[03:08:12] <morebots>	 Logged the message, Master
[03:37:51] <icinga-wm>	 PROBLEM - Disk space on lvs3004 is CRITICAL: DISK CRITICAL - free space: / 1774 MB (3% inode=97%):  
[03:59:51] <icinga-wm>	 PROBLEM - Disk space on lvs3003 is CRITICAL: DISK CRITICAL - free space: / 1765 MB (3% inode=97%):  
[04:15:30] <icinga-wm>	 PROBLEM - Disk space on lvs3002 is CRITICAL: DISK CRITICAL - free space: / 1773 MB (3% inode=97%):  
[04:15:30] <icinga-wm>	 PROBLEM - Disk space on lvs3001 is CRITICAL: DISK CRITICAL - free space: / 1640 MB (3% inode=97%):  
[04:24:30] <icinga-wm>	 RECOVERY - Disk space on lvs3001 is OK: DISK OK  
[04:26:30] <icinga-wm>	 RECOVERY - Disk space on lvs3002 is OK: DISK OK  
[04:32:51] <icinga-wm>	 RECOVERY - Disk space on lvs3003 is OK: DISK OK  
[04:32:51] <icinga-wm>	 RECOVERY - Disk space on lvs3004 is OK: DISK OK  
[05:31:07] <springle>	 !log mariadb sql dump in progress db1048 /a for rebuilding db1046. ok to kill if necessary
[05:31:15] <morebots>	 Logged the message, Master
[09:34:50] <icinga-wm>	 RECOVERY - Ubuntu mirror in sync with upstream on carbon is OK: /srv/ubuntu/project/trace/carbon.wikimedia.org is over 0 hours old.  
[12:56:01] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[12:57:00] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical  
[13:18:43] <Romaine>	 tools.wmflabs keeps on having troubles the past days
[13:19:28] <YuviPanda>	 what troubles, Romaine?
[13:19:30] <YuviPanda>	 scfc_de: ^
[13:19:40] <Romaine>	 Internal error
[13:21:30] <hoo>	 WORKSFORME, also this channel is not for wikimedia labs
[13:21:56] <Romaine>	 which channel I should go to?
[13:22:03] <Romaine>	 it works half the time
[13:22:06] <hoo>	 #wikimedia-labs I think
[13:22:17] <Romaine>	 ok, thanks
[14:44:59] <Steinsplitter>	 Romaine: toolsname/ works, but toolsname without "/" at the end is broken
[14:48:56] <paravoid>	 !log stopping pybal on lvs300[1-4] to avoid the logspam
[14:49:07] <morebots>	 Logged the message, Master
[14:59:18] <grrrit-wm>	 (03PS1) 10Matanya: subversion: move ferm rules from module to role [operations/puppet] - 10https://gerrit.wikimedia.org/r/129965 
[15:14:02] <scfc_de>	 Steinsplitter: That wasn't the problem; geohack was stuck.
[17:59:31] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[18:56:31] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[19:09:11] <matanya>	 ori: can tungsten get firewalled ?
[19:09:25] <ori>	 hm?
[19:09:36] <ori>	 tungsten is just the host running graphite
[19:09:43] <matanya>	 have ferm applied on the host
[19:09:44] <ori>	 the 5xx req/min is for the entire cluster
[19:09:55] <ori>	 icinga just needs to attribute it to a host
[19:10:01] <matanya>	 not related to 5xx
[19:10:03] <ori>	 but it's a blood libel, really :P
[19:10:21] <ori>	 oh. well, dunno -- it's a question for ops
[19:10:45] <matanya>	 hmm, fair enough
[19:10:59] <matanya>	 i should ask you about performance.w.o
[19:11:13] <matanya>	 that should have a ferm rule to allow tcp 80