[00:58:38] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 0 seconds  
[00:58:56] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds  
[01:16:29] <nagios-wm>	 PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours  
[01:39:36] <nagios-wm>	 PROBLEM - Disk space on stafford is CRITICAL: DISK CRITICAL - free space: /var/lib/puppet 758 MB (3% inode=92%):  
[01:55:03] <nagios-wm>	 PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours  
[02:17:02] <logmsgbot>	 !log LocalisationUpdate completed (1.19) at Sat Mar 24 02:17:02 UTC 2012
[02:17:07] <morebots>	 Logged the message, Master
[02:18:27] <juancarlos>	 Hi. I'm experiencing extreme slowness loading pages and files on the English Wikipedia. 
[02:23:57] <juancarlos>	 I would complain here: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#is_it_me_or_is_wiki_very_slow.3F but the page won't load all the way.
[04:29:05] <nagios-wm>	 RECOVERY - Disk space on stafford is OK: DISK OK  
[07:36:30] <nagios-wm>	 PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 4045 MB (3% inode=99%):  
[07:36:39] <nagios-wm>	 PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 4048 MB (3% inode=99%):  
[08:00:39] <nagios-wm>	 PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours  
[08:02:45] <nagios-wm>	 PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours  
[08:12:39] <nagios-wm>	 PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours  
[08:12:39] <nagios-wm>	 PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours  
[08:35:16] <Beau_>	 Does anyone knows about search engine issues on wmf wikis? I have received a report that new pages are not indexed by search engine on pl.wikipedia
[08:38:02] <p858snake|l>	 can you submit a BZ ticket please
[08:42:03] <Beau_>	 p858snake|l: I thought this may be a known issue, I'll just report it then.
[11:18:06] <nagios-wm>	 PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours  
[11:22:36] <nagios-wm>	 PROBLEM - Backend Squid HTTP on sq34 is CRITICAL: Connection refused  
[11:57:06] <nagios-wm>	 PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours  
[12:01:49] <penyulap>	 Hi, I'm wondering if anyone knows a way to display one image over the top of another image, not in the usual userpage way, but more like in a userbox ?
[12:48:26] <nagios-wm>	 PROBLEM - Backend Squid HTTP on knsq19 is CRITICAL: No route to host  
[12:48:35] <nagios-wm>	 PROBLEM - Frontend Squid HTTP on knsq25 is CRITICAL: No route to host  
[12:50:32] <nagios-wm>	 RECOVERY - Backend Squid HTTP on knsq19 is OK: HTTP OK HTTP/1.0 200 OK - 634 bytes in 0.218 seconds  
[12:50:41] <nagios-wm>	 RECOVERY - Frontend Squid HTTP on knsq25 is OK: HTTP OK HTTP/1.0 200 OK - 27735 bytes in 0.440 seconds  
[13:57:49] <Pikne>	 Hei! I wonder about hidepatrolled=1 in e.g http://meta.wikimedia.org/w/index.php?title=Special:RecentChanges&hidepatrolled=1. This link doesn't hide patrolled edits for me, is "hidepatrolled" parameter obsolete?
[15:15:10] <nagios-wm>	 PROBLEM - Router interfaces on mr1-eqiad is CRITICAL: CRITICAL: No response from remote host 10.65.0.1 for 1.3.6.1.2.1.2.2.1.8  with snmp version 2  
[15:17:16] <nagios-wm>	 RECOVERY - Router interfaces on mr1-eqiad is OK: OK: host 10.65.0.1, interfaces up: 32, down: 0, dormant: 0, excluded: 0, unused: 0  
[15:57:00] <mmovchin>	 What happened to the wikipedia servers?
[15:57:20] <multichill>	 Glitch?
[15:58:49] <mmovchin>	 It's still down in germany
[16:01:32] <tegra>	 wikipedia down ?
[16:02:46] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db52 is CRITICAL: CRIT replication delay 320 seconds  
[16:03:13] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db52 is CRITICAL: CRIT replication delay 346 seconds  
[16:06:40] <Reedy>	 mark: ^
[16:06:50] <tegra>	 ?
[16:06:57] <tegra>	 please
[16:07:06] <tegra>	 anyone can confirm ?
[16:07:11] <HardDisk_WP>	 and yeah, wikipedia is down here in munich
[16:07:38] <HardDisk_WP>	 interestingly, on  my server it works using lynx
[16:07:44] <HardDisk_WP>	 chrome at home stays loading
[16:07:54] <Reedy>	 Router work in underway at the moment
[16:08:14] <HardDisk_WP>	 ah ok
[16:08:27] <mmovchin>	 HardDisk_WP: You're from munich, too? :o
[16:08:35] <HardDisk_WP>	 yep, M-net
[16:08:40] <HardDisk_WP>	 whats your provider?
[16:08:47] <mark>	 can you show me a traceroute?
[16:08:50] <HardDisk_WP>	 sure
[16:08:51] <mmovchin>	 Vodafone
[16:08:56] <mmovchin>	 oh no
[16:08:56] <HardDisk_WP>	 omg, mobile?
[16:08:59] <mmovchin>	 it'S 1&1
[16:09:07] <HardDisk_WP>	 oh god, even worse
[16:09:08] <mmovchin>	 no, no it's 1&1
[16:09:17] <mmovchin>	 not, it's best
[16:09:27] <mmovchin>	 I never had any problems with them
[16:09:35] <Qcoder00>	 Load fine for me , UK routing (Thus)
[16:09:42] <mmovchin>	 also the support is responding 24h day if I need something
[16:09:53] <mark>	 stop talking, start tracerouting :)
[16:10:20] <mmovchin>	 mark: :)
[16:10:25] <krd1>	 traceroute to wikipedia-lb.esams.wikimedia.org (91.198.174.225), 30 hops max, 40 byte packets
[16:10:25] <krd1>	  1  fritz (192.168.0.8)  1.138 ms  1.242 ms  0.727 ms
[16:10:25] <krd1>	  2  rdsl-dsbg-de01.nw.mediaways.net (213.20.58.194)  22.82 ms  23.622 ms  23.199 ms
[16:10:25] <krd1>	  3  xmws-dsbg-de02-chan-18.nw.mediaways.net (195.71.242.190)  23.611 ms  23.476 ms  24.805 ms
[16:10:25] <krd1>	  4  rmws-dsbg-de02-gigaet-0-1-0.nw.mediaways.net (195.71.242.81)  24.091 ms  23.707 ms  24.128 ms
[16:10:26] <krd1>	  5  rmws-dsdf-de01-se-6-4.nw.mediaways.net (195.71.243.193)  25.223 ms  25.682 ms  24.893 ms
[16:10:26] <krd1>	  6  rmwc-dsdf-de01-xe-1-1-1-0.nw.mediaways.net (195.71.242.213)  24.354 ms  25.661 ms rmwc-dsdf-de01-chan-0-0.nw.mediaways.net (195.71.243.33)  25.05 ms
[16:10:27] <krd1>	  7  rmwc-frnk-de01-chan-3-0.nw.mediaways.net (195.71.254.145)  28.952 ms  28.364 ms  27.921 ms
[16:10:27] <krd1>	  8  xmws-frnk-de07-chan-0.nw.mediaways.net (213.20.249.202)  44.47 ms  28.219 ms  28.02 ms
[16:10:28] <krd1>	  9  ge1-2-cr0.ixf.de.as6908.net (78.41.155.221)  28.599 ms  28.924 ms  29.395 ms
[16:10:28] <krd1>	 10  te3-4-3502-cr0.nik.nl.as6908.net (78.41.154.17)  33.602 ms  34.592 ms  34.24 ms
[16:10:29] <krd1>	 11  ge-2-2.br1-knams.wikimedia.org (78.41.155.38)  35.329 ms  35.189 ms  35.225 ms
[16:10:31] <HardDisk_WP>	 mark, http://pastebin.com/Zgguxxgm
[16:10:35] <HardDisk_WP>	 both server and home
[16:10:54] <QueenOfFrance>	 krd1: use pastebin in future :)
[16:11:04] <krd1>	 sorry.
[16:11:48] <krd1>	 from different host it's up, although last traceroute hops are the same.
[16:12:32] <mark>	 krd1: what's your public ip address?
[16:12:46] <mmovchin>	 http://pastebin.com/feepKKuA
[16:12:51] <krd1>	 77.5.51.156
[16:13:22] <mmovchin>	 mine is: 93.134.207.224 
[16:13:22] <mark>	 it's not a routing problem, I can ping that just fine
[16:13:33] <tegra>	 mmm
[16:13:35] <mark>	 krd1: can you do a ping amssq31.esams.wikimedia.org ?
[16:13:46] <krd1>	 yes
[16:13:55] <tegra>	 yes
[16:13:59] <tegra>	 me too
[16:14:04] <HardDisk_WP>	 mark, mine is 89.238.66.122 for the server and 93.104.6.50 for home
[16:14:15] <krd1>	 mtr is clean, no packet loss.
[16:14:16] <Sebaso_WMDE>	 mark send you a tracerout via mail
[16:14:18] <mmovchin>	 me too, ping amssq31.esams.wikimedia.org works fine
[16:14:19] <Macropus>	 works for me also
[16:14:33] <HardDisk_WP>	 and I can ping amssq31.esams.wikimedia.org from both home and datacenter
[16:14:54] <tegra>	 there is a server problem.
[16:15:03] <tegra>	 seems
[16:15:05] <rillke>	 Status UP?
[16:15:42] <mark>	 can you ping 91.198.174.225 at all?
[16:15:44] <Sebaso_WMDE>	 --- amssq31.esams.wikimedia.org ping statistics ---
[16:15:45] <Sebaso_WMDE>	 35 packets transmitted, 35 packets received, 0.0% packet loss
[16:15:45] <Sebaso_WMDE>	 round-trip min/avg/max/stddev = 37.601/48.073/205.586/28.948 ms
[16:16:26] <mmovchin>	 yes, 0% loss
[16:16:30] <rillke>	 Service disruption!
[16:17:00] <Macropus>	 from here also 0%
[16:17:04] <Frakir>	 Ping de 91.198.174.225 0% packet loss, time 4004ms
[16:17:04] <HardDisk_WP>	 mark, yeah, 0% loss
[16:17:09] <Wiki13_>	 yup
[16:17:14] <Wiki13_>	 100 % loss here
[16:17:23] <Wiki13_>	 4 timeouts
[16:17:33] <Wiki13_>	 of 4
[16:17:35] <tegra>	 ping it's ok.
[16:17:41] <Sebaso_WMDE>	 --- 91.198.174.225 ping statistics ---
[16:17:41] <Sebaso_WMDE>	 19 packets transmitted, 19 packets received, 0.0% packet loss
[16:17:41] <Sebaso_WMDE>	 round-trip min/avg/max/stddev = 37.333/44.391/88.930/11.646 ms
[16:17:44] <tegra>	 but wikipedia doesn't loading
[16:17:48] <rillke>	 Don't get any image.
[16:17:56] <rillke>	 At Commons.
[16:17:58] <omtsh>	 Some wikipedias are down...
[16:18:58] <eptalon>	 heyas
[16:19:13] <eptalon>	 there seems to be connection probs to simple english wp
[16:19:23] <Reedy>	 Where are you?
[16:19:35] <eptalon>	 Reedy: Switzerland.
[16:19:48] <Sebaso_WMDE>	 Reedy: Berlin
[16:20:13] <Reedy>	 !log Some european users reporting oruting issues
[16:20:15] <mmovchin>	 looks that the problem is only in europe
[16:20:18] <morebots>	 Logged the message, Master
[16:20:27] <Wiki13_>	 yup
[16:20:29] <omtsh>	 Simplewikipedia down, I'm in Scotland...
[16:20:38] <Wiki13_>	 im from europe..
[16:20:51] <mmovchin>	 Germany => Munich
[16:20:52] <Qcoder00>	 I'm not having problems 
[16:21:02] <Qcoder00>	 I'm in the UK
[16:21:07] <HardDisk_WP>	 Reedy, munich (broken), server near Saarbruecken (works)
[16:21:22] <Sebaso_WMDE>	 reedy: Alice (O2/Telefonica) doesnt work. Telekome/T-Mobile works
[16:21:32] <mark>	 back
[16:21:36] <eptalon>	 HardDisk_WP: Switzerland connects to which server?
[16:21:48] <HardDisk_WP>	 no idea, the server in saarbruecken is mine^^
[16:21:49] <Reedy>	 Physical location won't so much make a difference. It's the routes that your isp/provider connects to the wikipedia servers in amsterdam
[16:23:01] <rillke>	 https://secure.wikimedia.org/wikipedia/de/wiki/Wikipedia:Hauptseite
[16:23:10] <rillke>	 gives you at least the main page...
[16:23:20] <Reedy>	 Evil
[16:23:34] <Macropus>	 Here works everything: KabelBW, Germany, connecting to USA (208.80.154.225)
[16:23:42] <Reedy>	 That's in the states, so you'll get to it via a different route
[16:24:06] <rillke>	 connected to 208.80.152.134
[16:24:20] <rillke>	 just the images come from a different server
[16:24:50] <eptalon>	 Reedy: simple can't be reached through secure?
[16:24:58] <mark>	 so, for whom is it still not working?
[16:25:09] <omtsh>	 o/
[16:25:09] <Reedy>	 Everything should be able to, but secure.wm.o really shouldn't be used
[16:25:18] <omtsh>	 https:// doesn't work either
[16:25:18] <Beau_>	 o/
[16:25:32] <Reedy>	 https:// will go the same as http:// would (for non secure.wm.o)
[16:25:41] <Nikerabbit>	 o7
[16:25:57] <Reedy>	 Nikerabbit does your arm hurt?
[16:25:58] <eptalon>	 mark: me (Switz)
[16:26:21] <eptalon>	 hangs in connecting
[16:26:42] <Nikerabbit>	 Reedy: nope, just adding +1 not working
[16:26:48] <Reedy>	 Orly
[16:27:22] <PCopp>	 esams squids seem to have a problem: when doing an http request they send the headers back but then the connection stalls without receiving any content
[16:27:27] <syslock>	 hm 91.198.174.225 responds with a 301, but following GET for /wiki/Wikipedia:Hauptseite yields no response at all.
[16:27:29] <PCopp>	 is it known?
[16:27:47] <Sebaso_WMDE>	 mark still not working
[16:27:59] <mark>	 ok
[16:28:02] <tegra>	 doesnt work.
[16:28:19] <tegra>	 it's a dns problem ?
[16:28:30] <Reedy>	 no
[16:29:07] <Wiki13_>	 hmm
[16:29:33] <Wiki13_>	 still not loading here
[16:29:52] <Beau_>	 PCopp same here, I just get the headers, no content
[16:29:56] <HardDisk_WP>	 mark, same as syslock here
[16:30:12] <mark>	 weird
[16:30:32] <mark>	 hmm
[16:30:34] <mark>	 perhaps an MTU problem
[16:31:04] <Wiki13_>	 some websites seem to load slower then normal here
[16:31:18] <tegra>	 doesn't loading atm.
[16:31:34] <omtsh>	 enwiki not loading for me now, scotland
[16:32:36] <Sebaso_WMDE>	 mark now wikipedia.org works, but neither de.wikipedia.org or en.wikipedia.org
[16:32:54] <omtsh>	 hm, enwiki back up but extremely slow
[16:33:03] <tegra>	 now i can't ping
[16:33:24] <omtsh>	 Yep, simplewiki back up
[16:33:40] <Sebaso_WMDE>	 mark traceroute de.wikipedia.org stops after  7  te3-4-3502-cr0.nik.nl.as6908.net (78.41.154.17)  70.504 ms  44.959 ms  40.345 ms
[16:33:42] <TBloemink>	 not for me yet
[16:33:46] <Wiki13_>	 not here
[16:34:20] <hoo>	 ok, fails for me now as well... traceroute failing after 8th hop
[16:34:51] <Beau_>	 wors for me
[16:34:54] <Beau_>	 works*
[16:35:46] <syslock>	 yes for me too
[16:35:57] <Sebaso_WMDE>	 now de.wikipedia.org works 
[16:35:57] <hoo>	 fine again :P
[16:35:59] <TBloemink>	 nlwiki back up
[16:36:00] <Sebaso_WMDE>	 10  wikipedia-lb.esams.wikimedia.org (91.198.174.225)  37.799 ms  37.137 ms  47.698 ms
[16:36:01] <mark>	 good
[16:36:12] <mark>	 damnit
[16:36:16] <mmovchin>	 back now
[16:36:30] <mark>	 !log Corrected MTU setting on cr2-knams's AMS-IX interface
[16:36:33] <morebots>	 Logged the message, Master
[16:36:34] <tegra>	 ok back
[16:36:44] <mark>	 is it working for everyone now?
[16:37:04] <Wiki13_>	 ehm
[16:37:08] <Wiki13_>	 i hope so
[16:37:17] <Sebaso_WMDE>	 seems so :)
[16:37:20] <syslock>	 thank you mark
[16:37:27] <omtsh>	 Thanks!
[16:37:40] <mark>	 well, I broke it too
[16:37:55] <eptalon>	 ok, looks like its back
[16:37:56] <syslock>	 ;)
[16:37:57] <Reedy>	 details! ;)
[16:38:04] <Sebaso_WMDE>	 :)
[16:41:30] <Superjuju10>	 Hi ! :)
[16:41:33] <David-suisse>	 coucou
[17:02:12] <nagios-wm>	 PROBLEM - Host br1-knams is DOWN: PING CRITICAL - Packet loss = 100%  
[17:07:00] <nagios-wm>	 PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused  
[17:10:45] <nagios-wm>	 PROBLEM - BGP status on csw2-esams is CRITICAL: CRITICAL: host 91.198.174.244, sessions up: 3, down: 2, shutdown: 0BRPeering with AS64600 not established - BRPeering with AS43821 not established - WIKIMEDIA-EUBR  
[17:21:15] <nagios-wm>	 RECOVERY - BGP status on csw2-esams is OK: OK: host 91.198.174.244, sessions up: 4, down: 0, shutdown: 0  
[17:28:00] <nagios-wm>	 RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.042 second response time  
[18:02:12] <nagios-wm>	 PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours  
[18:04:44] <nagios-wm>	 PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours  
[18:13:44] <nagios-wm>	 PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours  
[18:13:44] <nagios-wm>	 PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours  
[19:22:04] <jeblad>	 where are people when you need the.. They should be here right now! =(
[19:22:22] <jeremyb>	 uhuh
[19:22:35] <jeremyb>	 you haven't said anything to be responded to yet
[19:23:34] <jeblad>	 can say that, there could be some secret agents here..'
[19:23:47] * jeblad  puts on the tinfoilhat

[19:24:04] <jeblad>	 it can wait until monday
[19:37:18] <saper>	 hehe to should go into bugzilla quip or something
[19:37:25] <saper>	 s,to,it,
[19:44:35] <nagios-wm>	 PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%):  
[19:44:35] <nagios-wm>	 PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 195 MB (2% inode=61%): /var/lib/ureadahead/debugfs 195 MB (2% inode=61%):  
[19:44:35] <nagios-wm>	 PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 278 MB (3% inode=61%): /var/lib/ureadahead/debugfs 278 MB (3% inode=61%):  
[19:46:43] <logmsgbot>	 !log preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php  'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
[19:46:47] <morebots>	 Logged the message, Master
[19:46:57] <logmsgbot>	 !log preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php  'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()'
[19:47:01] <morebots>	 Logged the message, Master
[19:48:47] <nagios-wm>	 PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 211 MB (2% inode=61%): /var/lib/ureadahead/debugfs 211 MB (2% inode=61%):  
[19:48:47] <nagios-wm>	 PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 49 MB (0% inode=61%): /var/lib/ureadahead/debugfs 49 MB (0% inode=61%):  
[19:50:53] <nagios-wm>	 RECOVERY - Disk space on srv222 is OK: DISK OK  
[19:55:14] <nagios-wm>	 RECOVERY - Disk space on srv220 is OK: DISK OK  
[19:57:02] <nagios-wm>	 RECOVERY - Disk space on srv221 is OK: DISK OK  
[20:01:14] <nagios-wm>	 RECOVERY - Disk space on srv224 is OK: DISK OK  
[20:01:23] <nagios-wm>	 RECOVERY - Disk space on srv219 is OK: DISK OK  
[20:43:23] <nagios-wm>	 PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 248 MB (3% inode=61%): /var/lib/ureadahead/debugfs 248 MB (3% inode=61%):  
[20:57:50] <nagios-wm>	 PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 92 MB (1% inode=61%): /var/lib/ureadahead/debugfs 92 MB (1% inode=61%):  
[20:59:56] <nagios-wm>	 PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 148 MB (2% inode=61%): /var/lib/ureadahead/debugfs 148 MB (2% inode=61%):  
[21:02:54] <B-R-S>	 Who *is* this derp character I keep seeing lurking places?
[21:03:20] <Savh>	 A troll, vandal, etc.
[21:03:30] <omtsh>	 yes. That. ^
[21:04:08] <nagios-wm>	 PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 192 MB (2% inode=61%): /var/lib/ureadahead/debugfs 192 MB (2% inode=61%):  
[21:04:15] <B-R-S>	 Hmm. Just seems odd that I don't see 'em disrupt. I guess he's got enough chans to bother without worrying the ones I frequent (which suits me). 
[21:04:17] <nagios-wm>	 PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 210 MB (2% inode=61%): /var/lib/ureadahead/debugfs 210 MB (2% inode=61%):  
[21:04:17] <nagios-wm>	 PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 277 MB (3% inode=61%): /var/lib/ureadahead/debugfs 277 MB (3% inode=61%):  
[21:06:23] <nagios-wm>	 RECOVERY - Disk space on srv224 is OK: DISK OK  
[21:10:26] <nagios-wm>	 RECOVERY - Disk space on srv219 is OK: DISK OK  
[21:10:26] <nagios-wm>	 RECOVERY - Disk space on srv223 is OK: DISK OK  
[21:10:35] <nagios-wm>	 RECOVERY - Disk space on srv221 is OK: DISK OK  
[21:19:26] <nagios-wm>	 PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours  
[21:20:34] <jeremyb>	 hrmmmm, getting a stack from graphite
[21:20:53] <jeremyb>	 Thehelpfulone: can you fix that stack maybe? ;)
[21:20:59] <TBloemink>	 hello derp
[21:21:18] <jeremyb>	 or that works
[21:21:19] <Thehelpfulone>	 hmm?
[21:22:31] <TBloemink>	 and hi there again
[21:24:01] <TBloemink>	 *APPLAUSE*
[21:24:13] <omtsh>	 haha
[21:58:26] <nagios-wm>	 PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours  
[22:23:04] <Platonides>	 where are memcache servers in ganglia?
[22:43:18] <Joan>	 I love Asher.
[22:43:45] <||||||||||||||||>	 me too