[00:05:21] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[00:33:21] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 00:33:11 UTC 2013  
[00:33:21] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[01:05:52] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[01:32:42] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 01:32:41 UTC 2013  
[01:32:52] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[02:05:27] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[02:05:50] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf12) at Sun Aug 18 02:05:50 UTC 2013
[02:06:03] <morebots>	 Logged the message, Master
[02:11:07] <icinga-wm>	 PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours  
[02:11:07] <icinga-wm>	 PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours  
[02:13:27] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf13) at Sun Aug 18 02:13:27 UTC 2013
[02:13:38] <morebots>	 Logged the message, Master
[02:23:36] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 18 02:23:36 UTC 2013
[02:23:48] <morebots>	 Logged the message, Master
[02:32:47] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 02:32:41 UTC 2013  
[02:33:27] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[02:40:01] <grrrit-wm>	 (03PS1) 10TTO: Add namespace aliases (shortcuts) for dewikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/79550 
[02:40:28] <grrrit-wm>	 (03CR) 10TTO: "Just pinging this, since it has fallen off the bottom of the dashboard." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76277 (owner: 10TTO)
[03:06:16] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[03:35:06] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 03:35:02 UTC 2013  
[03:35:16] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[04:07:19] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[04:35:49] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 04:35:45 UTC 2013  
[04:46:04] <icinga-wm>	 PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours  
[05:06:18] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[05:33:18] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 05:33:13 UTC 2013  
[05:34:18] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[05:55:06] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search27 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 327 bytes in 0.054 second response time  
[06:06:08] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[06:32:48] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 06:32:38 UTC 2013  
[06:33:08] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[06:53:02] <legoktm>	 Hi, db1026 is still super lagged. https://www.wikidata.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb=
[07:00:26] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search1009 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 345 bytes in 0.002 second response time  
[07:05:48] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[07:07:58] <MaxSem>	 legoktm, looking
[07:08:12] <legoktm>	 MaxSem: thanks. it's been this way for over 24 hours now :/
[07:08:28] <MaxSem>	 [Time] => 140255
[07:09:30] <legoktm>	 ?
[07:09:38] <MaxSem>	 in seconds
[07:09:45] <MaxSem>	 o_0
[07:10:08] <icinga-wm>	 PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours  
[07:10:49] <legoktm>	 yeah :/
[07:11:12] <legoktm>	 Sleeping for 8.0 seconds, 2013-08-16 23:26:12 <-- that was when it started (pacific time)
[07:15:49] <MaxSem>	 dafuq
[07:15:59] <MaxSem>	 SELECT /* ApiQueryAllUsers::execute */  ipb_deleted,COUNT(*) AS recentedits,user_name
[07:16:25] <MaxSem>	 even this tiny part of the query is already batshit wrong
[07:18:47] <MaxSem>	 !log Killed a few hung queries on db1026 that were causing massive lag
[07:18:59] <morebots>	 Logged the message, Master
[07:19:02] <Elsie>	 Lag is decreasing.
[07:19:31] <Elsie>	 MaxSem: Did you grab the bad queries?
[07:19:37] <MaxSem>	 ^^^
[07:19:48] <Elsie>	 I mean the full queries.
[07:19:53] <Elsie>	 It'd be good to have a bug report.
[07:20:04] <MaxSem>	 SHOW PROCESSLIST doesn't show full queries
[07:20:15] <Elsie>	 show full processlist does. ;-)
[07:20:25] <Elsie>	 Was it just ApiQueryAllUsers?
[07:20:29] <MaxSem>	 yup
[07:20:31] <Elsie>	 K.
[07:20:36] <Elsie>	 I'll file a bug against that.
[07:20:38] <legoktm>	 yay thanks MaxSem!
[07:21:03] <Elsie>	 "show full processlist\G"
[07:22:27] <MaxSem>	 hey, my window has only 5000 lines of scrollback fror \G!
[07:22:58] <Elsie>	 :-)
[07:23:09] <Elsie>	 Mine has as many lines as there is memory, I think.
[07:23:26] <MaxSem>	 @lag all
[07:23:41] <Elsie>	 Blergh, SAL needs rotating.
[07:23:51] <Elsie>	 It's at 400,000+ bytes.
[07:26:15] <Elsie>	 MaxSem, legoktm: https://bugzilla.wikimedia.org/show_bug.cgi?id=52979
[07:26:21] <Elsie>	 MaxSem: Thanks for killing those queries. :-)
[07:26:40] <legoktm>	 :)
[07:31:16] <MaxSem>	 hahaha
[07:31:22] <MaxSem>	 auactiveusers
[07:31:56] <Elsie>	 https://git.wikimedia.org/blob/mediawiki%2Fcore.git/98c7138c3b079beef79ad34711bb2fece0f5d809/includes%2Fapi%2FApiQueryAllUsers.php
[07:32:15] <Elsie>	 The COUNT(*) is easy to spot.
[07:32:48] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 07:32:42 UTC 2013  
[07:33:23] <Elsie>	 Not totally sure why it was hanging.
[07:33:45] <Elsie>	 Hmmm.
[07:33:48] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[07:33:48] <icinga-wm>	 PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:33:56] <legoktm>	       <db host="db1026" lag="0" /> :DDD
[07:34:39] <icinga-wm>	 RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)  
[07:35:07] <Elsie>	 Something about Wikidata's unusually large recentchanges table, I suppose.
[07:36:05] <Elsie>	 Hmmm.
[07:37:04] <Elsie>	 6154667 <-- count(*) from recentchanges on wikidatawiki_p
[07:37:16] <MaxSem>	 hehehe
[07:37:17] <Elsie>	 5933693 <-- count(*) from recentchanges on enwiki_p
[07:37:38] <Elsie>	 They're comparable.
[07:37:48] <Elsie>	 You'd think other wikis would be lagging/hanging.
[07:38:06] <Elsie>	 Perhaps more scanning on wikidata due to fewer active users or something.
[07:38:17] <Elsie>	 Query optimization is hard. Let's go shopping.
[07:39:05] <apergos>	 wikidata gets more edits / sec than en wp, has for months now
[07:39:39] * apergos  adds self to the bug

[07:39:55] <Elsie>	 Right. Faster rate, but overall lower number per month, surprisingly.
[07:40:11] <Elsie>	 Oh, never mind.
[07:40:19] <Elsie>	 Higher number per month. Comparison is hard.
[07:40:19] <apergos>	 yes, nm
[07:40:26] <apergos>	 heh it's also sunday morning (here)
[07:40:30] <apergos>	 so yer excused
[07:40:36] <Elsie>	 :-)
[07:40:43] <Elsie>	 Yeah, I'm not fully awake.
[07:40:56] <Elsie>	 Just enough to file a bug. ;-)
[07:41:10] <apergos>	 speaking of which I should start thinking about that cmelette with beet greens I was gonna have this morning
[07:41:22] <apergos>	 omelette
[07:41:40] <Elsie>	 cmelette with cmjohnson
[07:41:52] <apergos>	 but I already got a director's shair fixed and a hammock hung this morning (the hammock had a lot of yaks first)
[07:42:22] <Elsie>	 I thought "director's shair" was a meal for a second, heh.
[07:42:37] <apergos>	 no it was another bizarre typo
[07:42:48] <MaxSem>	 mehhh, ishmael is broken
[07:42:51] <apergos>	 an off by one given where the s and the c are on the keyboard
[07:43:06] <apergos>	 what's wrong with it?
[07:43:07] <Elsie>	 Call me broken.
[07:43:27] <Elsie>	 I'm hilarious.
[07:43:34] <MaxSem>	 I don't see sklow queries
[07:44:04] <Elsie>	 Isn't there supposed to be a query killer for slow queries?
[07:44:14] <MaxSem>	 not yet
[07:44:17] <Elsie>	 Think of the Domases.
[07:44:32] <Elsie>	 I'm sure there was one at some point.
[07:44:35] <apergos>	 I thought it was set up
[07:44:42] <Elsie>	 I think it breaks or something.
[07:44:49] <MaxSem>	 not active atm
[07:44:54] * Elsie  nods.

[07:45:11] <MaxSem>	 killing select queries should be safe
[07:45:33] <apergos>	 it was going to kill anything running as wikiuser that took longer than 5 mins I think
[07:45:39] <apergos>	 I even remember seeing actual code
[07:46:06] <Elsie>	 MaxSem: If there's lag, perhaps.
[07:46:40] <Elsie>	 It'd also be good to track which queries are hanging and file bugs.
[07:46:46] <Elsie>	 Rather than just killing.
[07:47:40] <MaxSem>	 oh, those theoreticians:P
[07:54:31] <Elsie>	 Ryan_Lane: Yo. Is the server admin log supposed to be moved to the archive?
[07:54:42] <Elsie>	 It seems Daniel moved it rather than copying and pasting last time.
[07:56:15] <Elsie>	 Hmmm, seems Antoine did it before him with Archive 20.
[07:57:38] <Elsie>	 > 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Jul 27 02:31:17 UTC 2013
[07:57:59] <Elsie>	 Is there any system in which that timestamp makes sense?
[08:00:06] <apergos>	 rt claims we already have that pt-kill crn job.  so somthing's up, I'll point to that bug and see if someone knows what's going on with that
[08:00:41] <apergos>	 ah what user were those queries running as?
[08:00:53] <apergos>	 MaxSem: any chance you remember?
[08:02:05] <Elsie>	 LocalisationUpdate has update.php.
[08:02:14] <Elsie>	 But Wikimedia must have a wrapper of some sort.
[08:05:20] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[08:08:31] <Elsie>	 45 KiB/s from Gerrit.
[08:08:33] <Elsie>	 Woo.
[08:11:30] <MaxSem>	 apergos, some anon and some non-bot user
[08:11:47] <apergos>	 er I mean what mysql user... wikiuser? wikiadmin?
[08:11:51] <apergos>	 MaxSem: 
[08:11:55] <MaxSem>	 wikiuser
[08:11:59] <apergos>	 hmm
[08:12:02] <MaxSem>	 as it was coming from API
[08:12:39] <apergos>	 and one more thing, any chance you remember how long the query had been running (more than 15 mins, more than an hour, ...?)
[08:12:59] <MaxSem>	 [Time] => 140255
[08:13:25] <Elsie>	 :-)
[08:13:26] <apergos>	 ok thanks, that needs to go on an rt ticket
[08:13:51] <Elsie>	 Or that lovely Bugzilla bug.
[08:14:15] <Elsie>	 Cloning operations/puppet is painfully slow.
[08:14:28] <Elsie>	 By the time it finishes, I'll have forgotten what I was going to do with it.
[08:14:32] <apergos>	 uh well it needs to go into the rt ticket about pt-kill
[08:15:25] <Elsie>	 RT is a plague.
[08:15:43] <apergos>	 that's how it goes
[08:15:51] <Elsie>	 Indeed.
[08:16:21] <apergos>	 oh I see, sean took the ticket but it hasn't yet been implemented. meh
[08:16:26] <MaxSem>	 grrr
[08:16:27] <MaxSem>	 3393485913 | wikiuser    | 10.64.32.67:47517  | wikidatawiki | Query   |      93 | Copying to tmp table                                                        | SELECT /* Wikibase\TermSqlIndex::getMatchingIDs  */ DISTINCT term_entity_id,term_weight  FROM `wb_te
[08:17:00] <Elsie>	 I can hear Domas crying on the inside.
[08:17:09] <apergos>	 notme
[08:17:10] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:17:10] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:17:10] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:17:10] <icinga-wm>	 PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:17:10] <icinga-wm>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:17:11] <icinga-wm>	 PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:17:50] <MaxSem>	 full query: SELECT /* Wikibase\TermSqlIndex::getMatchingIDs  */ DISTINCT term_entity_id,term_weight  FROM `wb_terms`   WHERE (term_language='zh-cn' AND term_search_key LIKE 'te%' AND term_type='label' AND term_entity_type='item') OR (term_language='zh-cn' AND term_search_key LIKE 'te%' AND term_type='alias' AND term_entity_type='item')  LIMIT 5000
[08:18:04] <MaxSem>	 DAFUUUUUQ??????!1
[08:18:42] <Elsie>	 Hmmmm. So it's `date`'s output that we're seeing in !log.
[08:18:55] <Elsie>	 I wonder if that's worth a bug.
[08:19:42] <MaxSem>	 rows: 539106
[08:20:25] <MaxSem>	 if I were Domas, I would've already been trashing WIkidata from servers:P
[08:20:40] <Elsie>	 // Domas optimized
[08:20:46] <Elsie>	 I got jokes.
[08:20:54] <Elsie>	 Where do Puppet bugs go in Bugzilla?
[08:21:25] <Elsie>	 Wikimedia --> General/Unknown it is.
[08:22:17] <MaxSem>	 puppet bugs should really go to RT
[08:22:21] <apergos>	 using intersect, using where, using temporary
[08:22:48] <Elsie>	 RT is a plague.
[08:22:50] <apergos>	 it's not like explain wouldn't warn folks
[08:23:03] <Elsie>	 This is also a really stupid bug I'm filing.
[08:24:00] <MaxSem>	 aha, it's https://bugzilla.wikimedia.org/show_bug.cgi?id=45529
[08:24:39] <apergos>	 well that just got more urgent too didn't it
[08:24:42] <apergos>	 *sigh*
[08:26:05] <apergos>	 okoff to make omelette right now this second
[08:26:56] <Elsie>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=52986
[08:32:50] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 08:32:43 UTC 2013  
[08:33:20] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[09:07:15] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[09:24:53] <paravoid>	 hm, what's going on?
[09:25:00] <paravoid>	 going through backlog
[09:26:27] <p858snake>	 paravoid: wd has crappy db queries by the looks
[09:26:56] <paravoid>	 yeah
[09:32:45] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 09:32:39 UTC 2013  
[09:33:15] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[09:37:55] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[09:38:55] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[10:05:33] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[10:13:37] <springle>	 !og resuming slave db52 after bug 52853 investigation 
[10:14:04] <springle>	 !log resuming slave db52 after bug 52853 investigation 
[10:14:15] <morebots>	 Logged the message, Master
[10:33:13] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 10:33:09 UTC 2013  
[10:33:33] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[10:42:51] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[10:43:51] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[10:46:51] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[10:48:51] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[11:05:32] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[11:32:42] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 11:32:35 UTC 2013  
[11:33:32] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[11:34:20] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Fix mobile redirect [operations/puppet] - 10https://gerrit.wikimedia.org/r/79563 
[11:43:24] <mark>	 !log Moved eqiad text-varnish traffic back to Squid temporarily
[11:43:37] <morebots>	 Logged the message, Master
[11:45:44] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Fix mobile redirect [operations/puppet] - 10https://gerrit.wikimedia.org/r/79563 (owner: 10Mark Bergsma)
[11:59:05] <paravoid>	 gaahh
[11:59:13] <paravoid>	 it's the http://
[11:59:21] <paravoid>	 in req.host
[11:59:24] <paravoid>	 how did I not see this
[11:59:51] <paravoid>	 aaan I just saw you fixed this
[12:00:11] <mark>	 :)
[12:00:30] <paravoid>	 grmbl
[12:06:23] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[12:11:23] <icinga-wm>	 PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours  
[12:11:24] <icinga-wm>	 PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours  
[12:26:12] <paravoid>	 !log authdns-update: add AAAA for sodium
[12:26:23] <morebots>	 Logged the message, Master
[12:32:44] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 12:32:38 UTC 2013  
[12:33:23] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[12:37:45] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Add nagios group for misc caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/79567 
[12:50:51] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[12:50:57] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Add nagios group for misc caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/79567 (owner: 10Mark Bergsma)
[12:51:51] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[13:05:42] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[13:15:30] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Add role::cache::ssl::wikimedia, factor out common resources [operations/puppet] - 10https://gerrit.wikimedia.org/r/79570 
[13:16:36] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Add role::cache::ssl::wikimedia, factor out common resources [operations/puppet] - 10https://gerrit.wikimedia.org/r/79570 (owner: 10Mark Bergsma)
[13:20:03] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Switch the role::cache::misc cluster to role::cache::ssl::wikimedia [operations/puppet] - 10https://gerrit.wikimedia.org/r/79572 
[13:21:04] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Switch the role::cache::misc cluster to role::cache::ssl::wikimedia [operations/puppet] - 10https://gerrit.wikimedia.org/r/79572 (owner: 10Mark Bergsma)
[13:28:33] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:29:32] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.048 second response time  
[13:33:32] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 13:33:27 UTC 2013  
[13:33:42] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[13:42:53] <icinga-wm>	 PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100%  
[13:43:43] <icinga-wm>	 RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.60 ms  
[13:57:34] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:58:23] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time  
[14:06:16] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[14:22:40] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Serve a /robots.txt if the backend doesn't [operations/puppet] - 10https://gerrit.wikimedia.org/r/79574 
[14:27:23] <grrrit-wm>	 (03Abandoned) 10Mark Bergsma: Serve a /robots.txt if the backend doesn't [operations/puppet] - 10https://gerrit.wikimedia.org/r/79574 (owner: 10Mark Bergsma)
[14:34:56] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 14:34:49 UTC 2013  
[14:40:06] <grrrit-wm>	 (03PS1) 10Mark Bergsma: Add Icinga monitoring (critical) for the misc-web cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/79576 
[14:42:02] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] Add Icinga monitoring (critical) for the misc-web cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/79576 (owner: 10Mark Bergsma)
[14:46:13] <icinga-wm>	 PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours  
[14:48:53] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[14:48:53] <icinga-wm>	 PROBLEM - DPKG on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[14:49:44] <icinga-wm>	 RECOVERY - DPKG on searchidx1001 is OK: All packages OK  
[14:49:53] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[15:00:00] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[15:02:41] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 15:02:36 UTC 2013  
[15:03:00] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[15:11:20] <icinga-wm>	 PROBLEM - SSH on amslvs1 is CRITICAL: Server answer:  
[15:17:20] <icinga-wm>	 RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[15:20:30] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[15:22:40] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:23:30] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time  
[15:26:20] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[15:43:05] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[15:43:55] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[16:10:16] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[16:20:03] <grrrit-wm>	 (03CR) 10MZMcBride: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/78944 (owner: 10QChris)
[16:32:47] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 16:32:41 UTC 2013  
[16:33:16] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[17:06:02] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[17:10:52] <icinga-wm>	 PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours  
[17:32:42] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 17:32:41 UTC 2013  
[17:33:02] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[18:06:14] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[18:18:04] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:18:04] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:18:04] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:18:04] <icinga-wm>	 PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:18:04] <icinga-wm>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:18:05] <icinga-wm>	 PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:33:24] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 18:33:15 UTC 2013  
[18:34:14] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[18:54:29] <Elsie>	 !log rotated server admin log wikitech page; testing logging
[18:54:34] <morebots>	 Logged the message, Master
[18:54:47] <Elsie>	 All better.
[19:06:25] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[19:32:45] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 19:32:38 UTC 2013  
[19:33:25] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[19:43:59] <grrrit-wm>	 (03CR) 10QChris: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/78944 (owner: 10QChris)
[20:00:07] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[20:02:07] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[20:06:24] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[20:32:44] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 20:32:40 UTC 2013  
[20:33:24] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[21:02:18] <grrrit-wm>	 (03CR) 10Edenhill: [C: 031] "Looks good!" [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon)
[21:04:10] <grrrit-wm>	 (03CR) 10Edenhill: "(3 comments)" [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/78160 (owner: 10Edenhill)
[21:07:50] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[21:32:40] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 21:32:38 UTC 2013  
[21:32:50] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[22:06:07] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[22:11:27] <icinga-wm>	 PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours  
[22:11:27] <icinga-wm>	 PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours  
[22:24:55] <grrrit-wm>	 (03Abandoned) 10Edenhill: Link with -lrt [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/78161 (owner: 10Edenhill)
[22:25:21] <grrrit-wm>	 (03Abandoned) 10Edenhill: Added JSON formatter, field name identifers and type casting option. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/78160 (owner: 10Edenhill)
[22:25:33] <grrrit-wm>	 (03Abandoned) 10Edenhill: Added support for escaping troublesome characters in tag content. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/78157 (owner: 10Edenhill)
[22:32:47] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 22:32:42 UTC 2013  
[22:33:07] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[22:40:16] <grrrit-wm>	 (03PS1) 10Edenhill: Added support for escaping troublesome characters in tag content. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79745 
[22:40:17] <grrrit-wm>	 (03PS1) 10Edenhill: Added JSON formatter, field name identifers and type casting option. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79746 
[22:40:18] <grrrit-wm>	 (03PS1) 10Edenhill: Added 'output = null' for testing purposes. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79747 
[22:40:19] <grrrit-wm>	 (03PS1) 10Edenhill: When reading offline VSL files (-r ..) make a copy of each matched tags data since the data is volatile. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79748 
[22:40:20] <grrrit-wm>	 (03PS1) 10Edenhill: Handle "Var: Val" with empty " Val"s. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79749 
[22:40:21] <grrrit-wm>	 (03PS1) 10Edenhill: Indent fix and clarified comment. [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79750 
[22:40:22] <grrrit-wm>	 (03PS1) 10Edenhill: Dont redeclare 'len' [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79751 
[22:40:23] <grrrit-wm>	 (03PS1) 10Edenhill: Decrease default log.level to 6 (info) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/79752 
[23:06:14] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours  
[23:33:44] <icinga-wm>	 RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 18 23:33:38 UTC 2013  
[23:34:14] <icinga-wm>	 PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours