[00:01:05] <gerrit-wm>	 New patchset: Pyoungmeister; "comment for record keeping of dbs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58636
[00:02:18] <gerrit-wm>	 New patchset: Pyoungmeister; "comment for record keeping of dbs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58636
[00:02:33] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58636
[00:02:38] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[00:04:09] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[00:08:03] <gerrit-wm>	 New review: MZMcBride; "Topic for #wikimedia-dev: Developers' talk about WMF's projects https://www.mediawiki.org/wiki/WMF_P..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57752
[00:08:09] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 00:08:05 UTC 2013
[00:09:09] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[00:16:43] <Susan>	 Hmm.
[00:16:49] <Susan>	 502 Bad Gateway on en.wikipedia.org.
[00:17:40] <LeslieCarr>	 even upon reload ?
[00:17:46] <LeslieCarr>	 and from US ?
[00:22:45] <gerrit-wm>	 New review: Tim Starling; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56107
[00:23:36] <gerrit-wm>	 New review: Krinkle; "#wikimedia-dev has become what you describe. #mediawiki is now a support channel. #wikimedia-tech is..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57752
[00:25:39] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[00:25:46] <Susan>	 LeslieCarr: It worked on refresh.
[00:25:50] <Susan>	 Yes, from D.C.
[00:26:00] <Susan>	 I haven't gotten it again.
[00:26:05] <Susan>	 (Yet.)
[00:26:16] * Krinkle  fixed redirect /view -> /wiki and made goo.gl HTTPS

[00:26:17] <Susan>	 Hi Krinkle.
[00:26:45] <Susan>	 https://wikitech.wikimedia.org/wiki/Server_admin_log
[00:26:49] <Susan>	 https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:26:53] <Susan>	 Hm, what.
[00:27:13] <Krinkle>	 hehe
[00:27:15] <Susan>	 https://wikitech.wikimedia.org/w/index.php?title=Server_admin_log&action=edit
[00:27:16] <Krinkle>	 transclusion
[00:27:19] <Susan>	 Cute.
[00:27:25] <Krinkle>	 No idea why
[00:27:35] <Krinkle>	 It shouldn't be all uppercase
[00:27:37] <Susan>	 https://wikitech.wikimedia.org/w/index.php?title=Server_admin_log&action=history # Nemo is adorable.
[00:27:47] <Susan>	 Yeah, but Ryan moved it during the migration.
[00:27:58] <Krinkle>	 but renaming is not an option, the bot is fixed on editing the uppercase name
[00:28:04] <Susan>	 Right.
[00:28:14] <Krinkle>	 meh
[00:28:18] <Susan>	 :-)
[00:28:31] <Krinkle>	 jobqueue needs to have something to purge on every !log
[00:28:38] <Susan>	 LeslieCarr: Out of curiosity, what is a 502 gateway error?
[00:28:54] <Ryan_Lane>	 I want to get away from editing wiki pages for the bot
[00:28:56] <Ryan_Lane>	 it's absurd
[00:29:10] <Ryan_Lane>	 we had 60k revisions for the log
[00:29:15] <Ryan_Lane>	 large ones, at that
[00:29:16] <LeslieCarr>	 bad upstream response
[00:29:40] <Krinkle>	 Ryan_Lane: Yeah, could easily be a special page with an API.
[00:29:48] <Ryan_Lane>	 or we could use EventLogging
[00:29:55] <Krinkle>	 Ryan_Lane: I recall there being an extension for that, written by someone here I believe.
[00:29:57] <Ryan_Lane>	 and just write a simple frontend for what it sticks in the database
[00:30:18] <Susan>	 LeslieCarr: I know the feeling. ;-)
[00:30:26] <LeslieCarr>	 hehe
[00:30:49] <Susan>	 Krinkle: It was one of Trevor's early projects.
[00:30:58] <Susan>	 I'm not sure if it just made input easier or changed the backend, though.
[00:31:19] <Krinkle>	 Though it should have ability to delete entries, and needs pagination of course.
[00:31:28] <Ryan_Lane>	 indeed
[00:31:36] <Ryan_Lane>	 that would be needed for any extension, though ;)
[00:31:46] <Ryan_Lane>	 and it should archive, rather than delete
[00:31:57] <Susan>	 Wait, we shouldn't delete portions of the server admin log?
[00:32:09] <Ryan_Lane>	 permanently?
[00:32:15] <Susan>	 Heh.
[00:32:15] <Krinkle>	 The advantage is that it should be'd be able to properly link to an entry
[00:32:18] <Susan>	 I'm mostly teasing.
[00:32:19] <Krinkle>	 (or a range maybe, even)
[00:32:32] <Ryan_Lane>	 when I say archive, I mean stick into another table so that we can truncate the table if we want
[00:32:32] <Krinkle>	 It'd be able to*
[00:32:39] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[00:32:43] <Ryan_Lane>	 or restore messages, if we have a troll
[00:32:49] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 00:32:46 UTC 2013
[00:32:50] <Krinkle>	 a troll with rights
[00:32:53] <Ryan_Lane>	 yep
[00:32:56] <Susan>	 Truncate a database table?
[00:32:56] <Krinkle>	 Those are the best kind of trolls.
[00:32:58] <Ryan_Lane>	 which we'd then ban :)
[00:32:59] <Susan>	 Why...
[00:33:09] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[00:33:12] <Krinkle>	 "if we want"
[00:33:26] <Krinkle>	 to put in another table instead of deleting from a table.
[00:33:41] <Susan>	 I thought archive and revision had taught has that bit fields are much easier.
[00:33:43] <Ryan_Lane>	 it's probably not a ton of work
[00:33:50] <Susan>	 taught us *
[00:33:52] <Susan>	 Typing is hard.
[00:33:59] <Ryan_Lane>	 bit fields probably is, yeah
[00:34:06] <Krinkle>	 True, a bool field would be easier.
[00:34:17] <Krinkle>	 But at least something that doesn't wipe it permanently
[00:34:43] <Susan>	 If the messages are already truncated due to Twitter, etc., you could just use the logging table, heh.
[00:34:48] <Susan>	 log_comment gets you 255 bytes.
[00:34:57] <Ryan_Lane>	 ewww
[00:35:02] * Coren  needs to write log2twitter now. :-)

[00:35:08] <Ryan_Lane>	 Coren: we already have that
[00:35:13] <Coren>	 o_O
[00:35:18] <Ryan_Lane>	 adminbot writes to the wiki and also to twitter
[00:35:26] <Coren>	 And nobody put it on the varnish servers yet?  :-)
[00:35:30] <Ryan_Lane>	 hahaha
[00:35:34] <mutante>	 and even to identi.ca
[00:35:57] <Susan>	 I thought identi.ca was dead.
[00:36:21] <Krinkle>	 we still push to twitter, too?
[00:36:23] <Krinkle>	 Interesting
[00:36:24] <Susan>	 https://e14n.com/evan/note/CuaONBbwQcyenRzC93R6Mw
[00:36:25] <Susan>	 Guess not.
[00:36:39] <Ryan_Lane>	 Krinkle: sooo.....
[00:36:42] <mutante>	 http://identi.ca/wikimedia
[00:36:47] <Ryan_Lane>	 I'm adding in a jquery library
[00:36:48] <Krinkle>	 I thought that functionality was lost when we migrated to the labs version of logmsgbot
[00:36:50] <Ryan_Lane>	 it's failing lint
[00:36:52] <mutante>	 hmm, no, wait, thats another account
[00:36:54] <Krinkle>	 Ryan_Lane: link?
[00:37:07] <Ryan_Lane>	 https://integration.wikimedia.org/ci/job/mediawiki-core-jslint/6696/console
[00:37:08] <Susan>	 https://identi.ca/wikimediatech
[00:37:14] <Ryan_Lane>	 am I expected to reformat it?
[00:37:27] <Ryan_Lane>	 or do we ignore it?
[00:37:57] <Krinkle>	 Ryan_Lane: .jshintignore for third pary code
[00:37:58] <icinga-wm>	 PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours
[00:38:04] <Ryan_Lane>	 ah ok
[00:38:04] <Krinkle>	 Ryan_Lane: Though it should still be reviewed, of course.
[00:38:11] <Krinkle>	 (for conflicts etc.)
[00:38:15] * Ryan_Lane  nods

[00:38:19] <Krinkle>	 jquery plugins can cause lots of trouble.
[00:38:25] <Krinkle>	 And js is very tolerant :)
[00:38:30] <Krinkle>	 (and naïve)
[00:38:38] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[00:39:39] <Ryan_Lane>	 hm
[00:39:40] <Ryan_Lane>	 btw
[00:39:50] <Ryan_Lane>	 should I put this in its own directory, or stick it into the jquery directory?
[00:40:47] <gerrit-wm>	 New patchset: Jdlrobson; "Update mobile.uploads.schema and add modules to mobile OutputPage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58551
[00:43:56] <Krinkle>	 Ryan_Lane: Does it have any other files?
[00:44:17] <Krinkle>	 I see it does
[00:44:21] <Krinkle>	 Yeah, a directory makes sense.
[00:45:25] <Krinkle>	 Ryan_Lane: I'm not sure how well this fits inside HTMLForm. Choosen is a pretty large UI framework. It looks a bit out of place. Should probably have custom stylesheet instead.
[00:46:13] <Ryan_Lane>	 well, it's modifying elements in htmlform
[00:46:32] <Krinkle>	 Yes, that's javascript, not css.
[00:46:33] <Krinkle>	 I mean this: https://gerrit.wikimedia.org/r/#/c/58281/10/resources/jquery.chosen/chosen.css,unified
[00:46:45] <Ryan_Lane>	 ah
[00:47:27] <Ryan_Lane>	 all of the css is used dynamically by the js
[00:48:08] <Krinkle>	 I mean the CSS (last I used Choosen) beyond basic layout. It doesn't drop in and adapt, it is mostly styled like Twitter Bootstrap
[00:48:16] <Krinkle>	 I (or someone else) will have to revisit it more thoroughly, do you need it for anything soon?
[00:48:31] <Ryan_Lane>	 nothing really soon, no
[00:48:48] <Ryan_Lane>	 I'm changing the project filter to use this, since a shitload of checkboxes really sucks
[00:50:18] <Krinkle>	 True
[00:50:39] <Ryan_Lane>	 seems since I fixed it for the tests that's it's broken anyway. heh
[00:50:44] * Ryan_Lane  goes to fix that

[00:50:47] <Krinkle>	 So you'd change it to a <select multiple>, which is even worse than checkboxes, but make it awesome with Chosen.
[00:50:54] <Ryan_Lane>	 actually, no
[00:51:11] <Ryan_Lane>	 it falls back to htmlform multselect
[00:51:12] <Krinkle>	 Chosen support taking a checkbox spread as input as well?
[00:51:38] <Ryan_Lane>	 if you add cssclass => 'chosen' to the field, it'll change it into a chosen field
[00:52:03] <Ryan_Lane>	 I wrote the js to transform the htmlform field
[00:52:19] <Ryan_Lane>	 https://gerrit.wikimedia.org/r/#/c/58281/8/resources/mediawiki/mediawiki.htmlform.js,unified <--
[00:52:27] <Krinkle>	 That's what I mean. The HTMLForm you output will be <select>-multiple instead of a checkbox spread (server-side change). So the fallback is somewhat regressed (checkboxes are easier to use as you can see what you've selected wheraes in multiselect its often a narrow crappy view)
[00:52:40] <Krinkle>	 but Chosen will make that regression worth it for practically 100% of users.
[00:52:47] <Ryan_Lane>	 actually, no
[00:53:01] <Ryan_Lane>	 it outputs a normal htmlform multiselect with checkboxes
[00:53:12] <Krinkle>	 both?
[00:53:23] * Krinkle  looks at the code this time

[00:53:23] <Ryan_Lane>	 I convert that into a <select multiple> on the client side
[00:53:29] <Ryan_Lane>	 then apply chosen
[00:53:29] <Krinkle>	 oh
[00:53:41] <Ryan_Lane>	 no regression
[00:53:56] <Ryan_Lane>	 I would never inflict a multi select on anyone :)
[00:56:59] <Ryan_Lane>	 (a select multiple, that is )
[01:02:53] <Susan>	 Heh.
[01:03:38] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[01:05:25] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[01:06:36] <icinga-wm>	 RECOVERY - Solr on vanadium is OK: All OK
[01:07:08] <gerrit-wm>	 New patchset: Ryan Lane; "Remove labstore3/4 bricks from management" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58451
[01:07:24] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58451
[01:10:09] <Ryan_Lane>	 !log disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them
[01:10:16] <morebots>	 Logged the message, Master
[01:15:15] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 01:15:08 UTC 2013
[01:15:15] <icinga-wm>	 RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Thu Apr 11 01:15:13 UTC 2013
[01:15:15] <icinga-wm>	 RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Thu Apr 11 01:15:14 UTC 2013
[01:15:25] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[01:18:25] <icinga-wm>	 RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Thu Apr 11 01:18:20 UTC 2013
[01:19:36] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[01:21:55] <icinga-wm>	 RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Thu Apr 11 01:21:48 UTC 2013
[01:24:36] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[01:28:56] <Krinkle>	 http://pump.io/
[01:29:09] <Krinkle>	 Well, at leas the 'failing' badge suites their color scheme
[01:29:36] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[01:29:40] <Krinkle>	 Don't put it on your home page :P
[01:29:42] <Krinkle>	 xD
[01:32:36] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[01:32:55] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 01:32:49 UTC 2013
[01:33:25] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[01:52:09] <Ryan_Lane>	 Krinkle: $dataPlaceholder.plain() rather than $dataPlaceholder.escaped() ?
[01:52:40] <Krinkle>	 Ryan_Lane: indeed.
[01:52:48] <Ryan_Lane>	 ok
[01:53:00] <Krinkle>	 Or call mw.msg( key ) which does mw.message( key ).plain()
[01:53:14] <Krinkle>	 Ryan_Lane: $ var names is PHP
[01:53:44] <Ryan_Lane>	 eh?
[01:53:49] <Krinkle>	 Just noticed you used $ in like every variable
[01:53:52] <Ryan_Lane>	 ah
[01:54:39] <Krinkle>	 Ryan_Lane: The $ in some variable names is just by convention (like Hungarian notation) to indicate the varirable contains an instance of jQuery (aka $)
[01:55:12] <Ryan_Lane>	 var $oldContainer = $(this), <— so, correct?
[01:55:15] <Krinkle>	 Which can easily be omitted if you prefer. But to add it to a non-jQuery containing varible is like doing var intSum = 'hello';
[01:55:33] <Krinkle>	 Ryan_Lane: Yep, that's good.
[01:58:38] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[02:00:03] <Ryan_Lane>	 "Don't create an <td> (couple lines up) only to let it be garbage collected here for a <div> element when it is re-assigned. Do it right the first time."
[02:00:37] <Ryan_Lane>	 the container variable is either going to be a td or a div
[02:00:59] <Ryan_Lane>	 I need to define the variable in the above scope, or I get warnings
[02:01:03] <Ryan_Lane>	 what am I supposed to do there?
[02:01:18] <Ryan_Lane>	 Krinkle: ^^
[02:01:45] <Krinkle>	 var $foo;
[02:02:00] <Krinkle>	 Ryan_Lane: Declare without defining
[02:02:15] <Ryan_Lane>	 I see
[02:02:15] <Ryan_Lane>	 ok
[02:02:17] <Krinkle>	 like public $mFoo; in php
[02:02:22] <Krinkle>	 in a class (sort of)
[02:02:50] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[02:03:00] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[02:03:01] * Ryan_Lane  nods

[02:10:40] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[02:10:50] <icinga-wm>	 PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours
[02:16:36] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013
[02:16:43] <morebots>	 Logged the message, Master
[02:32:40] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[02:52:37] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[03:00:01] <gerrit-wm>	 New patchset: Tim Starling; "Getting rid of my fenari SSH agent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58656
[03:02:37] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[03:02:51] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58656
[03:04:40] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[03:58:30] <gerrit-wm>	 New review: Tim Starling; "It's already installed via wikimedia-task-appserver. So is lilypond." [operations/puppet] (production) C: -2;  - https://gerrit.wikimedia.org/r/58504
[04:16:24] <gerrit-wm>	 New review: Tim Starling; "Putting them in misc::deployment::scap_scripts is not much help. That class is only installed on the..." [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/57854
[04:21:07] <icinga-wm>	 PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours
[04:33:48] <gerrit-wm>	 New review: Tim Starling; "I support moving the bots to #wikimedia-dev and making it clearer that that channel can be used for ..." [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/57752
[04:57:40] <gerrit-wm>	 New patchset: Tim Starling; "Update documentation for /root/.ssh/authorized_keys" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58658
[04:59:13] <gerrit-wm>	 New patchset: Tim Starling; "On brewster: disable access.log and store.log" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58457
[04:59:31] <gerrit-wm>	 New review: Tim Starling; "No review? It is almost full again (300MB available). I guess I will just self-review." [operations/puppet] (production); V: 2 C: 2;  - https://gerrit.wikimedia.org/r/58457
[04:59:32] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58457
[05:01:49] <gerrit-wm>	 New patchset: Tim Starling; "Don't use stdout of mergeMessageFileList.php" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57697
[05:01:59] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57697
[05:07:41] <gerrit-wm>	 New patchset: Tim Starling; "Disable LocalisationUpdate cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58660
[05:08:12] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58660
[05:13:09] <gerrit-wm>	 New review: Nemo bis; "Removing myself from reviewers: conscientious objection on visual identity fragmentation." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57683
[05:18:52] <gerrit-wm>	 New patchset: Ori.livneh; "Move IRC bots from #mediawiki to #wikimedia-dev" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58661
[05:22:23] <gerrit-wm>	 New patchset: Ori.livneh; "Move IRC bots from #mediawiki to #wikimedia-dev" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57752
[05:23:13] <gerrit-wm>	 Change abandoned: Ori.livneh; "The 'depends-on:' line threw off git-review. I intended to amend an existing change-set." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58661
[05:35:38] <icinga-wm>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[05:36:42] <gerrit-wm>	 New patchset: Tim Starling; "Keep memcached memory out of swap" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23872
[05:38:22] <gerrit-wm>	 New review: Tim Starling; "Asher: is that better?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23872
[05:41:22] <gerrit-wm>	 New patchset: Tim Starling; "udp2log on fluorine: relay MW errors to vanadium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58047
[05:41:35] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58047
[05:43:28] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:44:18] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time
[05:57:28] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:58:18] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time
[06:01:58] <icinga-wm>	 PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:03:49] <icinga-wm>	 RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s)
[06:04:29] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[06:31:10] <gerrit-wm>	 New patchset: Tim Starling; "Comment out call to clearMessageBlobs.php in l10nupdate-1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58604
[06:31:20] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58604
[06:31:53] <gerrit-wm>	 New patchset: Tim Starling; "Revert "Disable LocalisationUpdate cron job"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58663
[06:32:49] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 06:32:46 UTC 2013
[06:33:29] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[06:33:39] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58663
[06:56:36] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:59:26] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time
[07:01:40] <hashar>	 !log Put a live hack on fenari for {{bug|47112}}
[07:01:48] <morebots>	 Logged the message, Master
[07:04:00] <gerrit-wm>	 New patchset: Hashar; "revert highlight.php changes (bug 47112)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58664
[07:05:16] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[07:08:08] <gerrit-wm>	 New review: Hashar; "emergency fix" [operations/mediawiki-config] (master) C: 2;  - https://gerrit.wikimedia.org/r/58664
[07:08:31] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58664
[07:11:05] <logmsgbot>	 !log hashar synchronized docroot/noc/conf/highlight.php  'Revert highlight.php recent modifications {{bug|47112}} {{gerrit|58664}}'
[07:11:12] <morebots>	 Logged the message, Master
[07:15:56] <icinga-wm>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[07:42:46] <gerrit-wm>	 New patchset: Hashar; "zuul: update python modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58667
[07:43:58] <hashar>	 paravoid: thx for the packaging help with debian modules :-}   Could get them deployed now with https://gerrit.wikimedia.org/r/58667 :-]
[07:44:04] <hashar>	 will try out on labs the zuul upgrade
[07:44:40] <hashar>	 oh and I am now a member of the Python Module packaging team and got access to their svn repo.  I will thus be able to upload the packages  \O/
[08:04:35] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[08:06:35] <icinga-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:07:25] <icinga-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time
[08:08:15] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 08:08:07 UTC 2013
[08:08:35] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[08:08:45] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 08:08:35 UTC 2013
[08:09:10] <Nikerabbit>	 plop
[08:09:35] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[08:14:55] <icinga-wm>	 PROBLEM - Puppet freshness on cp3007 is CRITICAL: No successful Puppet run in the last 10 hours
[08:14:56] <logmsgbot>	 !log nikerabbit synchronized php-1.22wmf1/extensions/Translate/  'to master'
[08:15:03] <morebots>	 Logged the message, Master
[08:17:00] <Nikerabbit>	 !log emptying Translate solr instance and starting rebuilds
[08:17:12] <morebots>	 Logged the message, Master
[08:17:14] <Nemo_bis>	 Nikerabbit: how long does that take?
[08:20:24] <Nikerabbit>	 Nemo_bis: best guess 20 minutes
[08:30:03] <Nikerabbit>	 Is it okay to run mw-update-l10n?
[08:32:45] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 08:32:43 UTC 2013
[08:40:28] <Nikerabbit>	 !log translation memory database rebuild finished
[08:40:35] <morebots>	 Logged the message, Master
[08:44:07] <hashar>	 !log jenkins: stop git cloning mw/core over and over by reusing existing clone {{gerrit|58502}}. That should help with I/O starvation on gallium.
[08:44:14] <morebots>	 Logged the message, Master
[08:50:24] <ori-l>	 hashar: recently I've been using this locally, and I like it a lot: https://github.com/git/git/blob/master/contrib/workdir/git-new-workdir
[08:50:39] <hashar>	 what is it ?
[08:51:05] <ori-l>	 git new-workspace <path-to-repo> <path-to-new-workspace>. it uses symlinks to share objects across repositories
[08:51:20] <hashar>	 ohh
[08:51:54] <hashar>	 well my change is merely that jobs were killing the workspace and recloning mw/core
[08:52:08] <hashar>	 so a patchset would trigger like 10 - 12 clone of a the full repo
[08:52:53] <ori-l>	 yeah, i saw your patch. i think it's good. i'm not really suggesting you do this instead, it just reminded me of it. (i only discovered it recently so i'm still quite happy about it.)
[08:53:12] <hashar>	 on my laptop I mostly uses branch
[08:53:15] <hashar>	 and checkout
[08:53:26] <hashar>	 but if you want different wikis in parallel that would be nice
[08:54:20] <ori-l>	 i like not having to switch between master and the deployment branch(es)
[08:54:27] <ori-l>	 it's nice to have them all checked out at once
[08:54:37] <hashar>	 yeah the submodules are a bit nasty :-]
[08:54:53] <ori-l>	 yeah, exactly
[09:04:35] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[09:23:20] <hashar>	 !log restarting Jenkins to get the git verbose mode :(
[09:23:27] <morebots>	 Logged the message, Master
[09:52:49] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[09:52:49] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[09:52:49] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[10:05:26] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[10:10:11] <hashar>	 !log Jenkins is back up
[10:10:17] <morebots>	 Logged the message, Master
[10:15:46] <icinga-wm>	 PROBLEM - Varnish HTCP daemon on cp3006 is CRITICAL: Connection refused by host
[10:15:56] <icinga-wm>	 PROBLEM - Varnish HTTP upload-backend on cp3006 is CRITICAL: Connection refused
[10:16:06] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp3006 is CRITICAL: Connection refused by host
[10:16:16] <icinga-wm>	 PROBLEM - SSH on cp3006 is CRITICAL: Connection timed out
[10:18:26] <icinga-wm>	 PROBLEM - Host cp3006 is DOWN: PING CRITICAL - Packet loss = 100%
[10:18:56] <icinga-wm>	 RECOVERY - Varnish HTTP upload-backend on cp3006 is OK: HTTP OK: HTTP/1.1 200 OK - 634 bytes in 3.178 second response time
[10:19:06] <icinga-wm>	 RECOVERY - Host cp3006 is UP: PING OK - Packet loss = 0%, RTA = 87.78 ms
[10:19:06] <icinga-wm>	 RECOVERY - SSH on cp3006 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[10:19:46] <icinga-wm>	 RECOVERY - Varnish HTCP daemon on cp3006 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker
[10:20:06] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp3006 is OK: PROCS OK: 3 processes with command name varnishncsa
[10:33:19] <gerrit-wm>	 Change abandoned: Hashar; "per J^" [operations/debs/libvpx] (upstream) - https://gerrit.wikimedia.org/r/58071
[10:35:11] <hashar>	 apergos: mark: can one of you please merge https://gerrit.wikimedia.org/r/#/c/58667/    That simply adds a a python module on gallium :)
[10:36:27] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58667
[10:36:42] <hashar>	 thx :-]
[10:37:20] <mark>	 hashar: what's the status of the varnish work for beta? is that waiting on me? :)
[10:38:20] <hashar>	 oh yeah that stuff
[10:38:35] <icinga-wm>	 PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours
[10:38:37] <hashar>	 I got a few messy patches pending in ops/puppet but they don't seem to work properly right now :(
[10:38:55] <hashar>	 the role::cache:* classes references lvs IP to setup the backends
[10:39:06] <mark>	 right
[10:39:06] <hashar>	 that does not work in labs since we points directly to the apaches
[10:39:07] <mark>	 i was thinking
[10:39:19] <mark>	 another problem is that if people want to setup different labs projects with working varnish setups
[10:39:21] <mark>	 it gets even messier
[10:39:43] <mark>	 beta is nice as a final testbed before deployment, but it's not a good place to experiment with varnish setups for example
[10:39:44] <hashar>	 and since the mobile varnish seems to work more or less, I must say I haven't worked on making my patchwork
[10:40:39] <hashar>	 mark:maybe we could get role class that are standadlone
[10:40:47] <hashar>	 with their backend point to localhost
[10:40:53] <hashar>	 pointing
[10:41:09] <mark>	 hmm
[10:41:10] <mark>	 that's a good idea
[10:41:22] <hashar>	 this way one could install say role::cache::mobile::standalone on an instance. That would install mediawiki + apache + varnish that would relay back on apache
[10:41:26] <mark>	 we can support beta in the main manifests
[10:41:29] <hashar>	 apache would need to be bound to another ip / port though
[10:41:32] <mark>	 but we can't support a gazillion other projects
[10:42:01] <hashar>	 yeah that role::cache::configuration hash will not scale to gazillion
[10:42:06] <mark>	 indeed
[10:42:24] <mark>	 of course people can do their own configuration with puppetmaster::self
[10:42:25] <mark>	 but still
[10:43:28] <hashar>	 ideally we would have the varying conf in Hiera  and have our manifest fetch the conf from it
[10:43:47] <hashar>	 then people could just tweak their settings in hiera without relying on a puppet hash
[10:43:55] <hashar>	 but that requires a lot of engineering I guess
[10:44:16] <mark>	 mhm
[11:04:42] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[11:10:12] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[11:15:42] <icinga-wm>	 PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours
[11:15:42] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours
[11:19:49] <gerrit-wm>	 New review: MZMcBride; "Tim: Okay. Can you update the ops list of #wikimedia-dev with the ops list from #mediawiki?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57752
[11:31:12] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[11:44:39] <gerrit-wm>	 New patchset: MaxSem; "Oh heh facepalm" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58681
[11:46:22] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58681
[11:56:05] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[12:01:04] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[12:03:54] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[12:08:44] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 12:08:41 UTC 2013
[12:08:54] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[12:09:54] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 12:09:46 UTC 2013
[12:10:54] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[12:11:34] <icinga-wm>	 PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours
[12:11:48] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 12:11:35 UTC 2013
[12:11:54] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[12:12:24] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 12:12:21 UTC 2013
[12:12:54] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[12:12:54] <icinga-wm>	 RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Thu Apr 11 12:12:50 UTC 2013
[12:13:04] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 12:12:59 UTC 2013
[12:13:54] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[12:17:17] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[12:32:54] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 12:32:46 UTC 2013
[12:33:14] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[12:33:59] <MaxSem>	 mark, is it possible to purge varnish for testwiki only?
[12:37:10] <icinga-wm>	 PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa
[12:38:50] <icinga-wm>	 PROBLEM - Host ms-be1 is DOWN: PING CRITICAL - Packet loss = 100%
[12:45:07] <gerrit-wm>	 New patchset: Hashar; "Add ack-grep to standard packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50306
[12:46:05] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50306
[12:46:28] <Susan>	 :D
[12:46:46] <gerrit-wm>	 New review: Hashar; "PS2 was a rebase since the array of packages got exploded to be one package per line." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50306
[12:51:54] <hashar>	 yeah my package is building apparently
[12:51:56] <hashar>	 \O/
[12:54:38] <mark>	 MaxSem: should be possible
[12:56:43] <hashar>	 mark: paravoid: I have successfully backported a python module packages from raring to precise.  I have the .deb and other files in a labs instance, I am wondering what is the process to get it uploaded to our apt repo :-]
[12:57:01] <MaxSem>	 mark, I've just properly enabled resource variance on testwiki and it would be great if the cache was purged so that people can start testing today without getting stale HTML
[12:57:22] <mark>	 hashar: it involves sacrificing chickens
[12:57:28] <mark>	 MaxSem: ok, i'll purge testwiki in a bit
[12:58:21] <mark>	 that's test.m.wikimedia.org right?
[12:59:10] <hashar>	 ¡log looking for chickens
[12:59:39] <mark>	 hashar: has your package been reviewed already?
[12:59:46] <MaxSem>	 mark, test.m.wikipedia.org
[12:59:51] <MaxSem>	 thanks!
[13:00:07] <hashar>	 mark: not by ops but it comes from OpenStack and got included in Ubuntu raring.
[13:00:11] <icinga-wm>	 RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa
[13:00:25] <hashar>	 mark: should I create a git repo and upload the debian tree there?
[13:01:23] <mark>	 MaxSem: banned on the mobile backend caches, frontends will expire in 10 mins or so
[13:01:45] <MaxSem>	 cheers!
[13:01:46] <mark>	 hashar: no local modifications?
[13:01:54] <hashar>	 none needed apparently
[13:01:55] <mark>	 it just builds?
[13:01:58] <hashar>	 yeah
[13:02:02] <mark>	 ok
[13:02:16] <mark>	 just put the debs and .changes file somewhere I can find them
[13:04:48] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[13:04:50] <hashar>	 mark: here they are http://noc.wikimedia.org/~hashar/debs/python-extras/ :-D
[13:05:16] <mark>	 yumm, extra python
[13:05:25] <hashar>	 http://noc.wikimedia.org/~hashar/debs/python-extras/ALL.tar.gz  gives you everything
[13:05:28] <hashar>	 saves you a few clicks
[13:05:39] <mark>	 you're so accommodating
[13:06:09] <hashar>	 ;]
[13:07:56] <mark>	 !log Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
[13:08:04] <morebots>	 Logged the message, Master
[13:09:55] <gerrit-wm>	 New patchset: Hashar; "zuul: install python-extras module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58690
[13:10:23] <hashar>	 mark: and https://gerrit.wikimedia.org/r/58690 would deploy python-extras on gallium :-]
[13:10:34] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58690
[13:10:50] <hashar>	 yeah
[13:10:54] <mark>	 so
[13:10:58] <mark>	 is zuul reall i/o limited?
[13:11:00] <mark>	 really
[13:11:21] <hashar>	 Zuul merely keeps a copy of the clone
[13:11:25] <hashar>	 grr
[13:11:48] <hashar>	 Zuul merely keeps a copy of the git repositories and applies incoming patch sets to it.  The result is then used by Jenkins
[13:12:06] <hashar>	 the I/O issue is mostly with Jenkins.  We do too many writes on disk
[13:12:27] <mark>	 right
[13:12:28] <mark>	 what does it do?
[13:12:53] <hashar>	 and this morning I made a change to avoid cloning the mediawiki/core repository everywhere. That should help.
[13:13:06] <gerrit-wm>	 New patchset: Reedy; "Disable ClickTracking extension" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57769
[13:13:12] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57769
[13:13:36] <hashar>	 Whenever a change is submitted against mediawiki/core we trigger 10 - 12 jobs in Jenkins that each clone the repository in a local directory, install mediawiki and run whatever test suite they are meant to run
[13:13:55] <mark>	 ouch
[13:14:03] <hashar>	 and we have more and more tests running such as parsing  simulating a web browser in javascript to run our javascript test suite
[13:14:05] <logmsgbot>	 !log reedy synchronized wmf-config/  'Disable ClickTracking :D'
[13:14:07] <hashar>	 linting files
[13:14:09] <mark>	 and all without write caching
[13:14:12] <morebots>	 Logged the message, Master
[13:14:15] * Reedy  waits

[13:14:34] <mark>	 that setup is really suboptimal with everything on the rootfs too
[13:15:35] <hashar>	 so Rob proposed to put some SSD disks in the box to make the I/O faster
[13:15:50] <hashar>	 guess we will mount them as  /a/one and /a/two or something
[13:16:01] <mark>	  /srv
[13:16:08] <mark>	 i'm gonna stab anyone who keeps adding /a ;p
[13:16:17] * hashar  ducks

[13:16:18] <mark>	 at least be consistent then and use /a and /b ;p
[13:16:37] <mark>	 just put one ssd
[13:16:39] <mark>	 two ssds is silly
[13:16:45] <mark>	 no point at all
[13:16:48] <hashar>	  /srv is our docroot
[13:17:08] <hashar>	 hmm no, /srv/ is where we put our docroots. So we can creates /srv/foo and /srv/bar
[13:17:33] <hashar>	 one SSD would be there to host the Zuul local repositories which are fetched by the Jenkins jobs
[13:17:41] <hashar>	 the other SSD for the jobs workspaces
[13:18:59] <hashar>	 mark: the RT is https://rt.wikimedia.org/Ticket/Display.html?id=4916
[13:20:59] <gerrit-wm>	 New patchset: coren; "Refactoring puppetmaster::self to allow for puppet clients." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58540
[13:22:12] <mark>	 why can't one ssd do that?
[13:25:11] <hashar>	 160GB only? I can look at the capacity that would actually be needed
[13:26:48] <icinga-wm>	 PROBLEM - RAID on ms-be3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[13:30:28] <mark>	 yeah, would you need more?
[13:30:58] <mark>	 152 GB used currently
[13:31:05] <mark>	 i guess you might
[13:31:17] <hashar>	 there is lot of jenkins history we probably don't need on a ssd
[13:32:14] <hashar>	 Zuul only uses 2.7GB right now
[13:35:01] <^demon|sick>	 We should schedule some gc on the repositories too, will help keep your disk size down.
[13:35:07] <^demon|sick>	 (and make git operations generally faster)
[13:35:36] <hashar>	 even the replicated ones?
[13:35:39] <mark>	 should run gc on all
[13:35:42] <mark>	 automatically
[13:38:58] <^demon|sick>	 hmm, well those aren't jgit-managed ones, so c git might already be taking care of it.
[13:39:33] <ottomata>	 heya Coren, just curious, what's the log2udp2 thing for?
[13:40:29] <Coren>	 ottomata: I'm not entirely certain.  That was a request from Tim.  :-)  AFAIK, it's used by analytics to collect and high volume logs from things like varnish servers.
[13:40:43] <ottomata>	 yeah, we use it, but its not used a lot
[13:40:58] <ottomata>	 but why rewrite it?  was there something wrong with the old one?
[13:41:37] <ottomata>	 we only use it  in 3 places right now
[13:41:43] <Coren>	 Might have other uses as well.  It has two (big) functions the old one doesn't: it can merge log lines to packets, and it can number the log lines.
[13:41:45] <ottomata>	 2 to send logs to 2 university research projects
[13:42:10] <ottomata>	 and one to send traffic to  domas' webstats collector process on the same node
[13:42:11] <Coren>	 Oh, and add a prefix to the lines.
[13:42:23] <ottomata>	 which is kind of silly, it woul dbe better if collector just read from stdin, but wahtevs
[13:42:25] <ottomata>	 interesting
[13:42:41] <ottomata>	 from what I see in puppet, those are the only uses
[13:43:14] <ottomata>	 there's 'udp2log' we use that extensively
[13:43:29] <Coren>	 ottomata: That's the other end of that same pipe.  :-)
[13:43:31] <ottomata>	 that collects logs from frontend caches (varnish, etc.)
[13:43:34] <ottomata>	 right
[13:43:46] <ottomata>	 but for the most part we use udp2log to just filter and save files to disk
[13:44:13] <ottomata>	 there's not much use for log2udp, which is why I was wondering why tim wants it rewritten..
[13:44:25] <ottomata>	 i guess there's some use case I don't know of?
[13:44:51] <Coren>	 Like I said, I dunno the use cases; but perhaps there are places where Tim wanted to use log2udp but couldn't because it would have needed X feature.  Where X is what he asked me.  :-)
[13:45:07] <ottomata>	 hmm, k cool!
[13:45:44] <Coren>	 Besides, I was glad to write it.  I rarely get a chance to write code.  :-)
[13:46:13] <gerrit-wm>	 New patchset: Demon; "Allow jenkins admins to manage replicated git repos" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58692
[13:46:14] <ottomata>	 yeah, that's a fun project, simple and in C and cool
[13:46:30] <mark>	 analytics should get an agile C coder
[13:46:39] <mark>	 and fix their anonymization problem in a day or two
[13:46:56] <mark>	 it's getting a bit silly
[13:47:06] <drdee>	 udp-filter already has anonymization support
[13:47:11] <Coren>	 "anonymization problem"?
[13:47:23] <mark>	 cool, make it work for the level of traffic needed
[13:47:31] <mark>	 it's trivial to parallellize
[13:48:32] <Coren>	 mark: I dunno about how complicated the anonymization is, but how can this possibly need parallelization?  Wouldn't net/io times dominate by orders of magnitude?
[13:48:34] <mark>	 if I didn't have a conflict of interest here I'd offer my services with a contract ;)
[13:48:53] <ottomata>	 mark, I can do C decently, i'm sure not amazingly for that job but with review I'd do it fine, its just a matter of getting time allocated to that
[13:49:11] <mark>	 well, then it's a good thing you're using agile I'd think
[13:49:13] <mark>	 obviously it's important
[13:49:19] <ottomata>	 also, until recently I hadn't heard that geocoding and anonymizing was on option on the frontend caches
[13:49:29] <mark>	 doesn't need to be on the frontend caches
[13:49:34] <mark>	 can also be tied to udp2log
[13:49:40] <mark>	 anywhere before kraken works for now :)
[13:49:51] <mark>	 just something temporary, as long as it works
[13:50:16] <ottomata>	 we could probably do it now for small enough streams, i'm not sure about the mobile stream we are currently importing
[13:50:29] <mark>	 i think you should be able to do the entire stream honestly
[13:50:36] <ottomata>	 on a single node?
[13:50:40] <mark>	 what packet rate does udp-filter currently do?
[13:50:56] <ottomata>	 i don't know
[13:51:25] <mark>	 I think a single box should be able to do it, but even if not
[13:51:29] <mark>	 you can shard it easily too
[13:51:51] <mark>	 you've already done it with awk, if you do it inside the anonimizer program it would be more efficient
[13:52:50] <ottomata>	 yeah, i wonder if udp-filter had sharding support...
[13:52:52] <ottomata>	 if that would work
[13:53:04] <mark>	 i think you should maybe not use udp-filter but start from scratch
[13:53:08] <mark>	 (although I haven't really seen its code)
[13:53:22] <ottomata>	 its not pretty code, but I don't think it is too bad, it isn't that complicated
[13:53:23] <ottomata>	 but sure
[13:53:24] <drdee>	 starting from scratch is probably a good idea
[13:53:24] <ottomata>	 actually
[13:53:27] <ottomata>	 that would be even better
[13:53:33] <ottomata>	 keep it simple
[13:53:36] <ottomata>	 just a sharder would be nice
[13:53:40] <mark>	 yeah keep it simple, keep it efficient
[13:53:47] <mark>	 avoid doing memory allocations in the loop and all that
[13:55:03] <ottomata>	 we would loooove to do this with storm…and if we had time dedicated to doing that I don't think it would take that long to actually do.  writing the topology bits is the only part i don't konw how to do.
[13:55:12] <mark>	 you can do it in storm later
[13:55:15] <mark>	 you need something now
[13:55:16] <ottomata>	 buuut, who knows, maybe it would take longer than we'd like
[13:55:44] <mark>	 I really don't think it should take more than a day or two to write
[13:56:05] <ottomata>	 if the sharder was standalone, and we could use udp-filter as is for geocoding + anonymization, then i agree
[13:56:21] <mark>	 sharder standalone is much less efficient
[13:56:38] <mark>	 anyway, you can add that later
[13:56:47] <mark>	 perhaps you should profile udp-filter first
[13:56:49] <ottomata>	 ja
[13:56:50] <mark>	 it's a good educational experience
[13:56:54] <mark>	 use 'perf'
[13:57:00] <mark>	 it will tell you exactly where it spends its time
[13:57:16] <mark>	 then try to optimize that (starting from scratch with the bare needs may be best)
[13:58:00] <ottomata>	 hmm cool
[13:58:15] <mark>	 drdee: and yes... 1:1000 sampled log, while really not ideal, is what we've been doing on the stat* boxes for ages now, and I guess it's fine
[13:58:26] <mark>	 it's a lot better anonimity than full logging
[13:58:46] <drdee>	 we will anonymize it before storing it in hdfs
[13:58:46] <mark>	 but really, get this anonimization of the stream sorted and you'll have a lot of your problems taken care of, refine it after
[13:59:03] <drdee>	 yes we will do this
[13:59:08] <drdee>	 promise
[13:59:14] <drdee>	 ty
[14:00:22] <mark>	 does udp-filter run anywhere currently?
[14:00:52] <drdee>	 yes on emery, oxygen
[14:01:12] <mark>	 cool
[14:01:16] <mark>	 let's run perf on one of them
[14:01:18] <mark>	 any preference?
[14:01:31] <drdee>	 we are upgrading emery today
[14:01:36] <drdee>	 so not that one ;)
[14:01:40] <drdee>	 ottomata?
[14:01:53] <ottomata>	 hm
[14:02:10] <ottomata>	 i mean, we can run it anywhere mark
[14:02:18] <ottomata>	 you can pipe the log files through it and perf it that way
[14:02:23] <mark>	 the profiling may drop a few packets whle it runs
[14:02:38] <mark>	 "perf top" shows it while it runs now
[14:02:41] <mark>	 so it's easiest
[14:02:46] <ottomata>	 could we perf it on a non prod box
[14:02:53] <mark>	 you could
[14:03:25] <hashar>	 mark: seems I only need 20GB for now :)
[14:03:46] <mark>	 hashar: if you need more than one ssd for space then by all means, but it sounds a bit overkill otherwise ;)
[14:03:59] <ottomata>	 ja, we can just spawn up a udp2log instance on an analytics box from the multicast stream and try to geocode and anoymize the full stream
[14:04:02] <ottomata>	 and perf that
[14:04:13] <mark>	 yes
[14:04:30] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[14:04:45] <hashar>	 mark: the only advantage would be the ability to spread I/O on different disks.  I am updating the RT 4916
[14:05:00] <mark>	 hashar: you can allow writes to be cached anyway
[14:05:15] <ottomata>	 paravoid, reviewy time?  https://gerrit.wikimedia.org/r/#/c/58540/ :)
[14:05:23] <mark>	 (the ssd itself does that anyway, you can't rely on fsync)
[14:05:35] <mark>	 paravoid is off for the rest of the week
[14:05:41] <mark>	 the dude is working too hard ;)
[14:05:55] <paravoid>	 kind of
[14:05:56] <paravoid>	 :P
[14:06:09] <mark>	 he doesn't know how to /disconnect
[14:06:11] <ottomata>	 haha, ees true, he's pretty much always on
[14:06:15] <mark>	 always
[14:06:38] <hashar>	 paravoid: try /part :-D
[14:07:07] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58635
[14:07:15] <hashar>	 mark:  Iam not sure how to allow writes to be cached. Is that something that should be tuned on the disks?
[14:07:58] <mark>	 hashar: using xfs, setting nobarrier would be a start
[14:08:12] <mark>	 noatime
[14:08:15] <hashar>	 hmm that is where my ops knowledge ends :-]
[14:13:33] <mark>	 where do the apps write now?
[14:13:37] <mark>	 you can just mount the SSD partitions there
[14:13:59] <hashar>	 each job has a different workspace
[14:14:12] <hashar>	 would have to make jenkins write to the SSD mount point
[14:14:19] <mark>	 where does it write now?
[14:14:29] <hashar>	  /var/lib/jenkins/jobs/*/workspace/
[14:15:04] <mark>	 and mounting the SSD on /var/lib/jenkins/jobs is not appropriate?
[14:16:06] <hashar>	 well each jobs also maintain an history of all the builds. Not sure whether I want them on ssd since that takes a bit of space
[14:16:26] <mark>	 true
[14:17:10] <mark>	 just mount under /var/lib/jenkins/workspace then
[14:21:55] <hashar>	 yeah I can point the jobs to use that
[14:22:00] <icinga-wm>	 PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours
[14:27:56] <hashar>	 I am off might be back tonight
[15:04:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[15:15:59] <icinga-wm>	 PROBLEM - SSH on cp1043 is CRITICAL: Server answer:
[15:17:59] <icinga-wm>	 PROBLEM - SSH on caesium is CRITICAL: Server answer:
[15:18:59] <icinga-wm>	 RECOVERY - SSH on caesium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[15:21:49] <icinga-wm>	 PROBLEM - RAID on ms-be10 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[15:23:04] <icinga-wm>	 RECOVERY - SSH on cp1043 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[15:27:59] <icinga-wm>	 PROBLEM - SSH on caesium is CRITICAL: Server answer:
[15:36:00] <icinga-wm>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[15:37:00] <icinga-wm>	 RECOVERY - SSH on caesium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[16:01:55] <gerrit-wm>	 New patchset: MaxSem; "WIP: OSM module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36222
[16:03:34] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[16:08:44] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 16:08:42 UTC 2013
[16:09:34] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[16:09:54] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 16:09:45 UTC 2013
[16:10:34] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[16:10:44] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 16:10:42 UTC 2013
[16:11:34] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[16:11:34] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 16:11:32 UTC 2013
[16:12:34] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[16:13:04] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 16:12:53 UTC 2013
[16:13:34] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[16:32:54] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 16:32:45 UTC 2013
[16:33:34] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[16:43:53] <gerrit-wm>	 New patchset: Aude; "move wikidata cron jobs to terbium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58708
[17:04:38] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[17:06:08] <anomie>	 For the bug 27320 I'm working on, RoanKattouw_away suggested we watch the DB logs to see if there is a stampede (even if it doesn't rise to the level of bringing down the site). Where are these logs?
[17:08:43] <gerrit-wm>	 New patchset: Aude; "Set rate limit for editing on wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58709
[17:10:04] <greg-g>	 the people I'd ask aren't here, anomie
[17:10:25] <odder>	 aude: You're actually raising the limits for anons and newbies
[17:11:15] <aude>	 odder: for everyone
[17:11:46] <greg-g>	 anomie: by "here" I meant "in the office, as they usually are in the office but I don't see them now"
[17:11:56] <aude>	 mainly bots, but can't imagine humans editing faster than bots
[17:11:59] <odder>	 aude: Plus,  I believe its 'ip' instead of 'anon'
[17:12:34] <Reedy>	 anomie: /a/mw-log/dberror.log on fluorine and https://ishmael.wikimedia.org/
[17:12:50] <Reedy>	 "DB logs" is pretty vague ;)
[17:13:53] <anomie>	 Reedy- That's what Roan said: "Depending on what the fix looks like, we could deploy it to one wiki first, trigger an l10n update on it, then watch the DB logs very carefully to verify there is no stampede, not even a mini-stampede."
[17:13:53] <aude>	 from looking at the code, ip seems checked only for isNewbie()
[17:14:03] <aude>	 anon is user id === 0
[17:14:56] <odder>	 kk
[17:14:57] <cmjohnson1>	 !log europium power disconnected during routine data center work
[17:15:04] <morebots>	 Logged the message, Master
[17:16:38] <icinga-wm>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[17:17:18] * aude  trust reedy ^demon or whoever approves the patch to check it's correct

[17:17:25] <aude>	 looks correct to me
[17:19:15] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 17:19:14 UTC 2013
[17:19:22] <morebots>	 Logged the message, Master
[17:20:13] <greg-g>	 here we go....
[17:20:35] <aude>	 ok, maybe ip would work better
[17:20:50] * aude  wonders if "anon" is treated like a single user

[17:20:54] <aude>	 ip is more unique
[17:21:03] <aude>	 and ip would be a newbie
[17:22:46] <gerrit-wm>	 New patchset: Aude; "Set rate limit for editing on wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58709
[17:23:42] <sbernardin>	 !log taking down db11 for decommissioning
[17:23:49] <morebots>	 Logged the message, Master
[17:24:05] <gerrit-wm>	 New review: Aude; "would like sanity check that 'ip' is the right way to handle non-logged in users (instead of 'anon')" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58709
[17:31:52] <gerrit-wm>	 New patchset: Dzahn; "remove frdata SSL cert because it will be managed via the frack puppet instance (RT-4895)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58720
[17:33:18] <gerrit-wm>	 New patchset: Aaron Schulz; "Moved remaining jobs to redis." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58721
[17:33:24] <cmjohnson1>	 sbernardin:rt4923
[17:33:50] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58721
[17:34:06] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58720
[17:34:21] <sbernardin>	 cmjohnson1: working on that now
[17:34:45] <cmjohnson1>	 ok..cool...the dell techs replacing mainboards like to turn that off
[17:35:24] <logmsgbot>	 !log aaron synchronized wmf-config/jobqueue-eqiad.php  'Moved remaining jobs to redis'
[17:35:31] <morebots>	 Logged the message, Master
[17:40:21] <greg-g>	 sweet
[17:40:24] <greg-g>	 (re redis)
[17:41:37] <ottomata>	 heya, mutatne, you around?
[17:41:46] <ottomata>	 i'm going to start on that emery upgrade, not 100% I know all the steps
[17:41:58] <cmjohnson1>	 sbernarding: find it?
[17:42:19] <cmjohnson1>	 sbernardin: ^
[17:43:00] <sbernardin>	 cmjohnson1: nope....doing some research online
[17:43:12] <cmjohnson1>	 in bios (f2) ...integrated devices
[17:43:30] <cmjohnson1>	 make sure embedded nic 1 is enabled w/pxe
[17:43:45] <mutante>	 andre__: helo
[17:43:58] <RobH>	 cuz the dell techs are putzes who cannot bother to glance at a customers existing bios setup and replicate it.
[17:44:11] <RobH>	 yet they somehow call it full service.
[17:44:32] <ottomata>	 mutante: you around for qs? i'm going to start on that emery upgrade, not 100% I know all the steps
[17:45:01] <andre__>	 mutante, heja
[17:45:07] <mutante>	 ottomata: not quite yet, got an appointment at 11, but it's on my calendar from 11:30 on
[17:45:23] <ottomata>	 ok cool
[17:45:35] <ottomata>	 i have my 1on1 with robla at 11:30, which is why I was going to start now
[17:45:49] <ottomata>	 maybe we'll start at 12:00, zat ok?
[17:46:10] <mutante>	 ottomata: that's ok
[17:46:12] <ottomata>	 cool
[17:46:33] <mutante>	 andre__: what do you think about https://gerrit.wikimedia.org/r/#/c/42089/   merge or more discussion
[17:46:38] <mutante>	 see the new comments
[17:46:55] <mutante>	 also, the report.png file.. looks like it's not been deployed on server yet
[17:47:15] <mutante>	 jeremyb_: ^
[17:47:50] <andre__>	 mutante: I need to check that tomorrow (plus deactivate the still-not-working "urgent issues" query in the Weekly Bugzilla report email), didn't get there today :-(
[17:48:25] <mutante>	 andre__: alright, no worries
[17:51:46] <Thehelpfulone>	 mutante, there's also https://bugzilla.wikimedia.org/show_bug.cgi?id=47121 - does that need to go in RT as well as BZ?
[17:53:32] <mutante>	 Thehelpfulone: it's ok like that, don't worry about it
[17:54:14] <robla>	 ottomata: I can have our 1:1 early if you'd like
[17:56:29] <robla>	 notpeter: (or anyone else who knows) are s1 and s5 the only masters not on mariadb?  just trying to gauge how risky next week's migration is.
[17:57:35] <ottomata>	 robla, yeah, we can do now?
[17:57:57] <robla>	 yup....lemme make sure I can get a conf room
[18:00:57] <AaronSchulz>	 1 	44.09 (218.956) 	23.31 (83) 	1.89 	/* User::invalidateCache */ update `user` set user_touched = ? where user_id = ? and (user_touched < ?)
[18:01:11] <AaronSchulz>	 it's always nice how removing one thing makes the next bubble to #1
[18:04:06] <notpeter>	 robla: currently no eqiad masters are maris
[18:04:10] <notpeter>	 *maria
[18:04:27] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[18:04:47] <icinga-wm>	 RECOVERY - Host search23 is UP: PING OK - Packet loss = 0%, RTA = 26.52 ms
[18:07:17] <icinga-wm>	 PROBLEM - Lucene disk space on search23 is CRITICAL: Connection refused by host
[18:07:37] <icinga-wm>	 PROBLEM - SSH on search23 is CRITICAL: Connection refused
[18:10:27] <icinga-wm>	 PROBLEM - Lucene on search23 is CRITICAL: Connection refused
[18:15:43] <gerrit-wm>	 New patchset: Bsitu; "Turn Echo off on testwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58727
[18:16:28] <gerrit-wm>	 Change merged: Bsitu; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58727
[18:19:17] <icinga-wm>	 PROBLEM - mysqld processes on db1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld
[18:19:27] <icinga-wm>	 PROBLEM - NTP on search23 is CRITICAL: NTP CRITICAL: No response from NTP server
[18:20:47] <logmsgbot>	 !log bsitu synchronized wmf-config/InitialiseSettings.php  'Turn Echo off on testwiki'
[18:20:54] <morebots>	 Logged the message, Master
[18:21:47] <icinga-wm>	 PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100%
[18:22:37] <icinga-wm>	 RECOVERY - SSH on search23 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[18:22:47] <icinga-wm>	 RECOVERY - Host search23 is UP: PING OK - Packet loss = 0%, RTA = 26.54 ms
[18:24:29] <mutante>	 ottomata: ready when you are
[18:25:30] <ottomata>	 cool, 1on1 almost over
[18:30:26] <robla>	 greg-g: see Peter's comment from 25 min ago
[18:31:03] <gerrit-wm>	 New review: Hashar; "OH yeah :-]" [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/58692
[18:34:19] <greg-g>	 robla: yeah, saw, no master on maria yet
[18:34:24] <gerrit-wm>	 New patchset: MaxSem; "Rm special-casing for root URLs of mobile sites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58729
[18:41:46] <gerrit-wm>	 New patchset: Bsitu; "Add checkDelay to echo email bundle job" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58731
[18:46:01] <icinga-wm>	 PROBLEM - Host ms-be9 is DOWN: PING CRITICAL - Packet loss = 100%
[18:46:08] <gerrit-wm>	 New patchset: Ottomata; "Prepping for emery upgrade to precise." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58136
[18:46:37] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58136
[18:47:38] <ottomata>	 !log emery going down for upgrade to precise
[18:47:46] <morebots>	 Logged the message, Master
[18:48:16] <notpeter>	 ottomata: as the captain, you're going to have to go down with the box
[18:48:30] <ottomata>	 hm
[18:48:32] <drdee>	 i love the moral support!
[18:48:33] <ottomata>	 do I get to come back up?
[18:48:40] <drdee>	 :D
[18:48:42] <notpeter>	 ottomata: only if the box comes back up
[18:48:49] <ottomata>	 hmmm, sounds fair
[18:49:31] <icinga-wm>	 PROBLEM - Host db1014 is DOWN: PING CRITICAL - Packet loss = 100%
[18:49:53] * drdee  is keeping his fingers crossed that ottomata, aka 'The Captain' safely returns from his journey to Emery island

[18:50:24] <notpeter>	 db1014 is me, btw
[18:51:36] <notpeter>	 drdee: thar be pirates
[18:51:36] <mutante>	 Attempting PXE Boot (emery)
[18:51:36] * drdee  the route to Emery Island is wrought with peril and dangerous lurking monsters

[18:51:36] <drdee>	 harhar
[18:51:36] <mutante>	 !log PXE booting emery into installer
[18:51:36] <mutante>	 worked
[18:51:36] <morebots>	 Logged the message, Master
[18:51:37] <ottomata>	 cool!
[18:51:38] <ottomata>	 Apr 11 18:51:20 brewster atftpd[23688]: Serving precise-installer/ubuntu-installer/amd64/pxelinux.0 to 208.80.152.184:2070
[18:51:39] <mutante>	 in precise installer
[18:51:41] <notpeter>	 see, I'm glad that we do our work in public channels :)
[18:52:01] <icinga-wm>	 PROBLEM - Host emery is DOWN: PING CRITICAL - Packet loss = 100%
[18:53:06] <mutante>	 ehm..timeout..waiting
[18:54:21] <icinga-wm>	 RECOVERY - NTP on search23 is OK: NTP OK: Offset -0.008543133736 secs
[18:54:41] <icinga-wm>	 RECOVERY - Host db1014 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms
[18:54:46] <ottomata>	 typical!
[18:54:48] <mutante>	 Boot failed: press a key to retry, or wait for reset...
[18:54:50] <mutante>	 :/
[18:55:10] <ottomata>	 it timed out on serving the image?
[18:55:16] <ottomata>	 Serving precise-installer/ubuntu-installer/amd64/linux
[18:55:17] <ottomata>	 ?
[18:55:18] <mutante>	 yep
[18:55:31] <ottomata>	 weird, that it did teh first stuff though
[18:55:34] <notpeter>	 brewster is working, btw
[18:55:39] <mutante>	 well, it already showed this:
[18:55:40] <notpeter>	 I just imaged db1014
[18:55:41] <gerrit-wm>	 New patchset: Bsitu; "Turn Echo back on for testwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58736
[18:55:42] <mutante>	 Ubuntu 12.04 Precise Pangolin AMD64 (Wikimedia edition)
[18:56:07] <ottomata>	 hm, maybe the partman stuff failed?  would we know?
[18:56:13] <ottomata>	 did it get that far?
[18:56:35] <ottomata>	 why does icinga send me attachements
[18:56:35] <gerrit-wm>	 New review: Aaron Schulz; "Jobs still don't get picked up on testwiki, we should just avoid it for now." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58736
[18:56:38] <ottomata>	 does that happen to you guys too?
[18:56:41] <icinga-wm>	 PROBLEM - Host db1015 is DOWN: PING CRITICAL - Packet loss = 100%
[18:56:51] <icinga-wm>	 PROBLEM - DPKG on db1014 is CRITICAL: Connection refused by host
[18:57:07] <icinga-wm>	 PROBLEM - SSH on db1014 is CRITICAL: Connection refused
[18:57:07] <icinga-wm>	 PROBLEM - Disk space on db1014 is CRITICAL: Connection refused by host
[18:57:21] <icinga-wm>	 PROBLEM - RAID on db1014 is CRITICAL: Connection refused by host
[18:57:31] <notpeter>	 ottomata: partman is later
[18:57:34] <ottomata>	 Content-Type: application/octet-stream
[18:57:39] <bsitu>	 AaronSchulz: I have disabled the email bundling, there should be no jobs created on testwiki
[18:57:40] <notpeter>	 that would drop you into a dialogue box
[18:57:45] <ottomata>	 aye
[18:57:50] <mutante>	 ottomata: yea, if there is a partman error it usually shows up later
[18:57:53] <ottomata>	 ok
[18:57:54] <notpeter>	 db1015 is me as well
[18:57:58] <Reedy>	 aude: https://www.mediawiki.org/wiki/Manual:$wgRateLimits
[18:58:07] <Reedy>	         'anon'   => null, // for any and all anonymous edits (aggregate)
[18:58:07] <Reedy>	         'user'   => null, // for each logged-in user
[18:58:07] <Reedy>	         'newbie' => null, // for each recent account; overrides 'user'
[18:58:07] <Reedy>	         'ip'     => null, // for each anon and recent account
[18:58:07] <Reedy>	         'subnet' => null, // ... with final octet removed
[18:58:08] <AaronSchulz>	 bsitu: and no other emailing for echo there uses the queue?
[18:58:13] <LeslieCarr>	 i just snuck a look at daniel's screen - interesting
[18:58:16] <Coren>	 Icinga sending you attachments is just the first step.  Next come the promises of l0ng.er man.h00d and forgotten Nigerian millions.  :-)
[18:58:23] <LeslieCarr>	 it's getting the installer but not completely loading it up
[18:58:40] <mutante>	 i feel like just trying it again
[18:58:54] <gerrit-wm>	 New review: Reedy; "Both are seemingly valid:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58709
[18:59:06] <notpeter>	 Coren: have you seen our icinga messages? :)
[18:59:15] <bsitu>	 AaronSchulz: there is a global variable to determine whether to use email bundling, it's off now
[18:59:22] <mutante>	 ottomata: repeating
[18:59:34] <bsitu>	 I just pushed out the code but email bundling is off
[18:59:47] <AaronSchulz>	 bsitu: so all echo emails there are synchronous and don't use any queueing?
[18:59:48] <Coren>	 notpeter: Yeah.
[19:00:07] <bsitu>	 correct, unless I set it back on
[19:00:25] <gerrit-wm>	 New review: Aaron Schulz; "Removing -2 per IRC" [operations/mediawiki-config] (master) C: 1;  - https://gerrit.wikimedia.org/r/58736
[19:00:27] <ottomata>	 k
[19:00:34] <notpeter>	 oh! are we ever going to turn off double-paging?
[19:00:36] <notpeter>	 LeslieCarr: ^
[19:00:39] <AaronSchulz>	 bsitu: oh, that's ok then
[19:00:43] <ottomata>	 cmoooon bwoostah!
[19:01:04] <bsitu>	 AaronSchulz: so when can we turn it on?
[19:01:05] <LeslieCarr>	 notpeter: oh yes, let's fix that
[19:01:10] <LeslieCarr>	 with uninstalling!
[19:01:12] <LeslieCarr>	 huzzah
[19:01:14] <notpeter>	 woo!
[19:01:21] <bsitu>	 AaronSchulz: only for test/test2/mediawiki
[19:01:47] <LeslieCarr>	 !log uninstalling nagios from spence
[19:01:51] <icinga-wm>	 RECOVERY - Host db1015 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms
[19:01:54] <morebots>	 Logged the message, Mistress of the network gear.
[19:01:55] <Reedy>	 Wheeeee
[19:02:05] <ottomata>	 timeouts
[19:02:49] <AaronSchulz>	 bsitu: any time I suppose
[19:02:57] <ottomata>	 mutante, i dunno, what do we do now?
[19:02:58] <icinga-wm>	 RECOVERY - SSH on db1014 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[19:03:18] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[19:03:47] <bsitu>	 AaronSchulz: what do you mean by "Jobs still don't get picked up on testwiki"?  you mean it's not added to jobs-loop.sh?
[19:04:08] <icinga-wm>	 PROBLEM - RAID on db1015 is CRITICAL: Connection refused by host
[19:04:08] <icinga-wm>	 PROBLEM - DPKG on db1015 is CRITICAL: Connection refused by host
[19:04:18] <icinga-wm>	 PROBLEM - SSH on db1015 is CRITICAL: Connection refused
[19:04:18] <icinga-wm>	 PROBLEM - Disk space on db1015 is CRITICAL: Connection refused by host
[19:04:58] <AaronSchulz>	 the runners check a hash in redis that says what wikis have jobs, but testwiki jobs are inserted via apaches in tampa and they update the hash on a different redis server that nothing checks
[19:05:08] <icinga-wm>	 PROBLEM - Disk space on search23 is CRITICAL: NRPE: Command check_disk_space not defined
[19:05:18] <icinga-wm>	 PROBLEM - RAID on search23 is CRITICAL: NRPE: Command check_raid not defined
[19:05:28] <icinga-wm>	 PROBLEM - search indices - check lucene status page on search23 is CRITICAL: Connection refused
[19:05:51] <AaronSchulz>	 it also had problems when it use memcached as well
[19:05:58] <icinga-wm>	 PROBLEM - DPKG on search23 is CRITICAL: NRPE: Command check_dpkg not defined
[19:06:08] <icinga-wm>	 RECOVERY - Lucene on search23 is OK: TCP OK - 0.027 second response time on port 8123
[19:06:14] <AaronSchulz>	 testwiki is probably broken in a lot of ways regarding memcached and redis
[19:06:29] <AaronSchulz>	 test2wiki is fine of course
[19:06:35] <sbernardin>	 !log taking down ms-be9 for hw replacement
[19:06:42] <morebots>	 Logged the message, Master
[19:06:53] <AaronSchulz>	 !log All jobs finished copying into redis
[19:07:00] <morebots>	 Logged the message, Master
[19:07:12] <bsitu>	 AaronSzhulz: I see, I guess mediawiki is fine as well?
[19:07:18] <AaronSchulz>	 yep
[19:07:19] <ottomata>	 heya LeslieCarr, notpeter
[19:07:24] <ottomata>	 do you guys get weird emails from icinga?
[19:07:26] <LeslieCarr>	 hey ottomata
[19:07:27] <ottomata>	 all the ones I get are
[19:07:28] <ottomata>	 Content-Type: application/octet-stream
[19:07:28] <ottomata>	 Content-Transfer-Encoding: quoted-printable
[19:07:32] <AaronSchulz>	 testwiki is the only thing with this kinds of hacks
[19:07:33] <ottomata>	 which makes them look like attachments to me
[19:07:42] <bsitu>	 AaronSchulz: thx, by the way, can I +2 and deploy this change https://gerrit.wikimedia.org/r/#/c/58731/?
[19:07:44] <aude>	 Reedy: ok, 'ip' is *right*
[19:07:46] <notpeter>	 AaronSchulz: woo!
[19:08:05] <AaronSchulz>	 bsitu: sure
[19:08:28] <icinga-wm>	 PROBLEM - Host db1015 is DOWN: PING CRITICAL - Packet loss = 100%
[19:08:33] <bsitu>	 AaronSchulz: thank you for your help, :)
[19:08:37] <LeslieCarr>	 not seeing those
[19:08:48] <LeslieCarr>	 maybe somehting funky with the analytics queue ?
[19:09:04] <LeslieCarr>	 i mean pages
[19:09:08] <icinga-wm>	 PROBLEM - NTP on db1014 is CRITICAL: NTP CRITICAL: No response from NTP server
[19:09:18] <icinga-wm>	 RECOVERY - SSH on db1015 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[19:09:28] <icinga-wm>	 RECOVERY - Host db1015 is UP: PING OK - Packet loss = 0%, RTA = 0.52 ms
[19:09:51] <gerrit-wm>	 New patchset: Hashar; "zuul: support cloning from a different branch" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58737
[19:09:51] <gerrit-wm>	 New patchset: Hashar; "zuul: in labs use the `labs` branch to install Zuul" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58738
[19:11:23] <AaronSchulz>	 notpeter: we can probably increase the number of jobs procs up again
[19:11:39] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58731
[19:11:44] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58736
[19:13:08] <icinga-wm>	 PROBLEM - Host kaulen is DOWN: PING CRITICAL - Packet loss = 100%
[19:13:45] <hashar>	 pooor kaulen
[19:14:10] <hashar>	 that is bugzilla iirc
[19:15:03] <Thehelpfulone>	 yep
[19:15:13] <notpeter>	 hashar: getting into console
[19:15:21] <hashar>	 \O/
[19:15:37] <Thehelpfulone>	 andre__, we should do that bugmail thing now ;)
[19:15:39] <logmsgbot>	 !log bsitu synchronized wmf-config/CommonSettings.php  'Add checkDelay to Echo email bundle jobs'
[19:15:45] <morebots>	 Logged the message, Master
[19:16:15] <gerrit-wm>	 New patchset: Aaron Schulz; "Set job dprioprocs back to 12 and iprioprocs to 6" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58739
[19:16:25] <siebrand>	 Bugzilla down, correct?
[19:16:35] <logmsgbot>	 !log bsitu synchronized wmf-config/InitialiseSettings.php  'Turn Echo back on for testwiki'
[19:16:37] <notpeter>	 siebrand: yep
[19:16:40] <notpeter>	 box unresponsive
[19:16:42] <morebots>	 Logged the message, Master
[19:16:42] <notpeter>	 logging in now
[19:16:49] <siebrand>	 oki. I'll announce on some dev channels.
[19:17:42] <siebrand>	 (done)
[19:18:08] <notpeter>	 ty
[19:18:11] <AaronSchulz>	 notpeter: https://gerrit.wikimedia.org/r/#/c/58739/
[19:19:15] <notpeter>	 siebrand: hashar it looks like a networking issue. hopefully not onboard nic....
[19:19:48] <notpeter>	 AaronSchulz: want to test? or are you feeling lucky? ;)
[19:20:14] <hashar>	 notpeter: are we monitoring the ports of our network switches
[19:20:20] <LeslieCarr>	 oh interesting
[19:20:27] <LeslieCarr>	 it's at 100mbit and also pegged otuput
[19:20:33] * AaronSchulz  hands notpeter a clover

[19:20:42] <LeslieCarr>	 hrm, it was set to 100mbit on purpose actually
[19:21:15] <notpeter>	 hashar: I don't believe so
[19:21:19] <notpeter>	 AaronSchulz: you got it!
[19:21:54] <hashar>	 and looking at nagios emery died too apparently
[19:22:07] <notpeter>	 AaronSchulz: ok, live
[19:22:08] <icinga-wm>	 RECOVERY - Host kaulen is UP: PING OK - Packet loss = 0%, RTA = 26.75 ms
[19:22:16] <notpeter>	 hashar: that's part of upgrades
[19:22:21] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58739
[19:22:39] <LeslieCarr>	 okay so now it's at gig and is only at 400mbit of traffic
[19:22:44] <LeslieCarr>	 weird
[19:22:53] <LeslieCarr>	 it was set to 100mbit a long time ago
[19:23:09] <LeslieCarr>	 must have been from an old version of kaulen
[19:23:13] <LeslieCarr>	 and just hit the limit
[19:23:27] <LeslieCarr>	 though now it's at 400mbit - unless anyone is doing anything, that is a big traffic increase
[19:23:28] <icinga-wm>	 PROBLEM - NTP on db1015 is CRITICAL: NTP CRITICAL: No response from NTP server
[19:23:34] <notpeter>	 LeslieCarr: is a hero! woo!
[19:23:45] <hashar>	 \O/
[19:23:48] <icinga-wm>	 PROBLEM - RAID on ms-be10 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:23:51] <LeslieCarr>	 someone should see why so much more traffic now thoguh
[19:24:08] <icinga-wm>	 RECOVERY - Disk space on search23 is OK: DISK OK
[19:24:11] <ottomata>	 yeah, hashar we are upgrading emery right now
[19:24:18] <icinga-wm>	 RECOVERY - RAID on search23 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0
[19:24:34] <hashar>	 and if one of our heroes could merge some change I made to the puppet Zuul module, that would be nice. They are simple change to use git::clone from a given branch :  https://gerrit.wikimedia.org/r/58737 https://gerrit.wikimedia.org/r/58738  =)
[19:24:38] <LeslieCarr>	 like a ton more traffic all of a sudden, we're talking orders of magnitude
[19:24:48] <gerrit-wm>	 New patchset: Dzahn; "use lucid installer on emery for testing because precise doesn't appear to work" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58740
[19:24:51] <hashar>	 LeslieCarr: maybe an upgrade?
[19:24:55] <LeslieCarr>	 andre__: can you look at what's up with bugzilla ?
[19:24:58] <icinga-wm>	 RECOVERY - DPKG on search23 is OK: All packages OK
[19:25:07] <LeslieCarr>	 no, though it needs some upgrades
[19:25:58] <LeslieCarr>	 so now it's also getting upgrades
[19:27:05] <gerrit-wm>	 New review: Dzahn; "testing if it's an issue with the precise image or unrelated" [operations/puppet] (production) C: 2;  - https://gerrit.wikimedia.org/r/58740
[19:27:05] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58740
[19:27:33] <LeslieCarr>	 the utilization has gone down to its normal level
[19:28:19] <hashar>	 LeslieCarr: thhhank you :-]
[19:29:05] <LeslieCarr>	 ahha, some sort of crashing -- has anyone seen http://pastebin.com/KuzWhC7S before ?
[19:30:28] <icinga-wm>	 RECOVERY - search indices - check lucene status page on search23 is OK: HTTP OK: HTTP/1.1 200 OK - 269 bytes in 0.056 second response time
[19:31:17] <mutante>	 !log running puppet on brewster, testing lucid image on emery, rebooting again
[19:31:18] <AaronSchulz>	 !log Refreshed the job queue aggregator hash table
[19:31:24] <morebots>	 Logged the message, Master
[19:31:30] <morebots>	 Logged the message, Master
[19:32:42] <gerrit-wm>	 New patchset: Bsitu; "Add new email job MWEchoNotificationEmailBundleJob to job-runner" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58742
[19:33:24] <hashar>	 LeslieCarr: usb keyboard and mouse got disconnected? :D
[19:33:37] <hashar>	 ah no must be the console
[19:33:41] <LeslieCarr>	 hehe
[19:34:02] <LeslieCarr>	 i've never seen that message before
[19:34:39] <hashar>	 check the webcam :D
[19:36:15] <cmjohnson1>	 hashar: Am I only installing the 1 ssd tomorrow or still going with 2?
[19:37:05] <hashar>	 cmjohnson1: I talked about it with mark this afternoon. Seems one SSD is enough.
[19:37:15] <hashar>	 !rt 4916
[19:37:16] <wm-bot>	 https://rt.wikimedia.org/Ticket/Display.html?id=4916
[19:37:21] <greg-g>	 hashar: the space issue wasn't big enough?
[19:37:35] <hashar>	 I filled some capacity data on the RT ticket (4916).  160GB is going to be enough
[19:37:44] <greg-g>	 cool
[19:37:49] <cmjohnson1>	 I followed along with your conversation but wasn't completely sure about the final decision
[19:37:50] <gerrit-wm>	 New patchset: Bsitu; "Disable VectorSectionEditLinks on testwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58743
[19:38:02] <hashar>	 greg-g: that is mostly I/O latency /  throughput :-]
[19:38:16] <hashar>	 cmjohnson1: let me confirm on the RT ticket that one is enough
[19:38:26] <cmjohnson1>	 thank you
[19:38:27] <icinga-wm>	 RECOVERY - Host emery is UP: PING WARNING - Packet loss = 61%, RTA = 26.48 ms
[19:38:43] <hashar>	 cmjohnson1: also mark said no  /a  :-]
[19:39:21] <gerrit-wm>	 Change merged: Kaldari; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58743
[19:40:34] <Thehelpfulone>	 kaldari, hmm for that change, can't you just remove those vectorsectioneditlinks lines altogether?
[19:40:39] <Jeff_Green>	 !log running authdns-update to propagate frdata cname
[19:40:39] <Thehelpfulone>	 or do you plan to re-enable in the future?
[19:40:46] <morebots>	 Logged the message, Master
[19:41:07] <icinga-wm>	 PROBLEM - udp2log log age for emery on emery is CRITICAL: Connection refused by host
[19:41:17] <icinga-wm>	 PROBLEM - SSH on emery is CRITICAL: Connection refused
[19:41:45] <kaldari>	 Thehelpfulone: yes, that would have been my way to do it, but we only have 20 minutes left so I'm not going to make benny redo it right now...
[19:42:08] <AaronSchulz>	 notpeter: seems to burning through lots of ChangeNotification jobs
[19:42:17] <AaronSchulz>	 memory footprint is dropping a lot now
[19:42:18] <kaldari>	 lest we get in trouble to going over our deployment window again :P
[19:42:35] <Thehelpfulone>	 heh, yeah that can be cleaned up afterwards
[19:42:51] <gerrit-wm>	 New patchset: Ori.livneh; "Exclude BLPs from GettingStarted tasks on enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58745
[19:43:54] <hashar>	 cmjohnson1: RT updated :-]
[19:44:11] <cmjohnson1>	 great thx
[19:45:34] <notpeter>	 AaronSchulz: woo!
[19:45:39] <notpeter>	 burn'em all!
[19:45:56] <logmsgbot>	 !log kaldari synchronized wmf-config/InitialiseSettings.php  'syncing InitialiseSettings to turn off VectorSectionEditLinks on testwiki'
[19:46:03] <morebots>	 Logged the message, Master
[19:49:13] <gerrit-wm>	 New review: Mattflaschen; "Use an InitialiseSettings gate, like wgOverrideSiteFeed." [operations/mediawiki-config] (master) C: -1;  - https://gerrit.wikimedia.org/r/58745
[19:50:17] <icinga-wm>	 RECOVERY - SSH on emery is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[19:50:32] <AaronSchulz>	 notpeter: though it also means lots of OOMs ;)
[19:50:38] * AaronSchulz  already has a bug for that

[19:51:38] <AaronSchulz>	 ugh, flood
[19:53:27] <icinga-wm>	 PROBLEM - NTP on emery is CRITICAL: NTP CRITICAL: No response from NTP server
[19:53:47] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[19:53:47] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[19:53:47] <icinga-wm>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[19:57:55] <mutante>	 !log emery - upgrading from lucid to precise
[19:58:02] <morebots>	 Logged the message, Master
[20:04:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[20:05:05] <Thehelpfulone>	 mutante, are we planning that upgrade for sodium too?
[20:08:49] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 20:08:41 UTC 2013
[20:09:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[20:09:49] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 20:09:45 UTC 2013
[20:10:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[20:10:49] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 20:10:42 UTC 2013
[20:11:06] <logmsgbot>	 !log kaldari Started syncing Wikimedia installation... :
[20:11:13] <morebots>	 Logged the message, Master
[20:11:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[20:11:39] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 20:11:32 UTC 2013
[20:12:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[20:12:29] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 20:12:18 UTC 2013
[20:12:33] <gerrit-wm>	 New patchset: Ori.livneh; "Exclude BLPs from GettingStarted tasks on enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58745
[20:13:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[20:13:29] <icinga-wm>	 RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 11 20:13:25 UTC 2013
[20:13:33] <gerrit-wm>	 New patchset: Dzahn; "switch emery back to default precise installer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58752
[20:14:19] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[20:14:30] <gerrit-wm>	 Change merged: Mattflaschen; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58745
[20:15:15] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58752
[20:16:41] <gerrit-wm>	 New patchset: Aude; "Reduce batch size for wikidata dispatcher" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58753
[20:17:02] <aude>	 AaronSchulz: https://gerrit.wikimedia.org/r/#/c/58753/
[20:17:19] <aude>	 we can try smaller batch size for the dispatcher and see if it helps
[20:17:33] <aude>	 only good for not-yet dispatched jobs
[20:19:30] <gerrit-wm>	 New review: Ori.livneh; "Should I abandon this patch? (I don't mind.)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116
[20:20:29] <icinga-wm>	 PROBLEM - Host emery is DOWN: PING CRITICAL - Packet loss = 100%
[20:21:29] <icinga-wm>	 PROBLEM - Host labstore3 is DOWN: PING CRITICAL - Packet loss = 100%
[20:21:49] <MaxSem>	 mark, looks like https://gerrit.wikimedia.org/r/#/c/32866/ is not live yet, likely due to varnish::common-vcl not causing a refresh
[20:26:18] <gerrit-wm>	 New review: Diederik; "I have no objection to the patch itself, but really would like not to install python-pandas, see my ..." [operations/puppet] (production) C: -1;  - https://gerrit.wikimedia.org/r/54116
[20:29:53] <AaronSchulz>	 aude: what is the size now? 1000?
[20:30:07] <aude>	 AaronSchulz: 1000
[20:30:10] <AaronSchulz>	 is that actually 1000 rows or 1000*X or something?
[20:30:21] <aude>	 1000 changes, so should be 1000 rows
[20:30:31] <AaronSchulz>	 there must be far more than 1000 rows, unless they are large text blobs
[20:30:32] <aude>	 smaller batch is probably good
[20:30:33] <icinga-wm>	 RECOVERY - Host emery is UP: PING OK - Packet loss = 0%, RTA = 26.45 ms
[20:30:39] <aude>	 they are blobs
[20:30:42] <aude>	 largish
[20:30:53] <Coren>	 Peeps, I'm having mega packet loss to WMF right now because the route to eqix-dc2.wikimediafound.com keeps flapping.  Something odd with our peers?
[20:30:59] <AaronSchulz>	 well that explains why it took a bit to copy into redis
[20:31:05] <aude>	 ah
[20:31:23] <aude>	 especially if the wiki gets phase 2 stuff, the change might be larger
[20:31:33] <AaronSchulz>	 that seems fine..maybe notpeter can merge it
[20:31:37] <aude>	 k
[20:31:53] <aude>	 see how 500 is, or we can adjust it more
[20:32:06] <logmsgbot>	 !log kaldari Finished syncing Wikimedia installation... :
[20:32:13] <morebots>	 Logged the message, Master
[20:32:25] <aude>	 also see https://gerrit.wikimedia.org/r/#/c/58708/ (if we're ready to move the cron jobs)
[20:32:33] <Coren>	 Ah, more to the point, my route to fenari keeps flapping between eqix-dc2 and xe-1-1-0.cr1-sdtpa(!)
[20:32:38] <aude>	 or whenever ready
[20:33:32] <gerrit-wm>	 New review: Ori.livneh; "What is the problem with the dependencies, exactly?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116
[20:34:19] <logmsgbot>	 !log mflaschen synchronized wmf-config  'Config change for GettingStarted'
[20:34:26] <morebots>	 Logged the message, Master
[20:36:52] <Nemo_bis>	 Coren: will bots in labs have possibility to choose between eqiad and pmtpa? seems silly to act on the cluster from another cluster, sometimes
[20:37:03] <bsitu>	 notpeter: ping
[20:37:30] <Coren>	 Nemo_bis: Not really.  pmtpa is going away; Labs is there only until it moves to eqiad.
[20:38:05] <Coren>	 Nemo_bis: (Which, admitedly, is not like "next week" or anything like it)
[20:38:45] <icinga-wm>	 PROBLEM - Host labstore4 is DOWN: PING CRITICAL - Packet loss = 100%
[20:38:55] <icinga-wm>	 PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours
[20:39:32] <superm401>	 kaldari, you guys have a merge commit on fenari that's not in Gerrit.
[20:39:37] <superm401>	 Not the end of the world, but better to avoid.
[20:40:02] <kaldari>	 ?
[20:40:04] <bsitu>	 superm401: what change is it?
[20:40:16] <superm401>	 49c6e9d3cd40a752560137360523f11b222b64bd
[20:40:27] <superm401>	 It's merging in 'Update Echo to master'
[20:40:39] <superm401>	 But it looks like you did the merge directly on fenari, not in Gerrit.
[20:41:04] <superm401>	 Not on https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=log;h=refs/heads/wmf/1.22wmf1
[20:42:22] <superm401>	 It looks like Niklas and/or Chris Steipp did the same first.
[20:43:15] <kaldari>	 I'm afraid I don't understand what you mean. what's abnormal?
[20:43:55] <superm401>	 kaldari, merges (including deployment merges) should be done through Gerrit (Publish and Submit).
[20:44:01] <superm401>	 Not manually on fenari.
[20:44:58] <kaldari>	 I don't understand how you would do a merge other than through gerrit
[20:45:03] <superm401>	 It may be they were left on fenari due to a security issue patch, in which disregard.
[20:45:23] <superm401>	 kaldari, it can happen if you do git pull instead of git pull --ff-only
[20:45:30] <superm401>	 git pull == "git fetch; git merge".
[20:45:58] <kaldari>	 the deployment instructions say to use 'git pull'
[20:46:12] <kaldari>	 https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Step_2:_get_the_code_on_fenari
[20:47:06] <chaos_generator>	 "git pull -r" might be fun, too
[20:48:21] <superm401>	 kaldari, true.  Anyway, apparently it was deliberately left as part of a security release, so don't worry about it.
[20:48:29] <greg-g>	 see the thread "[Engineering] Security patches on cluster" on engineer@
[20:48:42] <greg-g>	 that explains how to deal with those
[20:48:53] <greg-g>	 engineering@, that is (obviously)
[20:49:32] <kaldari>	 superm401, greg-g: sorry about that
[20:49:57] <kaldari>	 I hadn't read the email :(
[20:50:02] <gerrit-wm>	 New patchset: Demon; "Allow jenkins admins to manage replicated git repos" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58692
[20:50:07] <gerrit-wm>	 New review: Erosen; "I can understand a reluctance to install it on stat1001 (as it is not necessary for file hosting), b..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116
[20:51:29] <greg-g>	 kaldari: I'm actually doing an audit of the deploy documentation (on a very high level), so I'll note this as something to add to it, or just do it myself.
[20:51:30] <^demon>	 kaldari, greg-g, others: Fwiw, starting with 1.22wmf2 we'll be setting branch.autosetuprebase = always when cloning the repo.
[20:51:43] <^demon>	 So people can keep on being lazy and typing git pull, but not creating silly merge commits.
[20:51:49] <greg-g>	 oh, interesting
[20:52:22] <kaldari>	 nice
[20:52:34] <greg-g>	 btw, that means "starting monday" :)
[20:52:44] <logmsgbot>	 !log mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/  'Update GettingStarted extension for E3 deployment'
[20:52:50] <morebots>	 Logged the message, Master
[20:52:53] <superm401>	 Oh, okay, I was just about to ask whether we should do git pull --rebase.
[20:53:48] <hashar>	 never
[20:54:25] <hashar>	 git fetch,  then inspect with git log and finally do git rebase :D
[20:54:32] <hashar>	 you don't want to pull someone else change
[20:54:36] <hashar>	 (ok i am paranoid
[20:55:32] <superm401>	 hashar, yeah, I'll probably keep doing git pull --ff-only.  And then only resort to regular git pull if I see there is a security patch.
[20:55:54] <superm401>	 And the git log, which the instructions already say to do.
[20:55:57] <mutante>	 for the record: upgrading from lucid to precise fails   "This bug affects 46 people "
[20:55:58] <superm401>	 Anyway, E3 deployment is done.
[20:56:23] <mutante>	 "Please don't just comment on this bug or reopen it. It isn't helpful, there are too many different issues here, already." lol
[20:56:47] <mutante>	 that's what people pay them for..
[20:56:47] <^demon>	 hashar: I want https://gerrit.wikimedia.org/r/#/c/58692/ :)
[20:56:58] <superm401>	 We're aware of the GettingStarted downtime.  ori-l is fixing it.
[20:57:56] <icinga-wm>	 PROBLEM - DPKG on db1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[21:01:11] <logmsgbot>	 !log olivneh synchronized php-1.22wmf1/extensions/GettingStarted/SpecialGettingStarted.php
[21:01:19] <morebots>	 Logged the message, Master
[21:01:39] <superm401>	 E3 deployment issue is fixed.
[21:01:45] <ori-l>	 superm401, thanks
[21:02:32] <kaldari>	 are you guys done with your deployment then?
[21:02:47] <kaldari>	 no rush :)
[21:02:48] <ori-l>	 kaldari: one sec
[21:03:15] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[21:04:31] <logmsgbot>	 !log olivneh synchronized php-1.22wmf1/extensions/ArticleFeedback/ArticleFeedback.hooks.php  'Remove jquery.clickTracking from AFT dependencies'
[21:04:38] <morebots>	 Logged the message, Master
[21:04:41] <ori-l>	 kaldari: done
[21:06:48] <hashar>	 ^demon: yeah voted for it. Need someone to merge it :-]
[21:07:24] <hashar>	 ^demon: I found that adding in the commit summary some context and what the patch is fixing  helps a lot getting changes merged
[21:08:44] <kaldari>	 ori-l: thanks
[21:09:05] <icinga-wm>	 PROBLEM - Host db1001 is DOWN: PING CRITICAL - Packet loss = 100%
[21:09:13] <gerrit-wm>	 New patchset: Demon; "Allow jenkins admins to manage replicated git repos" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58692
[21:09:32] <^demon>	 hashar: Expanded the summary
[21:09:37] <mutante>	 ottomata: Ubuntu 12.04.2 LTS emery ttyS1
[21:09:44] <mutante>	 holy ... this was annoying
[21:09:45] <gerrit-wm>	 New review: Dsc; "I, for one, think python-pandas is a great library and think stat1 is exactly the sort of environmen..." [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/54116
[21:09:54] <hashar>	 ^demon: yeah much better
[21:09:58] <AaronSchulz>	 notpeter: getting a merge for https://gerrit.wikimedia.org/r/#/c/58753/ would be sweet
[21:10:16] <hashar>	 ^demon: also it sees fast forward only might not works well for ops/puppet .  They have to keep rebasing
[21:10:35] <^demon>	 That was changed back to Merge If Necessary
[21:10:36] <hashar>	 ^demon: I am wondering whether cherry pick would be suitable.
[21:10:58] <hashar>	 ah ok
[21:11:03] <hashar>	 works for them I guess
[21:12:21] <kaldari>	 greg-g, ^demon: just tried doing 'git pull --ff-only' on fenari and it says 'fatal: Not possible to fast-forward, aborting.'
[21:12:34] <greg-g>	 ^demon: that's you ;)
[21:12:59] <notpeter>	 AaronSchulz: sure
[21:13:02] <^demon>	 kaldari: Well --ff-only isn't going to work when there's security patches.
[21:13:02] <notpeter>	 bsitu: what's up?
[21:13:16] <kaldari>	 do I need to do a git rebase origin/wmf/1.22wmf1
[21:13:18] <^demon>	 And since there's divergent commits (silly merges), a pull --rebase won't work either.
[21:13:32] <^demon>	 You can try rebasing.
[21:13:35] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58753
[21:13:40] <mutante>	 !log emery - revoking old puppet cert, resiging, initial puppet run
[21:13:42] <^demon>	 If you can get it all rebased with the security patches on top that'd be awesome.
[21:13:49] <^demon>	 (I did that the other day)
[21:13:49] <morebots>	 Logged the message, Master
[21:14:05] <notpeter>	 AaronSchulz: live
[21:14:21] <gerrit-wm>	 New patchset: Mwalker; "Enable CentralNotice/Translate On Cluster" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58826
[21:15:12] <kaldari>	 "It looks like git-am is in progress. Cannot rebase." I don't even know what 'git am' is :P
[21:15:45] <greg-g>	 oh git
[21:15:47] <bsitu>	 notpeter: https://gerrit.wikimedia.org/r/#/c/58742/, I also sent an email to ops list
[21:16:08] <gerrit-wm>	 New patchset: Mwalker; "Enable CentralNotice/Translate On Cluster" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58826
[21:16:41] <kaldari>	 ^demon: ^
[21:17:27] <kaldari>	 I just need to pull the Echo extension update into 1.22wmf1. If you want to handle it, I wouldn't object ;)
[21:18:11] <superm401>	 git am is used for applying patches.
[21:18:26] <superm401>	 Someone may have left in the middle of doing that.
[21:19:35] <icinga-wm>	 RECOVERY - Host db1001 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms
[21:20:08] <LeslieCarr>	 cmjohnson1: are you there ?
[21:20:33] <cmjohnson1>	 hi lesliecarr
[21:20:35] <cmjohnson1>	 whats up
[21:20:41] <LeslieCarr>	 are you int he dc right now ?
[21:20:58] <cmjohnson1>	 no...left about 30 mins ago
[21:20:59] <gerrit-wm>	 New patchset: Hashar; "zuul: prevent fetching from pypi" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58827
[21:21:04] <LeslieCarr>	 ok no problem
[21:21:08] <cmjohnson1>	 i can go back in a little bit
[21:21:24] <notpeter>	 bsitu: cool. get AaronSchulz to +1 it and I'll happily merge and deploy
[21:21:27] <notpeter>	 :)
[21:21:55] <icinga-wm>	 PROBLEM - RAID on db1001 is CRITICAL: Connection refused by host
[21:21:55] <icinga-wm>	 PROBLEM - Disk space on db1001 is CRITICAL: Connection refused by host
[21:21:55] <icinga-wm>	 PROBLEM - SSH on db1001 is CRITICAL: Connection refused
[21:21:55] <icinga-wm>	 PROBLEM - MySQL disk space on db1001 is CRITICAL: Connection refused by host
[21:22:21] <notpeter>	 db1001 is also me
[21:22:25] <icinga-wm>	 PROBLEM - MySQL Recent Restart on db1001 is CRITICAL: Connection refused by host
[21:22:27] <cmjohnson1>	 lesliecarr: do you need something...i will have time later to go back
[21:22:50] <bsitu>	 AaronSchulz: ^^
[21:22:53] <^demon>	 kaldari: I'll take a look.
[21:22:58] <AaronSchulz>	 already done
[21:23:02] <kaldari>	 ^demon: thank you!!
[21:23:07] <notpeter>	 AaronSchulz: thanks!
[21:23:14] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58742
[21:23:45] <kaldari>	 ^demon: I'm not much of a git cowboy and I don't want to screw things up on fenari
[21:23:54] <notpeter>	 bsitu: AaronSchulz ok, live on sockpuppet
[21:24:13] <AaronSchulz>	 I assume the runners were restarted?
[21:24:18] <bsitu>	 notpeter, AaronSchulz: thx
[21:24:35] <AaronSchulz>	 does that happen automatically periodically?
[21:25:30] <notpeter>	 AaronSchulz: the service is subscribed to the file
[21:25:38] <notpeter>	 so they will all be restarted in the next 30 minutes
[21:25:51] <AaronSchulz>	 ok, I was wondering about how the subscription was handled
[21:25:55] <AaronSchulz>	 good
[21:26:51] <^demon>	 The heck...
[21:26:59] <^demon>	 git status ain't supposed to be that slow
[21:27:53] <mutante>	 ottomata: pheew.. so .. it's precise, puppet runs, the users have been created, the packages have been installed, no errors on puppet run (but it was hell getting there;)
[21:27:59] <mutante>	 notice: Finished catalog run in 182.83 seconds
[21:28:38] <drdee>	 awesome mutante! thank you veryyyyyyy muuuuuuch!
[21:28:46] <bsitu>	 notpeter: The job runner will pick up the new job in 30 minutes?
[21:28:55] <icinga-wm>	 RECOVERY - SSH on db1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[21:29:13] <notpeter>	 bsitu: should, yes
[21:29:19] <notpeter>	 well, in at most 30 minutes
[21:29:24] <bsitu>	 notpeter: cool, thank you
[21:30:23] <mutante>	 drdee: :) (it was such a pita, glad it's done;)
[21:30:30] <LeslieCarr>	 cmjohnson1: so i do need to get 2 sfp's swapped but since it's been dead a while there's no need to go back tonight
[21:31:15] <gerrit-wm>	 New patchset: Hashar; "zuul: no fetch from pypi and drop statsd dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58827
[21:31:36] <^demon>	 !log fenari: php-1.21wmf12 and php-1.22wmf1 both rebased to remove silly merge commits.
[21:31:42] <morebots>	 Logged the message, Master
[21:31:47] <cmjohnson1>	 lesliecarr: can you give me a ticket and i will get it done
[21:32:01] <^demon>	 kaldari: ^
[21:32:04] <LeslieCarr>	 will do
[21:32:22] <kaldari>	 ^demon: yay
[21:32:30] <kaldari>	 I'll see if I can finish up now...
[21:32:55] <icinga-wm>	 RECOVERY - NTP on db1015 is OK: NTP OK: Offset -0.007087230682 secs
[21:33:05] <icinga-wm>	 RECOVERY - NTP on db1014 is OK: NTP OK: Offset -0.002544999123 secs
[21:33:11] <^demon>	 kaldari: So, if git complains about being in the middle of a rebase or an am, but you're not actually in the middle of it, `rm -R .git/rebase-apply/`
[21:33:20] <^demon>	 Then you can rebase or do other fun stuff again.
[21:33:24] <kaldari>	 thanks
[21:34:14] <icinga-wm>	 PROBLEM - NTP on db1001 is CRITICAL: NTP CRITICAL: No response from NTP server
[21:35:13] <gerrit-wm>	 New patchset: Ottomata; "Fixing stat1 rsync udp2log module path" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58828
[21:35:27] <ottomata>	 !log emery back up and upgraded to precise (thanks mutante!)
[21:35:33] <morebots>	 Logged the message, Master
[21:35:59] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58828
[21:37:38] <LeslieCarr>	 dude, emery and precise did not get along
[21:38:17] <Ryan_Lane>	 heh
[21:38:20] <Ryan_Lane>	 why not?
[21:39:19] <Amgine_>	 emery isn't detail oriented.</pun>
[21:39:30] <LeslieCarr>	 not sure honestly , the precise installer image wouldn't boot on it
[21:39:35] <gerrit-wm>	 New patchset: Hashar; "zuul: no fetch from pypi and drop statsd dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58827
[21:39:37] <LeslieCarr>	 the normal do-release-upgrade failed miserably
[21:40:40] <gerrit-wm>	 New patchset: Ottomata; "Monitoring the udp2log instance on analytics1008" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58829
[21:40:53] <ottomata>	 yeah
[21:40:59] <logmsgbot>	 !log kaldari synchronized php-1.22wmf1/extensions/Echo  'syncing Echo for bugfix'
[21:41:01] <ottomata>	 i dunno what mutante just went through to get it to precise
[21:41:06] <morebots>	 Logged the message, Master
[21:41:07] <ottomata>	 but it took 3.5 hours
[21:41:11] <ottomata>	 to get it there
[21:41:53] <gerrit-wm>	 New patchset: Ottomata; "Monitoring the udp2log instance on analytics1008" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58829
[21:42:23] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58829
[21:43:21] <gerrit-wm>	 New patchset: Ottomata; "Removing location udp-filter on emery.  It is no longer needed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58831
[21:43:32] <gerrit-wm>	 Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58831
[21:47:35] <gerrit-wm>	 New patchset: Bsitu; "Set bundle time interval for test2wiki and mediawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58832
[21:56:04] <icinga-wm>	 RECOVERY - NTP on db1001 is OK: NTP OK: Offset -0.0145919323 secs
[22:03:38] <icinga-wm>	 PROBLEM - DPKG on cp1042 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:03:52] <gerrit-wm>	 Change merged: Adamw; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58826
[22:03:58] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[22:05:08] <icinga-wm>	 PROBLEM - DPKG on cp1041 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:05:18] <icinga-wm>	 RECOVERY - DPKG on cp1042 is OK: All packages OK
[22:05:59] <icinga-wm>	 RECOVERY - DPKG on cp1041 is OK: All packages OK
[22:07:59] <icinga-wm>	 PROBLEM - Varnish HTTP mobile-backend on cp1041 is CRITICAL: Connection refused
[22:08:26] <gerrit-wm>	 New patchset: Pyoungmeister; "adding db1016 as eqiad m1 snapshot host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58835
[22:09:36] <notpeter>	 db1016 is me restarting too
[22:11:38] <icinga-wm>	 PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours
[22:11:38] <icinga-wm>	 PROBLEM - Host db1016 is DOWN: PING CRITICAL - Packet loss = 100%
[22:12:28] <gerrit-wm>	 New patchset: Pyoungmeister; "adding db1016 as eqiad m1 snapshot host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58835
[22:13:48] <icinga-wm>	 PROBLEM - DPKG on mw1005 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:13:58] <icinga-wm>	 PROBLEM - DPKG on mw1004 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:13:58] <icinga-wm>	 PROBLEM - DPKG on mw1009 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:13:59] <icinga-wm>	 PROBLEM - DPKG on mw1008 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:13:59] <icinga-wm>	 PROBLEM - DPKG on mw1007 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:14:08] <icinga-wm>	 PROBLEM - DPKG on mw1003 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:14:08] <icinga-wm>	 PROBLEM - Apache HTTP on mw100 is CRITICAL: Connection refused
[22:14:22] <icinga-wm>	 PROBLEM - DPKG on mw1006 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:14:28] <icinga-wm>	 PROBLEM - DPKG on mw1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:16:08] <icinga-wm>	 RECOVERY - DPKG on mw1003 is OK: All packages OK
[22:16:18] <icinga-wm>	 RECOVERY - DPKG on mw1006 is OK: All packages OK
[22:16:28] <icinga-wm>	 RECOVERY - DPKG on mw1001 is OK: All packages OK
[22:16:45] <LeslieCarr>	 sorry, that was me upgrading some packages
[22:16:48] <icinga-wm>	 RECOVERY - Host db1016 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms
[22:16:48] <icinga-wm>	 RECOVERY - DPKG on mw1005 is OK: All packages OK
[22:16:54] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58835
[22:16:58] <icinga-wm>	 RECOVERY - DPKG on mw1004 is OK: All packages OK
[22:16:58] <icinga-wm>	 RECOVERY - DPKG on mw1009 is OK: All packages OK
[22:16:58] <icinga-wm>	 RECOVERY - DPKG on mw1008 is OK: All packages OK
[22:16:58] <icinga-wm>	 RECOVERY - DPKG on mw1007 is OK: All packages OK
[22:17:08] <icinga-wm>	 RECOVERY - Apache HTTP on mw100 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.146 second response time
[22:17:23] <notpeter>	 LeslieCarr: ok, cool. I will not worry then :)
[22:18:58] <icinga-wm>	 PROBLEM - DPKG on db1016 is CRITICAL: Connection refused by host
[22:18:58] <icinga-wm>	 PROBLEM - Disk space on db1016 is CRITICAL: Connection refused by host
[22:19:08] <icinga-wm>	 PROBLEM - SSH on db1016 is CRITICAL: Connection refused
[22:19:18] <icinga-wm>	 PROBLEM - RAID on db1016 is CRITICAL: Connection refused by host
[22:21:48] <LeslieCarr>	 !log upgrading mw1110 to mw1119
[22:21:55] <morebots>	 Logged the message, Mistress of the network gear.
[22:25:08] <icinga-wm>	 PROBLEM - DPKG on mw1011 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:08] <icinga-wm>	 PROBLEM - DPKG on mw1013 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:18] <icinga-wm>	 PROBLEM - Apache HTTP on mw101 is CRITICAL: Connection refused
[22:25:18] <icinga-wm>	 PROBLEM - DPKG on mw1019 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:18] <icinga-wm>	 PROBLEM - DPKG on mw1014 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:58] <icinga-wm>	 PROBLEM - DPKG on mw1015 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:58] <icinga-wm>	 PROBLEM - DPKG on mw1017 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:58] <icinga-wm>	 PROBLEM - DPKG on mw1012 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:58] <icinga-wm>	 PROBLEM - DPKG on mw1018 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:25:59] <icinga-wm>	 PROBLEM - DPKG on mw1016 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:26:08] <icinga-wm>	 RECOVERY - SSH on db1016 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[22:26:18] <icinga-wm>	 RECOVERY - Apache HTTP on mw101 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.153 second response time
[22:26:21] <logmsgbot>	 !log mwalker synchronized php-1.22wmf1/extensions/CentralNotice  'Updating CentralNotice to the latest and greatest (from 68f0262be838320ecc9c87eb4485c28b0699bc1f)'
[22:26:28] <morebots>	 Logged the message, Master
[22:27:08] <icinga-wm>	 RECOVERY - DPKG on mw1011 is OK: All packages OK
[22:27:08] <icinga-wm>	 RECOVERY - DPKG on mw1013 is OK: All packages OK
[22:27:36] <mwalker>	 RobH: as rt duty person; I should probably inform you that in my most recent (and first ever) sync-dir; I got a permission denied on spence
[22:27:58] <icinga-wm>	 RECOVERY - DPKG on mw1015 is OK: All packages OK
[22:27:58] <icinga-wm>	 RECOVERY - DPKG on mw1012 is OK: All packages OK
[22:27:59] <icinga-wm>	 RECOVERY - DPKG on mw1016 is OK: All packages OK
[22:28:18] <icinga-wm>	 RECOVERY - DPKG on mw1014 is OK: All packages OK
[22:28:48] <icinga-wm>	 PROBLEM - Apache HTTP on mw1017 is CRITICAL: Connection refused
[22:28:59] <icinga-wm>	 RECOVERY - Varnish HTTP mobile-backend on cp1041 is OK: HTTP OK: HTTP/1.1 200 OK - 632 bytes in 0.007 second response time
[22:29:08] <icinga-wm>	 PROBLEM - Apache HTTP on mw1018 is CRITICAL: Connection refused
[22:29:18] <icinga-wm>	 PROBLEM - Apache HTTP on mw1019 is CRITICAL: Connection refused
[22:29:18] <icinga-wm>	 RECOVERY - DPKG on mw1019 is OK: All packages OK
[22:29:58] <icinga-wm>	 RECOVERY - DPKG on mw1017 is OK: All packages OK
[22:29:59] <icinga-wm>	 RECOVERY - DPKG on mw1018 is OK: All packages OK
[22:31:01] <RobH>	 mwalker: hrmm, for a .git directory or something else?
[22:31:08] <icinga-wm>	 PROBLEM - NTP on db1016 is CRITICAL: NTP CRITICAL: Offset unknown
[22:31:18] <mwalker>	 dunno -- this was for extensions/CentralNotice
[22:31:27] <mwalker>	 oh
[22:31:32] <mwalker>	 it was an SSH connection error
[22:31:36] <mwalker>	 my public key was denied
[22:32:26] <RobH>	 what server?
[22:32:30] <RobH>	 oh, fenari, you said
[22:32:38] <RobH>	 ....didnt you deploy from fenari?
[22:32:46] <logmsgbot>	 !log mwalker synchronized php-1.21wmf12/extensions/CentralNotice
[22:32:52] <notpeter>	 spence
[22:32:53] <morebots>	 Logged the message, Master
[22:32:54] <notpeter>	 he said spence
[22:33:01] <mwalker>	 no no; deploying from fenari
[22:33:03] <RobH>	 oh, sorry, i see
[22:33:08] <icinga-wm>	 RECOVERY - NTP on db1016 is OK: NTP OK: Offset 0.1130144596 secs
[22:33:08] <icinga-wm>	 RECOVERY - Apache HTTP on mw1018 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time
[22:33:10] <mwalker>	 I get "spence: Permission denied (publickey)"
[22:33:12] <RobH>	 mwalker: spence is dying.
[22:33:18] <icinga-wm>	 RECOVERY - Apache HTTP on mw1019 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.065 second response time
[22:33:22] <RobH>	 it doesnt puppet update and is sticking around only ... well, i dunno why its still around
[22:33:27] <RobH>	 LeslieCarr: ^?
[22:33:41] <RobH>	 mwalker: but yea, its not doing anything but old nagios instance, so you should be ok
[22:33:41] <LeslieCarr>	 oh because of ishmael
[22:33:45] <LeslieCarr>	 stupid stupid ishmael
[22:33:49] <LeslieCarr>	 need to get that shit puppetized
[22:33:54] <RobH>	 does it need deployment updates?
[22:33:54] <gerrit-wm>	 Change merged: Mwalker; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58463
[22:33:57] <icinga-wm>	 RECOVERY - Apache HTTP on mw1017 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.056 second response time
[22:34:02] <RobH>	 ie: do we need to fix mwalker's access on it?
[22:34:47] <mutante>	 re: problems with spence and deployments:
[22:34:49] <mutante>	 root@fenari:~# grep spence /etc/dsh/group/*
[22:35:06] <mutante>	 gotta remove it there when if really dead
[22:36:14] <LeslieCarr>	 doesn't need it
[22:36:39] <RobH>	 so we can pull from the dsh groups for mediawiki deployment
[22:36:45] <RobH>	 as mutante states that will clean up possible issues
[22:36:55] <RobH>	 unless someone disagrees?
[22:37:05] <mutante>	 !log removing spence from dsh group mediawiki-installation
[22:37:12] <morebots>	 Logged the message, Master
[22:37:28] <mutante>	 it's still in ALL and pmtpa.. hmm and  mw-pmtpa
[22:38:00] <mutante>	 but maybe we need to add neon.. afair it was added for a monitoring reason
[22:38:17] <LeslieCarr>	 !log upgrading mw1120 to mw1129
[22:38:19] <mutante>	 i think job_queue ..
[22:38:24] <morebots>	 Logged the message, Mistress of the network gear.
[22:38:25] <LeslieCarr>	 um, i made them move that
[22:38:26] <LeslieCarr>	 i htink to hume
[22:38:45] <mutante>	 ah,ok nice
[22:38:55] <logmsgbot>	 !log mwalker synchronized docroot/bits/favicon/wikipedia.ico  'Bi resolution wiki favicon'
[22:39:02] <morebots>	 Logged the message, Master
[22:40:41] <logmsgbot>	 !log mwalker synchronized wmf-config/CommonSettings.php  'Enable translate on CentralNotice'
[22:40:48] <morebots>	 Logged the message, Master
[22:41:57] <icinga-wm>	 PROBLEM - DPKG on mw1026 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:41:57] <icinga-wm>	 PROBLEM - DPKG on mw1024 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:41:57] <icinga-wm>	 PROBLEM - DPKG on mw1020 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:41:58] <icinga-wm>	 PROBLEM - DPKG on mw1021 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:41:58] <icinga-wm>	 PROBLEM - DPKG on mw1027 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:42:17] <icinga-wm>	 PROBLEM - DPKG on mw1022 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:42:45] <mutante>	 takes a look at that
[22:42:57] <icinga-wm>	 PROBLEM - DPKG on mw1028 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[22:44:17] <icinga-wm>	 RECOVERY - DPKG on mw1022 is OK: All packages OK
[22:44:25] <mutante>	 ah, Leslie upgrading sprint:)
[22:44:32] <LeslieCarr>	 hehehe
[22:44:33] <LeslieCarr>	 yep
[22:44:39] <LeslieCarr>	 dpkg alerts quickly !
[22:44:57] <icinga-wm>	 PROBLEM - Apache HTTP on mw1020 is CRITICAL: Connection refused
[22:44:57] <icinga-wm>	 RECOVERY - DPKG on mw1026 is OK: All packages OK
[22:44:57] <icinga-wm>	 RECOVERY - DPKG on mw1028 is OK: All packages OK
[22:44:57] <icinga-wm>	 PROBLEM - Apache HTTP on mw1026 is CRITICAL: Connection refused
[22:44:57] <icinga-wm>	 RECOVERY - DPKG on mw1024 is OK: All packages OK
[22:44:57] <icinga-wm>	 RECOVERY - DPKG on mw1020 is OK: All packages OK
[22:44:58] <icinga-wm>	 PROBLEM - Apache HTTP on mw1021 is CRITICAL: Connection refused
[22:44:58] <icinga-wm>	 RECOVERY - DPKG on mw1021 is OK: All packages OK
[22:44:58] <icinga-wm>	 RECOVERY - DPKG on mw1027 is OK: All packages OK
[22:45:07] <icinga-wm>	 PROBLEM - Apache HTTP on mw1027 is CRITICAL: Connection refused
[22:45:07] <icinga-wm>	 PROBLEM - Apache HTTP on mw1024 is CRITICAL: Connection refused
[22:45:07] <icinga-wm>	 PROBLEM - Apache HTTP on mw1022 is CRITICAL: Connection refused
[22:45:17] <icinga-wm>	 PROBLEM - Apache HTTP on mw1028 is CRITICAL: Connection refused
[22:47:17] <logmsgbot>	 !log mwalker synchronized php-1.22wmf1/extensions/CentralAuth  'Peter's log line change to CentralAuth'
[22:47:24] <morebots>	 Logged the message, Master
[22:50:07] <icinga-wm>	 RECOVERY - Apache HTTP on mw1021 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.853 second response time
[22:50:07] <icinga-wm>	 RECOVERY - Apache HTTP on mw1027 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time
[22:50:07] <icinga-wm>	 RECOVERY - Apache HTTP on mw1024 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.084 second response time
[22:50:07] <icinga-wm>	 RECOVERY - Apache HTTP on mw1022 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.055 second response time
[22:50:17] <icinga-wm>	 RECOVERY - Apache HTTP on mw1028 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.062 second response time
[22:50:58] <icinga-wm>	 RECOVERY - Apache HTTP on mw1020 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.073 second response time
[22:50:58] <icinga-wm>	 RECOVERY - Apache HTTP on mw1026 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time
[22:59:46] <LeslieCarr>	 !log upgrading mw1030 - 1039
[22:59:51] <LeslieCarr>	 (maybe i should have said "breaking"
[22:59:52] <LeslieCarr>	 :)
[22:59:53] <morebots>	 Logged the message, Mistress of the network gear.
[23:00:31] <mutante>	 upbreaking
[23:00:41] <LeslieCarr>	 i like that
[23:00:44] <LeslieCarr>	 upbreaking :)
[23:01:36] <mutante>	 do-release-upbreak -p
[23:01:38] <notpeter>	 LeslieCarr: please be careful when upgrading mw11(49|5[0-2]), yeah?
[23:01:43] <LeslieCarr>	 ok
[23:01:45] <notpeter>	 like, 2 at a time
[23:01:50] <notpeter>	 thanks!
[23:02:02] <LeslieCarr>	 cool
[23:02:06] <LeslieCarr>	 so all at once ;)
[23:02:16] <notpeter>	 sure, we can stress test our bits setup :)
[23:03:02] <LeslieCarr>	 yay stress test!
[23:03:38] <mutante>	 !log install package upgrades on neon
[23:03:38] <icinga-wm>	 PROBLEM - DPKG on mw1032 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:03:40] <morebots>	 Logged the message, Master
[23:03:57] <icinga-wm>	 PROBLEM - DPKG on mw1036 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:03:57] <icinga-wm>	 PROBLEM - DPKG on mw1039 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:03:57] <icinga-wm>	 PROBLEM - DPKG on mw1030 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:03:57] <icinga-wm>	 PROBLEM - Apache HTTP on mw103 is CRITICAL: Connection refused
[23:04:07] <icinga-wm>	 PROBLEM - DPKG on mw1037 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:04:07] <icinga-wm>	 PROBLEM - DPKG on mw1033 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:04:17] <icinga-wm>	 PROBLEM - DPKG on mw1035 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:04:29] <notpeter>	 LeslieCarr: here, let me attach this bs device to your arm and I'll tell you that you're stressed and need to give all your money to some crazy cult
[23:04:47] <LeslieCarr>	 awesome, do i need to clear my thetans ?
[23:05:06] <notpeter>	 totally.
[23:05:15] <greg-g>	 everything I know about thetans I learned from South Park
[23:05:15] <mutante>	 reboots xenu
[23:05:29] <notpeter>	 oh, wait,  ithought you said "free my tibetans"
[23:06:00] <icinga-wm>	 RECOVERY - DPKG on mw1037 is OK: All packages OK
[23:06:10] <icinga-wm>	 PROBLEM - Apache HTTP on mw1035 is CRITICAL: Connection refused
[23:06:10] <icinga-wm>	 RECOVERY - DPKG on mw1039 is OK: All packages OK
[23:06:10] <icinga-wm>	 PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours
[23:06:10] <icinga-wm>	 RECOVERY - DPKG on mw1033 is OK: All packages OK
[23:06:10] <icinga-wm>	 RECOVERY - DPKG on mw1035 is OK: All packages OK
[23:06:20] <icinga-wm>	 PROBLEM - Apache HTTP on mw1039 is CRITICAL: Connection refused
[23:06:30] <icinga-wm>	 RECOVERY - DPKG on mw1032 is OK: All packages OK
[23:06:40] <icinga-wm>	 RECOVERY - DPKG on mw1036 is OK: All packages OK
[23:07:00] <icinga-wm>	 RECOVERY - DPKG on mw1030 is OK: All packages OK
[23:07:00] <icinga-wm>	 RECOVERY - Apache HTTP on mw103 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.144 second response time
[23:07:10] <icinga-wm>	 RECOVERY - Apache HTTP on mw1035 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.092 second response time
[23:07:20] <icinga-wm>	 RECOVERY - Apache HTTP on mw1039 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time
[23:08:10] <icinga-wm>	 PROBLEM - MySQL Recent Restart Port 3306 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_recent_restart_3306 not defined
[23:08:10] <icinga-wm>	 PROBLEM - MySQL Slave Running Port 3306 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_slave_running_3306 not defined
[23:08:10] <icinga-wm>	 PROBLEM - mysqld processes on labsdb1002 is CRITICAL: NRPE: Command check_mysqld not defined
[23:08:20] <icinga-wm>	 PROBLEM - MySQL Recent Restart Port 3307 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_recent_restart_3307 not defined
[23:08:20] <icinga-wm>	 PROBLEM - MySQL Slave Running Port 3307 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_slave_running_3307 not defined
[23:08:20] <icinga-wm>	 PROBLEM - DPKG on labsdb1002 is CRITICAL: NRPE: Command check_dpkg not defined
[23:08:30] <icinga-wm>	 PROBLEM - MySQL Recent Restart Port 3308 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_recent_restart_3308 not defined
[23:08:30] <icinga-wm>	 PROBLEM - MySQL Slave Running Port 3308 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_slave_running_3308 not defined
[23:08:30] <icinga-wm>	 PROBLEM - Disk space on labsdb1002 is CRITICAL: NRPE: Command check_disk_space not defined
[23:08:40] <icinga-wm>	 PROBLEM - MySQL Slave Delay Port 3306 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_slave_delay_3306 not defined
[23:08:40] <icinga-wm>	 PROBLEM - MySQL Idle Transactions Port 3306 on labsdb1002 is CRITICAL: NRPE: Command check_mysql_idle_transaction_3306 not defined
[23:08:50] <icinga-wm>	 PROBLEM - MySQL Idle Transactions Port 3307 on labsdb1002 is CRITICAL: Connection refused by host
[23:08:50] <icinga-wm>	 PROBLEM - RAID on labsdb1002 is CRITICAL: Connection refused by host
[23:08:50] <icinga-wm>	 PROBLEM - MySQL Slave Delay Port 3307 on labsdb1002 is CRITICAL: Connection refused by host
[23:09:00] <icinga-wm>	 PROBLEM - MySQL Slave Delay Port 3308 on labsdb1002 is CRITICAL: Connection refused by host
[23:09:06] <icinga-wm>	 PROBLEM - MySQL Idle Transactions Port 3308 on labsdb1002 is CRITICAL: Connection refused by host
[23:09:10] <icinga-wm>	 RECOVERY - MySQL Recent Restart Port 3306 on labsdb1002 is OK: OK seconds since restart
[23:09:10] <icinga-wm>	 RECOVERY - MySQL Slave Running Port 3306 on labsdb1002 is OK: OK replication
[23:09:20] <icinga-wm>	 RECOVERY - MySQL Slave Running Port 3307 on labsdb1002 is OK: OK replication
[23:09:20] <icinga-wm>	 RECOVERY - MySQL Recent Restart Port 3307 on labsdb1002 is OK: OK seconds since restart
[23:09:20] <icinga-wm>	 RECOVERY - DPKG on labsdb1002 is OK: All packages OK
[23:09:30] <icinga-wm>	 RECOVERY - MySQL Recent Restart Port 3308 on labsdb1002 is OK: OK seconds since restart
[23:09:30] <icinga-wm>	 RECOVERY - Disk space on labsdb1002 is OK: DISK OK
[23:09:30] <icinga-wm>	 RECOVERY - MySQL Slave Running Port 3308 on labsdb1002 is OK: OK replication
[23:09:40] <icinga-wm>	 RECOVERY - MySQL Slave Delay Port 3306 on labsdb1002 is OK: OK replication delay seconds
[23:09:40] <icinga-wm>	 RECOVERY - MySQL Idle Transactions Port 3306 on labsdb1002 is OK: OK longest blocking idle transaction sleeps for seconds
[23:09:50] <icinga-wm>	 RECOVERY - MySQL Idle Transactions Port 3307 on labsdb1002 is OK: OK longest blocking idle transaction sleeps for seconds
[23:09:50] <icinga-wm>	 RECOVERY - MySQL Slave Delay Port 3307 on labsdb1002 is OK: OK replication delay seconds
[23:10:00] <icinga-wm>	 RECOVERY - MySQL Slave Delay Port 3308 on labsdb1002 is OK: OK replication delay seconds
[23:10:00] <icinga-wm>	 RECOVERY - MySQL Idle Transactions Port 3308 on labsdb1002 is OK: OK longest blocking idle transaction sleeps for seconds
[23:11:27] <LeslieCarr>	 !log upgrading 1040 to 1049
[23:11:34] <morebots>	 Logged the message, Mistress of the network gear.
[23:15:00] <icinga-wm>	 PROBLEM - DPKG on mw1048 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:15:00] <icinga-wm>	 PROBLEM - DPKG on mw1047 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:15:00] <icinga-wm>	 PROBLEM - DPKG on mw1040 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:15:10] <icinga-wm>	 PROBLEM - DPKG on mw1041 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:15:10] <icinga-wm>	 PROBLEM - DPKG on mw1043 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:15:10] <icinga-wm>	 PROBLEM - DPKG on mw1044 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:15:20] <icinga-wm>	 PROBLEM - DPKG on mw1042 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:15:30] <icinga-wm>	 PROBLEM - DPKG on mw1049 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:17:10] <icinga-wm>	 RECOVERY - DPKG on mw1044 is OK: All packages OK
[23:17:30] <icinga-wm>	 RECOVERY - DPKG on mw1049 is OK: All packages OK
[23:18:00] <icinga-wm>	 PROBLEM - Apache HTTP on mw1043 is CRITICAL: Connection refused
[23:18:00] <icinga-wm>	 PROBLEM - Apache HTTP on mw1041 is CRITICAL: Connection refused
[23:18:00] <icinga-wm>	 RECOVERY - DPKG on mw1048 is OK: All packages OK
[23:18:00] <icinga-wm>	 RECOVERY - DPKG on mw1047 is OK: All packages OK
[23:18:00] <icinga-wm>	 RECOVERY - DPKG on mw1040 is OK: All packages OK
[23:18:10] <icinga-wm>	 RECOVERY - DPKG on mw1041 is OK: All packages OK
[23:18:10] <icinga-wm>	 RECOVERY - DPKG on mw1043 is OK: All packages OK
[23:18:20] <icinga-wm>	 RECOVERY - DPKG on mw1042 is OK: All packages OK
[23:19:00] <icinga-wm>	 RECOVERY - Apache HTTP on mw1043 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.045 second response time
[23:19:00] <icinga-wm>	 RECOVERY - Apache HTTP on mw1041 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.048 second response time
[23:19:47] <LeslieCarr>	 !log upgrading mw1050
[23:19:53] <morebots>	 Logged the message, Mistress of the network gear.
[23:22:21] <mutante>	 !log shutting down xenon
[23:22:28] <morebots>	 Logged the message, Master
[23:23:20] <icinga-wm>	 PROBLEM - DPKG on mw1050 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:23:33] <logmsgbot>	 !log csteipp synchronized php-1.22wmf1/includes/
[23:23:41] <morebots>	 Logged the message, Master
[23:24:00] <icinga-wm>	 PROBLEM - Host xenon is DOWN: PING CRITICAL - Packet loss = 100%
[23:24:40] <icinga-wm>	 ACKNOWLEDGEMENT - Host xenon is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn used to be parsoid, now spare and decom
[23:25:50] <icinga-wm>	 PROBLEM - Apache HTTP on mw1050 is CRITICAL: Connection refused
[23:26:20] <icinga-wm>	 RECOVERY - DPKG on mw1050 is OK: All packages OK
[23:26:50] <icinga-wm>	 RECOVERY - Apache HTTP on mw1050 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.045 second response time
[23:29:49] <LeslieCarr>	 !log upgrading mw1051
[23:29:56] <morebots>	 Logged the message, Mistress of the network gear.
[23:30:22] <logmsgbot>	 !log csteipp synchronized php-1.22wmf1/extensions/RSS/
[23:30:29] <morebots>	 Logged the message, Master
[23:31:58] <csteipp>	 Uh, is enwiki unreachable for anyone else? (hopefully it's just me?)
[23:32:12] <LeslieCarr>	 what kind of unreachaable
[23:32:17] <LeslieCarr>	 timeout 404 ?
[23:32:26] <csteipp>	 dns
[23:32:33] <LeslieCarr>	 dns is not working ?
[23:32:54] <LeslieCarr>	 hrm, dns appears working here...
[23:32:59] <greg-g>	 csteipp: wfm
[23:33:00] <icinga-wm>	 PROBLEM - DPKG on mw1051 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:33:04] <csteipp>	 Just me  then :)
[23:33:19] <LeslieCarr>	 what's your resolver ?
[23:33:20] <csteipp>	 Right as soon as I synced, so that freaked me out...
[23:33:21] <notpeter>	 csteipp: wfm
[23:33:24] <csteipp>	 4.2.2.2
[23:33:27] <greg-g>	 csteipp: yeah, and my server in MN, so yeah, probably just you (thankfully)
[23:33:28] <csteipp>	 wieard
[23:33:34] <LeslieCarr>	 hrm
[23:33:36] <notpeter>	 csteipp: y u no use 8.8.8.8?
[23:33:44] <LeslieCarr>	 the 4.2.2.2 i just hit works, but they have a ton
[23:33:45] <csteipp>	 becuz google evil
[23:34:03] <notpeter>	 csteipp: come over here and say that into my google glass ;)
[23:34:14] <csteipp>	 Oh, hah, just Chrome... stupid google
[23:34:20] <csteipp>	 Sorry all.
[23:34:57] <icinga-wm>	 RECOVERY - DPKG on mw1051 is OK: All packages OK
[23:35:03] <LeslieCarr>	 !log upgrading mw1052 to mw1060
[23:35:10] <morebots>	 Logged the message, Mistress of the network gear.
[23:36:17] <icinga-wm>	 PROBLEM - MySQL Recent Restart on db1016 is CRITICAL: NRPE: Command check_mysql_recent_restart not defined
[23:36:30] <icinga-wm>	 PROBLEM - MySQL disk space on db1016 is CRITICAL: NRPE: Command check_disk_6_3 not defined
[23:36:37] <icinga-wm>	 PROBLEM - mysqld processes on db1016 is CRITICAL: NRPE: Command check_mysqld not defined
[23:36:57] <icinga-wm>	 RECOVERY - Disk space on db1016 is OK: DISK OK
[23:37:07] <icinga-wm>	 RECOVERY - DPKG on db1016 is OK: All packages OK
[23:37:17] <icinga-wm>	 RECOVERY - MySQL Recent Restart on db1016 is OK: OK seconds since restart
[23:37:17] <icinga-wm>	 RECOVERY - RAID on db1016 is OK: OK: State is Optimal, checked 2 logical device(s)
[23:37:27] <icinga-wm>	 RECOVERY - MySQL disk space on db1016 is OK: DISK OK
[23:38:37] <icinga-wm>	 RECOVERY - mysqld processes on db1016 is OK: PROCS OK: 1 process with command name mysqld
[23:38:47] <icinga-wm>	 PROBLEM - Apache HTTP on mw105 is CRITICAL: Connection refused
[23:38:47] <icinga-wm>	 PROBLEM - DPKG on mw1058 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:38:57] <icinga-wm>	 PROBLEM - DPKG on mw1055 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:39:07] <icinga-wm>	 PROBLEM - DPKG on mw1059 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:39:07] <icinga-wm>	 PROBLEM - DPKG on mw1052 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:39:27] <icinga-wm>	 PROBLEM - DPKG on mw1053 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:41:07] <icinga-wm>	 RECOVERY - DPKG on mw1059 is OK: All packages OK
[23:41:07] <icinga-wm>	 RECOVERY - DPKG on mw1052 is OK: All packages OK
[23:41:47] <icinga-wm>	 RECOVERY - DPKG on mw1058 is OK: All packages OK
[23:41:47] <icinga-wm>	 PROBLEM - Apache HTTP on mw1055 is CRITICAL: Connection refused
[23:41:59] <icinga-wm>	 RECOVERY - DPKG on mw1055 is OK: All packages OK
[23:41:59] <icinga-wm>	 PROBLEM - Apache HTTP on mw1059 is CRITICAL: Connection refused
[23:41:59] <icinga-wm>	 PROBLEM - Apache HTTP on mw1058 is CRITICAL: Connection refused
[23:42:07] <icinga-wm>	 PROBLEM - Apache HTTP on mw1052 is CRITICAL: Connection refused
[23:42:27] <icinga-wm>	 RECOVERY - DPKG on mw1053 is OK: All packages OK
[23:42:57] <icinga-wm>	 PROBLEM - Apache HTTP on mw1053 is CRITICAL: Connection refused
[23:44:47] <icinga-wm>	 RECOVERY - Apache HTTP on mw105 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.311 second response time
[23:44:47] <icinga-wm>	 RECOVERY - Apache HTTP on mw1055 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.081 second response time
[23:44:58] <icinga-wm>	 RECOVERY - Apache HTTP on mw1059 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time
[23:44:58] <icinga-wm>	 RECOVERY - Apache HTTP on mw1053 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.063 second response time
[23:44:58] <icinga-wm>	 RECOVERY - Apache HTTP on mw1058 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.088 second response time
[23:45:07] <icinga-wm>	 RECOVERY - Apache HTTP on mw1052 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.087 second response time