[00:07:29] <icinga-wm>	 PROBLEM - Offline Content Generation - Collection on rhodium is CRITICAL: Connection refused  
[00:18:39] <logmsgbot>	 !log ori synchronized php-1.23wmf5/extensions/PagedTiffHandler  'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359'
[00:18:54] <morebots>	 Logged the message, Master
[00:20:58] <ori-l>	 the exception log is full
[00:21:19] <ori-l>	 Invalid IP given in XFF ..
[00:21:22] <ori-l>	 is there a bug for that?
[00:21:54] <AaronSchulz>	 not that I know of
[00:22:06] <logmsgbot>	 !log ori synchronized php-1.23wmf4/extensions/PagedTiffHandler  'Update PagedTiffHandler to master for I52fe2ec25 / bug 57359'
[00:22:11] <ori-l>	 https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&x=0.5&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception&gtype=stack&glegend=show&aggregate=1&embed=1
[00:22:18] <morebots>	 Logged the message, Master
[00:22:37] <AaronSchulz>	 ori-l: the tif no longer times out now
[00:22:44] <ori-l>	 yay!
[00:23:02] <AaronSchulz>	 the tortilla hat :)
[00:23:23] <ori-l>	 heheh
[00:23:47] <ori-l>	 the exceptions are all for favico
[00:23:50] <ori-l>	 favicon.ico
[00:25:15] <aude>	 Reedy: https://gerrit.wikimedia.org/r/#/c/97149/
[00:25:35] <aude>	 trivial enough to self merge, but if someone else wants to.... :)
[00:25:48] * aude  waits for jenkins

[00:26:36] <andrewbogott>	 ori-l: Do you have a minute to talk me through getting a local branch that tracks an upstream branch in core?
[00:27:01] <andrewbogott>	 (my ultimate goal -- to commit get an extension version part of that branch)
[00:27:02] <Reedy>	 andrewbogott: git branch --track wmf/1.23wmf5 origin/wmf/1.23wmf5
[00:27:21] <andrewbogott>	 where the first arg is my local branch?
[00:27:35] <Reedy>	 yup
[00:27:45] <andrewbogott>	 Should the local branch exist already, or does that create it?
[00:28:08] <andrewbogott>	 (and, is there really a wmf5 already?  Or is wmf4 the latest?)
[00:28:15] <ori-l>	 there's a wmf5
[00:28:25] <ori-l>	 <andrewbogott> (my ultimate goal -- to commit get an extension version part of that branch)
[00:28:27] <Reedy>	 That was ^R and typing --track
[00:28:29] <ori-l>	 i didn't understand that
[00:29:09] <ori-l>	 what's this favicon nonsense
[00:29:17] <andrewbogott>	 ori-l: Ryan tells me that when I commit a patch to an extension I should make a corresponding patch in core so that 'submodule update' in core doesn't clobber the extension version
[00:29:21] <andrewbogott>	 is that crazy talk?
[00:29:59] <Reedy>	 ori-l: We use a php script to serv favicons
[00:30:01] <ori-l>	 no, he's right
[00:30:41] <ori-l>	 a production branch of mediawiki consists of mediawiki core and a set of submodules representing all the extensions that are currently deployed
[00:30:49] * andrewbogott  brushes cobwebs away from submodule knowledge

[00:31:15] <Reedy>	 given in XFF 'unknown, 10.64.0.126'
[00:31:17] <Reedy>	 lol.
[00:32:17] <andrewbogott>	 ori-l: ok, so I feel like my checkout of core should have reference to a bunch of extensions, and I need to 'git submodule init' to get the submodule actually checked out...
[00:32:43] <ori-l>	 if it's a production branch, yes; 'vanilla' core doesn't have submodules
[00:33:07] <andrewbogott>	 oh, hah, I'm on the wrong branch
[00:33:30] <andrewbogott>	 now it works just like I expected :)
[00:33:44] <ori-l>	 :)
[00:33:46] <ori-l>	 it's a bit tricky
[00:34:20] <aschulz|laptop>	 !log Manually fixed img_metadata for File:Zentralbibliothek_Zürich_-_Heinrich_Bullingers_Westerhemd_-_000012135.tif by forcing /bin/identify usage
[00:34:23] <andrewbogott>	 By the way, I note that you use marching branches rather than tagging a single main branch...
[00:34:35] <morebots>	 Logged the message, Master
[00:34:44] <andrewbogott>	 Are new branches always derived from the head of the former one?  So that e.g. wmf5 is a branch of wmf4 which is a branch from wmf3, etc. etc.?
[00:34:50] <Reedy>	 Nope
[00:34:55] <Reedy>	 From master and diverge from there
[00:35:07] <ori-l>	 by our very own Reedy
[00:35:10] <andrewbogott>	 ok, so if I want my submodule patch to actually matter I need to make it on master
[00:35:12] <andrewbogott>	 and on a branch
[00:35:15] <Reedy>	 but in theory everything in an earlier branch should be in a newer one
[00:35:17] <ori-l>	 no
[00:35:23] <andrewbogott>	 ...
[00:35:45] <andrewbogott>	 no, which?
[00:36:01] <ori-l>	 when a new branch is cut a set of php scripts populate the set submodules
[00:36:04] <ori-l>	 the set of submodules
[00:36:13] <ori-l>	 it's not something we like to talk about in public
[00:36:35] <ori-l>	 like, if you were giving a presentation about our use of git, this wouldn't go on slide #1
[00:37:15] <aude>	 :)
[00:37:23] <andrewbogott>	 So how does a new branch decide what version of <extension> to track?
[00:37:37] <Reedy>	 Using the submodule commit in the branch of core
[00:37:53] <andrewbogott>	 in… what branch of core?
[00:37:54] <aude>	 it's always master branch, except in certain excptions
[00:37:57] <aude>	 exceptions*
[00:38:02] <aude>	 master of the extensions
[00:38:34] <aude>	 whatever was master at the time the core branch was made
[00:38:43] <andrewbogott>	 That suggests that, indeed, I should make my change to master (so that it gets picked up by future branches) and to current branch (so it gets picked up now.)
[00:38:52] <andrewbogott>	 And yet ori-l tells me… 'no'
[00:39:04] <ori-l>	 oh, you meant the master of the extension
[00:39:09] <ori-l>	 if so, then yes, you are correct
[00:39:13] <andrewbogott>	 No!
[00:39:15] * andrewbogott  scowls

[00:39:26] <andrewbogott>	 OK, so, I misread.
[00:39:37] <andrewbogott>	 You're saying that new branches just get HEAD of all the extensions.
[00:39:46] <andrewbogott>	 Which seems optimistic, but makes my life easy :)
[00:39:55] <Reedy>	 Not all of them, but most of them yes
[00:40:21] <andrewbogott>	 So anything that I commit to OpenStackManager will eventually land on the wmf branches.  Just not in the current one.
[00:40:52] <bd808>	 It will be included in future branches
[00:41:08] <AaronSchulz>	 ori-l: heh, it's scary that parses, gallery views, and such can trigger loads of file metadata updates (say if all jpg files had a metadata version change)
[00:42:20] <andrewbogott>	 Reedy, ori-l, aude, so in short, all I need to do is this, and I'm done:  https://gerrit.wikimedia.org/r/#/c/97158/
[00:42:29] <andrewbogott>	 (not sure why I am asking about this in ops)
[00:43:15] <ori-l>	 AaronSchulz: eeeeeep
[00:44:13] <aude>	 andrewbogott: yes, 
[00:44:25] <andrewbogott>	 ok then!
[00:44:31] <andrewbogott>	 Thank you all!
[00:44:41] <greg-g>	 uh, the merge to wmf5 won't go out unless you deploy it
[00:44:41] <ori-l>	 marktraceur: ResourceLoaderFileModule::readStyleFile: style file not found: "/usr/local/apache/common-local/php-1.23wmf4/extensions/UploadWizard/resources/ext.uploadWizard.uploadCampaign.list.css"
[00:44:48] * aude  always reads "OSM" = openstreetmap :)

[00:45:07] <greg-g>	 wmf5 is already out there
[00:45:20] <andrewbogott>	 greg-g: That extension is only used on Labs' own weird private system which is not currently attached to the deployment system.
[00:45:25] <greg-g>	 and if you merge something to wmf5 without deploying, you're saying "hey next person, you manage the fall out from this breakage"
[00:45:27] <andrewbogott>	 So I'll just check out by hand when the time comes.
[00:45:43] <greg-g>	 andrewbogott: then merge to master, not to wmf5?
[00:46:20] <marktraceur>	 ori-l: That sounds like a YuviProblem, but I'll take a look
[00:46:23] <andrewbogott>	 um… I feel like we just established that 'merge to master' is meaningless since new branches automatically get...
[00:46:38] <AaronSchulz>	 ori-l: https://bugzilla.wikimedia.org/buglist.cgi?title=Special%3ASearch&quicksearch=upgradeRow&list_id=252640
[00:46:45] <marktraceur>	 Wait, are you guys deploying something? :/
[00:46:49] <andrewbogott>	 greg-g: Other than that 'merge to master' thing I don't disagree with what you're saying though.
[00:47:01] <logmsgbot>	 !log reedy synchronized php-1.23wmf4/extensions/MobileFrontend/  'Fix history fatal'
[00:47:14] <aude>	 andrewbogott: is this for wikitech wiki?
[00:47:15] <morebots>	 Logged the message, Master
[00:47:22] <andrewbogott>	 aude: yes
[00:47:34] <aude>	 unless something is odd, it's on 1.23wmf3 https://wikitech.wikimedia.org/wiki/Special:Version
[00:47:38] <andrewbogott>	 It is currently on wmf/1.23wmf3
[00:47:42] <aude>	 not in sync with the rest of stuff
[00:47:47] <aude>	 since it's separate
[00:47:58] <andrewbogott>	 My plan was to get my OSM patch into the latest, then to update wikitech to the 'latest branch' which I take to be 1.23wmf5
[00:48:00] <logmsgbot>	 !log reedy synchronized php-1.23wmf5/extensions/MobileFrontend/  'Fix history fatal'
[00:48:12] * aude  nods

[00:48:13] <greg-g>	 gotcha, I came in late
[00:48:14] <morebots>	 Logged the message, Master
[00:48:17] <aude>	 don't listen to me :)
[00:48:41] <logmsgbot>	 !log reedy synchronized php-1.23wmf5/extensions/Wikibase/client/  'Fix InfoAction fatal'
[00:48:56] <morebots>	 Logged the message, Master
[00:49:02] <andrewbogott>	 greg-g, if trying to insert things into branches causes distress then I can just do nothing and wait for my extension to get updated in the next branch.
[00:49:02] <marktraceur>	 ori-l: Yuvi fucked up a filename in a module declaration, fixing now
[00:49:07] <andrewbogott>	 Not ideal, but acceptable.
[00:49:49] <andrewbogott>	 (someday soon wikitech will follow standard deployment scheme, at which point that's how it'll work anyway)
[00:49:53] <greg-g>	 I guess I don't undersatnd why it needs to be merged to 1.23wmf5, can't it be a 1.23wikitechX or osmething since it's specific to that?
[00:50:00] <greg-g>	 whatever
[00:50:53] <andrewbogott>	 I have… no real opinion, just trying to do what Ryan most recently suggested.
[00:53:11] <greg-g>	 gotcha
[00:53:15] <greg-g>	 andrewbogott: thanks for taking care of it
[00:53:21] <greg-g>	 andrewbogott: sorry for being a johnny come lately
[00:53:28] <grrrit-wm>	 (03PS1) 10Reedy: Disable OAuth on fishbowl wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97166 
[00:53:49] <AaronSchulz>	 https://upload.wikimedia.org/wikipedia/commons/thumb/d/d5/Vladimir_Putin%27s_press_conference_on_2012-12-20.ogv/320px--Vladimir_Putin%27s_press_conference_on_2012-12-20.ogv.jpg
[00:53:54] <AaronSchulz>	 great looking thumbnail :p
[00:53:56] <grrrit-wm>	 (03CR) 10CSteipp: [C: 031] Disable OAuth on fishbowl wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97166 (owner: 10Reedy)
[00:54:40] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Disable OAuth on fishbowl wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97166 (owner: 10Reedy)
[00:54:59] <ori-l>	 marktraceur: it's low-impact, get it reviewed and plan to deploy it next week
[00:55:26] <marktraceur>	 ori-l: Aye sah
[00:57:00] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'Disable OAuth on fishbowl wikis'
[00:57:16] <morebots>	 Logged the message, Master
[00:57:46] <grrrit-wm>	 (03CR) 10Reedy: [V: 032] Disable OAuth on fishbowl wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97166 (owner: 10Reedy)
[00:57:55] <ori-l>	 the XFF thing is exclusively enwiki
[01:04:16] <Reedy>	 Logged the xff favicon thing at https://bugzilla.wikimedia.org/show_bug.cgi?id=57467
[01:06:04] <ori-l>	 thanks, I can't focus any more, heading off
[01:06:08] <ori-l>	 thanks for filing that
[01:06:28] <Reedy>	 the unknown should be another IP, right?
[01:06:47] <ori-l>	 no idea
[01:07:11] <ori-l>	 i think the zero folks were doing something XFF related
[01:07:19] <ori-l>	 dr0ptp4kt  yurik ^
[01:07:23] <ori-l>	 i'll cc them on the bug
[01:07:41] <Reedy>	 cool
[02:14:03] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf4) at Sat Nov 23 02:14:03 UTC 2013
[02:14:20] <morebots>	 Logged the message, Master
[02:18:02] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 16h 59m 39s  
[02:26:19] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf5) at Sat Nov 23 02:26:18 UTC 2013
[02:26:32] <morebots>	 Logged the message, Master
[03:12:24] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 23 03:12:24 UTC 2013
[03:12:40] <morebots>	 Logged the message, Master
[03:17:45] <grrrit-wm>	 (03CR) 10Spage: "If we're disabling VE for Flow on every wiki, then doing it in CommonSettings.php seems clearer." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96161 (owner: 10EBernhardson)
[04:00:40] <grrrit-wm>	 (03PS3) 10Spage: Enable Flow discussions on a few test wiki pages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 
[04:11:55] <grrrit-wm>	 (03CR) 10Spage: [C: 04-1] "Do not +2 until we have our window." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 (owner: 10Spage)
[04:23:07] <icinga-wm>	 PROBLEM - etherpad_lite_process_running on zirconium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^node node_modules/ep_etherpad-lite/node/server.js  
[04:27:18] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[04:33:27] <icinga-wm>	 PROBLEM - MySQL InnoDB on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[04:33:28] <icinga-wm>	 PROBLEM - MySQL Idle Transactions on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[04:33:28] <icinga-wm>	 PROBLEM - MySQL Recent Restart on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[04:33:28] <icinga-wm>	 PROBLEM - MySQL Slave Delay on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[04:34:28] <icinga-wm>	 RECOVERY - MySQL Recent Restart on db1021 is OK: OK 4040006 seconds since restart  
[04:34:28] <icinga-wm>	 RECOVERY - MySQL Slave Delay on db1021 is OK: OK replication delay 35 seconds  
[04:35:17] <icinga-wm>	 RECOVERY - MySQL InnoDB on db1021 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[04:35:17] <icinga-wm>	 RECOVERY - MySQL Idle Transactions on db1021 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[04:46:18] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (202733)  
[04:54:18] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[04:57:18] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (210185)  
[05:01:07] <icinga-wm>	 RECOVERY - etherpad_lite_process_running on zirconium is OK: PROCS OK: 1 process with regex args ^node node_modules/ep_etherpad-lite/node/server.js  
[05:02:18] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[05:03:37] <spagewmf>	 etherpad.wikimedia.org was 503, but came back to life
[05:05:26] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200408)  
[05:06:17] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[05:10:26] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (204281)  
[05:12:17] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[05:15:07] <greg-g>	 not a fun night, eh
[05:18:06] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 3h 0m 4s  
[05:18:26] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (203759)  
[05:21:26] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[05:21:40] <greg-g>	 job_queue flapping, fun times
[05:32:26] <icinga-wm>	 PROBLEM - MySQL InnoDB on db1056 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[05:32:56] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[05:33:16] <icinga-wm>	 RECOVERY - MySQL InnoDB on db1056 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[05:33:26] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (203633)  
[05:36:18] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[05:41:27] <icinga-wm>	 PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200627)  
[05:42:18] <icinga-wm>	 RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000  
[06:10:16] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Add Twitter account to Varnish's error page [operations/puppet] - 10https://gerrit.wikimedia.org/r/97190 
[06:12:11] <jeremyb>	 hrmmmmmm
[06:12:53] <jeremyb>	 paravoid: does that actually have anything relevant to outages?
[06:13:10] <paravoid>	 I think it does, on large outages
[06:13:26] <jeremyb>	 ok. well guillom would know :)
[06:13:40] <jeremyb>	 at first i assumed you were adding @wikimediatech
[06:13:57] <jeremyb>	 @wikimediatech needs some work to be more reliable than it is now
[06:15:27] <jeremyb>	 case in point! no tweets for ~9 days
[06:16:30] <paravoid>	 Erik proposed that one too, dunno
[06:16:36] <paravoid>	 it's too technical/terse
[06:16:42] <jeremyb>	 ori-l: you mentioned something about a rewrite? (was done but not converted yet?)
[06:17:16] <jeremyb>	 paravoid: watchmouse maybe?
[06:17:19] <ori-l>	 rewrite of what?
[06:17:26] <jeremyb>	 ori-l: morebots
[06:17:37] <paravoid>	 I did not
[06:17:38] <ori-l>	 i keep insisting i won't do it
[06:17:41] <paravoid>	 I was evaluating fedmsg
[06:18:08] <jeremyb>	 ori-l: i was thinking you had said something was already written
[06:18:19] <jeremyb>	 paravoid: right, i addressed ori-l :)
[06:18:28] <paravoid>	 oh sorry
[06:18:33] <ori-l>	 fedmsg looks good
[06:18:45] <jeremyb>	 paravoid: i was saying instead of twitter maybe use watchmouse (status.wm.o)
[06:19:07] <paravoid>	 ori-l: it looks shiny, it isn't as much
[06:19:10] <paravoid>	 I did evaluate it
[06:20:15] <paravoid>	 very undocumented, things such as hardcoded environments from fedora's infrastructure in the python source ('prod', 'stg', 'dev'), terrible Debian package
[06:20:33] <paravoid>	 nothing unfixable, but it was more than an apt-get and a little fiddling away and I didn't find the time
[06:21:07] <ori-l>	 speaking of undocumented, i was finally curious enough about systemd to try and grok it
[06:21:12] <ori-l>	 the docs are really bad
[06:21:28] <ori-l>	 upstart is exemplary in this respect
[06:21:40] <ori-l>	 like, just look at this:
[06:21:41] <ori-l>	 http://www.freedesktop.org/wiki/Software/systemd/
[06:21:50] <ori-l>	 the first paragraph is presumably a joke, except it runs on for wayyy to long
[06:22:04] <ori-l>	 and then you have to scroll fifty pages to get to the actual documentation
[06:22:33] <ori-l>	 err, the spelling paragraph, i mean
[06:23:08] <paravoid>	 you know Debian had multiple flamewars on whether it should switch to upstart or systemd, right?
[06:24:11] <ori-l>	 only what you told me, but yeah
[06:24:22] <paravoid>	 oh I did
[06:24:25] <paravoid>	 I'm forgetful, sorry :)
[06:25:18] <apergos>	 I read a tiny bit of one of those threads
[06:25:24] <apergos>	 then gave it up as a bad job
[06:27:19] <paravoid>	 ori-l: holtWintersForecast(diffSeries(stats.job-insert.count,stats.job-pop.count))
[06:27:44] <paravoid>	 see where I'm going with this?
[06:28:25] <ori-l>	 clever
[06:28:42] <paravoid>	 check_graphite
[06:29:14] <paravoid>	 I want to alert when the trend is upwards for a while
[06:29:23] <ori-l>	 well, just holtWintersForecast(stats.job-insert.count) should do
[06:29:41] <paravoid>	 why?
[06:30:04] <paravoid>	 we may have a large number of jobs inserted but processing them just fine
[06:30:46] <ori-l>	 we may also be processing jobs quickly if they're cheap
[06:31:07] <ori-l>	 hmmm
[06:31:13] <paravoid>	 exactly
[06:31:29] <paravoid>	 hence the difference
[06:33:55] <paravoid>	 hm, I guess I'd like the diff in a larger series
[06:33:56] <ori-l>	 sure, let's try it
[06:34:01] <paravoid>	 than just each data point
[06:43:31] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[06:44:50] <paravoid>	 hm
[06:45:01] <paravoid>	 I broke the ganglia job queue graph
[06:45:06] <paravoid>	 when I removed the check from hume
[06:45:21] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical  
[06:58:39] <grrrit-wm>	 (03PS3) 10Ori.livneh: rewrite nginx module [operations/puppet] - 10https://gerrit.wikimedia.org/r/96961 
[07:00:29] <grrrit-wm>	 (03CR) 10Ori.livneh: "> it's preferable to manage notifies on the ssl servers" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96961 (owner: 10Ori.livneh)
[07:03:31] <grrrit-wm>	 (03CR) 10Ori.livneh: "Also: I was able to test the module in general and the notify control in particular on Vagrant. Still a bit anxious about weird conflicts " [operations/puppet] - 10https://gerrit.wikimedia.org/r/96961 (owner: 10Ori.livneh)
[07:32:51] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[08:18:08] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 6h 0m 6s  
[08:54:29] <guillom>	 jeremyb: @wikimedia does have updates about outages; when I happen to be around on IRC and either I see the outage or someone thinks of pinging me :)
[09:04:51] <apergos>	 we need another you to cover the non-you timezones
[09:05:04] <apergos>	 or to cover sick-you
[09:05:25] <p858snake|l>	 or we could just steal the TARDIS
[09:05:32] <p858snake|l>	 and opensource it!
[09:05:44] <apergos>	 thanks for volunteering
[09:05:55] <apergos>	 but if we steal it isn't that good enough?  you only need one...
[09:06:22] <apergos>	 brb 
[09:06:50] <p858snake|l>	 well we are abit against using closed source tools generally
[09:32:50] <grrrit-wm>	 (03PS1) 10Odder: Add an alias for NS_PROJECT on Bengali Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97206 
[09:33:30] <Nemo_bis>	 apergos: it's enough to !log and add a #Wikimedia #Wikipedia hashtag
[09:34:01] <Nemo_bis>	 sadly twitter doesn't have a feature to allow subscription to topics, unlike the old identi.ca, but there isn't much one can do about it
[09:37:03] <apergos>	 well having someone actually translate our rather terse log entries to something more readable would be nice
[09:37:17] <apergos>	 and 
[09:37:25] * apergos  is sad abou identi.ca's demise too 

[09:37:40] <apergos>	 ok, time to find breakfast-like items
[09:45:36] <icinga-wm>	 PROBLEM - etherpad_lite_process_running on zirconium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^node node_modules/ep_etherpad-lite/node/server.js  
[09:49:36] <icinga-wm>	 RECOVERY - etherpad_lite_process_running on zirconium is OK: PROCS OK: 1 process with regex args ^node node_modules/ep_etherpad-lite/node/server.js  
[09:52:36] <icinga-wm>	 PROBLEM - etherpad_lite_process_running on zirconium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^node node_modules/ep_etherpad-lite/node/server.js  
[10:00:38] <icinga-wm>	 RECOVERY - etherpad_lite_process_running on zirconium is OK: PROCS OK: 1 process with regex args ^node node_modules/ep_etherpad-lite/node/server.js  
[10:13:04] <apergos>	 !log stopped etherpadlite, removed all but the last 5000 lines of its log file, (root running out of space), restarted
[10:13:19] <morebots>	 Logged the message, Master
[10:13:45] <apergos>	 !log ...on zirconium
[10:13:53] <apergos>	 still not awake
[10:13:59] <morebots>	 Logged the message, Master
[10:23:40] <MaxSem>	 apergos, euro staff slowly turns into night creatures. mornings aren't for us (even if they're in afternoon):P
[10:24:51] <apergos>	 actually it would be fine if I hadn't stayed up late last night in The Matrix
[10:24:53] <apergos>	 (of rfps)
[10:25:52] <apergos>	 also haven't had morning tea/hot chocolate/whatever yet
[10:26:38] <apergos>	 did wake up in time to look in on neon, it's a happy camper, which makes me happy :-)
[11:18:51] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 9h 0m 49s  
[14:04:07] <springle>	 !log testing 5.5.34 mariadb.org debs on db60
[14:04:21] <morebots>	 Logged the message, Master
[14:11:15] <apergos>	 oohhh
[14:15:07] <springle>	 apergos: found a way to pin stuff to keep puppet happy
[14:15:26] <apergos>	 oh excellent
[14:15:34] <springle>	 so can at least test without messing around
[14:15:37] <apergos>	 yep
[14:15:45] <apergos>	 care to share? :-)
[14:17:41] <springle>	 set http_proxy to brewster. added upstream mariadb.org repo. pinned that version as preferred. installed. removed everything.. and puppet thinks all is still well
[14:19:45] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 12h 1m 44s  
[14:20:09] <Reedy>	 apergos: First you need a REALLY big hammer...
[14:20:44] <apergos>	 hold still reedy I think I've got one right there, and funny how everything does look like a nail...  :-P
[14:21:29] <Reedy>	 The following packages will be REMOVED
[14:21:29] <Reedy>	   libmariadbclient18 mariadb-client-5.5 mariadb-client-core-5.5 mariadb-server mariadb-server-5.5 mariadb-server-core-5.5
[14:21:29] <Reedy>	 The following packages will be upgraded:
[14:21:29] <Reedy>	   libmysqlclient18 mysql-common
[14:21:32] <Reedy>	 gj ubuntu
[14:22:28] <Reedy>	 Ah. mariadb source was commented out from saucy upgrade
[14:22:36] <Reedy>	 I can't remember if I had to pin it too :/
[14:22:50] <Reedy>	 seemingly not :)
[14:24:27] <apergos>	 that reminds me, time to see how the f20 roadmap is coming along
[14:25:25] <apergos>	 still on track
[15:35:40] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] "didn't you mean https://twitter.com/wikimediatech instead of https://twitter.com/wikimedia ? that's the more technical feed that includes" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97190 (owner: 10Faidon Liambotis)
[15:36:23] <grrrit-wm>	 (03CR) 10Dzahn: "didn't you mean https://twitter.com/wikimediatech instead of https://twitter.com/wikimedia ? that's the more technical feed that includes" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/97190 (owner: 10Faidon Liambotis)
[15:41:33] <grrrit-wm>	 (03CR) 10Dzahn: "besides, unfortunately it seems it stopped logging on Nov 14 ? is a bot script down again?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97190 (owner: 10Faidon Liambotis)
[16:24:34] <jeremyb>	 mutante: yes, the twitter doesn't work on the grid, (at least AFAICT) only tools-login. but the twitter api library package is installed... haven't gotten around to investigating more
[16:24:43] <jeremyb>	 mutante: also see above about choice of twitter account
[16:26:39] <jeremyb>	 guillom: sure, but i had just gone back several days looking for relevant tweets and saw nothing. maybe it wasn't a representative sample or there wasn't a big enough outage. but if e.g. only 2 in 5 outages make it to the feed then I don't think we want to be linking to it that way. (no idea if that's an accurate value, essentially just made up a number)
[16:26:57] <mutante>	 jeremyb: thanks for looking. so did people want to have just "wikimedia" on it instead of "wikimediatech".. on the error page? 
[16:27:12] <mutante>	 jeremyb: could put "wikimediatwork" too :o
[16:28:29] <mutante>	 i see logs now.. so you noticed it was broken too and that's why.. nod
[16:29:25] <mutante>	 can we just fix the SAL/tech one?
[16:30:01] <mutante>	 or can they all be in one account but with different hash tags and then you filter? shrug
[16:31:45] <grrrit-wm>	 (03CR) 10Dzahn: "ok, i saw the IRC backlog about this now. so it was chosen because the tech account is currently not updating. still think we should fix t" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97190 (owner: 10Faidon Liambotis)
[16:39:54] <jeremyb>	 mutante: errr, no
[16:40:08] <jeremyb>	 mutante: it was chosen separate from account working or not
[16:40:37] <jeremyb>	 mutante: 06:10:16 UTC
[16:40:42] <jeremyb>	 this channel
[16:41:34] <jeremyb>	 mutante: anyway, tech account not working right i knew (I even !log'd when i made relevant changes). didn't realize it broke again though (i had moved it to tools-login which fixes twitter)
[16:42:03] <jeremyb>	 not sure who originally set it up there
[16:42:17] <mutante>	 hmm.ok. oh well. i think SAL is more relevant in an outage than the latest nice images on commons
[16:43:10] <mutante>	 jeremyb: thanks! see, i didn't even realize it was moved, i usually just check SAL on wikitech but never on twitter
[16:44:19] <jeremyb>	 mutante: maybe someone moved it back to the grid or it happened automatically somehow. you're not supposed to run long-term stuff on tools-login...
[16:44:32] <jeremyb>	 (when i checked last night it was on the grid)
[16:47:43] <mutante>	 on tools-login? can't it run where the bots run?  ok, i'm sure it will be figured out .. not worrying more about it right now
[17:20:12] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 15h 2m 10s  
[19:28:11] <grrrit-wm>	 (03PS1) 10Dereckson: Undeploy ExpandTemplates extension [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97331 
[19:43:30] <icinga-wm>	 PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run for 1d 10h 25m 6s  
[19:54:51] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[20:07:57] <guillom>	 jeremyb: It was examples of me not being around, or not noticing it, or not being notified, or all of the above, then :)
[20:12:50] <icinga-wm>	 RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Sat Nov 23 20:12:44 UTC 2013  
[20:19:11] <jeremyb>	 guillom: not the outage was too minor :)
[20:19:46] <guillom>	 jeremyb: Well, since I wasn't there, I can't say that :)
[20:20:20] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 18h 2m 18s  
[21:35:16] <grrrit-wm>	 (03CR) 10Umherirrender: "Same question as on Ifad0cad215cd0e91746dd08c3f8888c9fbe01c20:" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97331 (owner: 10Dereckson)
[21:53:57] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[22:03:31] <grrrit-wm>	 (03CR) 10Dereckson: "As noted in bug 57484 description, this change should only deployed after I7ef63488dc3ad3885bcf99ff52852e1c6981942b has reached all wmf br" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97331 (owner: 10Dereckson)
[22:11:14] <grrrit-wm>	 (03PS2) 10Nemo bis: Add Twitter account to Varnish's error page [operations/puppet] - 10https://gerrit.wikimedia.org/r/97190 (owner: 10Faidon Liambotis)
[22:33:51] <icinga-wm>	 PROBLEM - SSH on amslvs1 is CRITICAL: Server answer:  
[22:34:51] <icinga-wm>	 RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)  
[23:20:21] <icinga-wm>	 PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 0d 21h 2m 19s