[00:00:08] huh, pageant was bottlenecking...must faster with another agent [00:00:38] at least scap-1skins [00:07:29] (03PS1) 10Aaron Schulz: Made scap cdb scripts only show output if anything happened [operations/puppet] - 10https://gerrit.wikimedia.org/r/103872 [00:19:13] (03PS1) 10Lcarr: adding python-stompy for awight [operations/puppet] - 10https://gerrit.wikimedia.org/r/103876 [00:21:10] (03PS6) 10BryanDavis: [WIP] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 [00:23:11] (03CR) 10Lcarr: [C: 032] adding python-stompy for awight [operations/puppet] - 10https://gerrit.wikimedia.org/r/103876 (owner: 10Lcarr) [00:27:54] ori: how useful is it actually doing /usr/local/bin/scap-1skins ? [00:29:29] I assume RL will still have whatever problems that was trying to avoid anyway [00:30:46] and the dsh hang really isn't helping... [00:36:47] Aaron|home: heh, i forgot about that script [00:36:50] i don't think it's useful at all [00:38:18] ori: some of the rsync servers are apaches right? [00:40:43] Aaron|home: based on the naming scheme, probably [00:41:11] yeah mw1070 is [00:41:23] so scap1-skins actually makes the RL stuff *worse* [00:43:02] really it's a mixed bag, I'm sure there are a few additive things it would help with, but I don't see it being worth it [00:44:24] Aaron|home: gotta run for a bit [00:47:14] (03PS7) 10BryanDavis: [WIP] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 [00:47:36] (03PS1) 10Aaron Schulz: Removed scap-1skins for performance [operations/puppet] - 10https://gerrit.wikimedia.org/r/103880 [00:48:17] (03CR) 10BryanDavis: "Responding to Antoine's comments." (035 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [00:50:59] Aaron|home: what was the original reason for that script? [00:51:50] ori: if some servers have new code and link to new assets, they won't work if the user loads if from another server that doesn't yet have it [00:54:23] the first wikis on any new branch are the test wikis...which is probably why people don't complain about asset issues afaik [00:55:29] after that, the time it takes to switch versions for other wikis is very low [00:55:38] sync-wikiversions [01:25:38] Reedy: is there any use for the 9 old mw versions lying around in /common? [01:47:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [01:50:59] Aaron|home: wasn't that due to a cacheing issue? / Reedy [01:51:22] as people were getting served old stuff be we couldn't drop the rate much lower [02:10:22] !log LocalisationUpdate completed (1.23wmf7) at Fri Dec 27 02:10:22 UTC 2013 [02:10:41] Logged the message, Master [02:18:20] !log LocalisationUpdate completed (1.23wmf8) at Fri Dec 27 02:18:20 UTC 2013 [02:18:35] Logged the message, Master [02:35:52] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 27 02:35:51 UTC 2013 [02:36:08] Logged the message, Master [03:27:14] (03CR) 10Reedy: "I moved the sync-wikiversions to its current position so we'd only switch wikis over after the code had been migrated." [operations/puppet] - 10https://gerrit.wikimedia.org/r/103870 (owner: 10Aaron Schulz) [03:34:43] (03CR) 10Aaron Schulz: "You can always have a wiki in mixed version for a period (same with sync-wikiversions), it's just a matter of how long. The first deploy o" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103870 (owner: 10Aaron Schulz) [03:53:19] (03PS1) 10Aaron Schulz: Use delete-delay in MW_RSYNC_ARGS to make it a bit more atomic [operations/puppet] - 10https://gerrit.wikimedia.org/r/103895 [03:54:50] ori: did you mean to add that when delay-updates was added? [03:59:29] (03CR) 10Aaron Schulz: "I see we are missing --delete-delay though..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/103870 (owner: 10Aaron Schulz) [04:02:43] Reedy: what files are needed in the old MW versions we have? Is it just /skins and /resources? [04:02:55] all those have to be slowing down rsync [04:36:23] (03PS1) 10Aaron Schulz: Fixed minor comment typo [operations/puppet] - 10https://gerrit.wikimedia.org/r/103897 [04:48:32] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [05:20:11] (03CR) 10Ori.livneh: [C: 032] Fixed minor comment typo [operations/puppet] - 10https://gerrit.wikimedia.org/r/103897 (owner: 10Aaron Schulz) [07:49:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [09:01:02] (03PS1) 10Ori.livneh: perf: use strtok_r + strtod / strtoull instead of scanf [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104010 [09:02:08] (03CR) 10Ori.livneh: [C: 032 V: 032] perf: use strtok_r + strtod / strtoull instead of scanf [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104010 (owner: 10Ori.livneh) [10:50:46] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [13:51:52] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [15:48:21] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:49:16] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [16:52:42] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [17:12:55] Does anyone have some good openstack-fu? I'm building a small OS cluster in the office, and haven problems with the networking. (just need a simple flat DHCP network).. Right now no address is being assigned to the VM. (they boot, but I can't get to them...) [17:12:57] using nova-network, not neutron [17:13:59] if so: email joel -- I'll be tweaking some other stuff and SFK [17:14:18] AFK [17:32:55] cajoel: idk how much you've invested so far but you may also want to try https://code.google.com/p/ganeti/ [17:33:32] I took that to be 'just another dashboard' .. no? [17:33:52] I was also trying to model as much as possible on production environments... [17:34:33] cajoel: why try to model on prod? [17:34:53] cajoel: no, it's not a dashboard at all [17:35:34] jeremyb: knowledge sharing/overlap is a good reason [17:36:14] greg-g: ok, but are there any others? [17:36:16] what's cajoel's use case? [17:36:35] i bet it's not the same as prod [17:36:55] if it were the same as prod then cajoel could instruct his users to use wmflabs.org instead of whatever he's building [17:36:55] :-) [17:37:12] you know, the use case of my laptop isn't the same as a server, but I still use the linux kernel ;) [17:37:35] but yeah, point taken, no idea what he's doing :) [17:37:35] how about your phone? [17:37:55] in theory, it should be easier to support if using similar toolsets.. :) [17:38:16] cajoel: i think that's just an extension of greg-g's reason? [17:38:36] sure, he's confirming :) [17:38:36] * greg-g speaks for cajoel  [17:38:37] hah [17:38:42] jeremyb: trying to virtualize some of our internal services.. -- ldap, puppet master, dns, nagios, cacti, etc.. [17:38:58] right [17:39:09] that's approximately what i had guessed [17:39:14] and, openstack was built to do that, sooo... ;) [17:39:37] * greg-g has no horse in this race, bows out [17:39:57] Coren: andrewbogott ^^ [17:39:57] we have a mishmash of different hardware -- standardizing on some gear we just got from tampa, and trying to use the same tools.. [17:40:34] anyway, go the controller up, got some images all set, got a compute node up.. stuck on networking. [17:40:58] Seems like a pretty good idea if it doesn't take 1000 hours to configure. I really only have practice with devstack which is the dev/test rig. [17:41:18] Folks in #openstack were always helpful though... [17:41:19] I'll wander over there... [17:41:38] cheers [17:41:38] * andrewbogott is happy to help but not sure he'll be more useful than the manual :( [17:42:19] I need to do block storage today too, but that looks pretty straightforward [17:42:19] was just trying to avoid neutorn [17:42:39] I'd think a flat network would be enough. [17:43:16] cajoel: so, openstack is pretty complicated. or can be at least. as is AWS. one of the main reasons i mentioned ganeti is because some things seem to be much simpler than openstack. and seems like ganeti is a good match for your use case [17:43:34] I'm pretty comfortable with AWS, so this all feels normal. [17:44:11] I can try ganeti, but I'm already 3-4 hours in to OS [17:44:27] i mean AWS has like a 100 +/- different "services" to offer. and openstack is somewhat following that model [17:44:40] jeremyb: have you set up ganeti? (it wasn't clear about how it does block storage) [17:45:21] cajoel: IIRC there's 3 options. the one I think is most common is just using local disks. (with drbd) [17:46:30] so, with drbd you always have one ganeti node on standby for a given instance. (in addition to where it's actually running). then to live migrate you just swap master and standby [17:47:02] (and copy swap/ram/do a final drbd sync/etc.) [17:47:17] there's also a #ganeti :) [17:47:27] not changing horses just yet. :) [17:47:43] sure [17:47:43] but thanks for the pointers. [17:48:07] how can/does one follow along with developments? [17:48:16] no public git i guess? [17:51:15] yurik-road: how's the gas mileage? [17:55:30] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: [17:55:49] Logged the message, Master [17:57:10] hmm, 20sec [17:58:26] AaronSchulz: seriously? [18:01:53] ori: it seems odd that two measly files take soooo long [18:02:05] actually 20.787sec [18:08:31] PROBLEM - Disk space on ms-be5 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sde1 is not accessible: Input/output error [18:08:42] PROBLEM - RAID on ms-be5 is CRITICAL: CRITICAL: 1 failed LD(s) (Offline) [18:16:04] (03CR) 10Aaron Schulz: "This is ~25sec" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103880 (owner: 10Aaron Schulz) [18:22:28] ori: just doing 'echo "ok"' in with dsh -F30 takes 13sec for me [18:22:46] s/in/ [18:23:08] (03PS2) 10Andrew Bogott: Move lighttpd_config into mail.pp and rename. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103104 [18:24:49] (03CR) 10Jeremyb: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 (owner: 10Jeremyb) [18:25:56] AaronSchulz: ori: anyone care for conf changes today? :) either just https://gerrit.wikimedia.org/r/103730 or the whole gerrit topic. [18:26:08] full fanout takes it from 13.5 to 2.4...which sort of makes sense [18:26:26] AaronSchulz: 2.4? [18:26:40] I'd hope for a bit lower, but meh [18:27:11] (03CR) 10Jeremyb: "recheck" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/101220 (owner: 10John F. Lewis) [18:27:35] AaronSchulz: ohhh, you mean unlimited threads? [18:27:52] * jeremyb had misread [18:27:53] AaronSchulz: also, try salt! :P [18:30:54] (or at least reviews if not merges :P) [18:30:54] I don't think I have access to that [18:33:56] (03PS2) 10Jeremyb: import sources: move chapter wikis to own section [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103732 [18:34:37] AaronSchulz: well idk for sure. but you should eventually [18:35:17] (03CR) 10Jeremyb: import sources: move chapter wikis to own section (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103732 (owner: 10Jeremyb) [18:36:59] (deleted and rebuilt network in nova, now we're working... odd) [18:37:19] the stack is alive! [18:37:23] give it a turing test [18:37:39] hey greg-g, how do we schedule new deploys of updated versions of gwtoolset? [18:38:40] (03CR) 10Andrew Bogott: [C: 032] Move lighttpd_config into mail.pp and rename. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103104 (owner: 10Andrew Bogott) [18:39:20] (03CR) 10Jeremyb: [C: 04-1] "Move this to the chapter wikis section" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/101220 (owner: 10John F. Lewis) [18:41:28] cajoel: Nice work! And condolences that it fixed itself spontaneously, I hate that :( [18:44:04] when's a good time to get apache conf deployed? [18:44:44] i have a bunch of changes committed to a sandbox but i'm not going to work on them if no one's deploying. (one apache change is ready to deploy right now and has been sitting for a week) [18:45:25] <^d> jeremyb: When mutante is around and not on a friday? [18:45:25] <^d> :) [18:45:25] doesn't asking about deploy before commit just *add* work? :) [18:45:28] unless you're afraid the idea might be rejected [18:46:25] Nemo_bis: the point is if it's going to take a couple weeks to be deployed I don't want to start now because it may then need rebasing. one commit is waiting to go out already and getting nowhere [18:46:28] like my https://gerrit.wikimedia.org/r/101117 which I've no idea how's going to end up [18:46:46] jeremyb: not during holidays [18:46:46] aude: tell the people that asked for it :) [18:47:06] :) [18:47:13] aude: https://rt.wikimedia.org/Ticket/Display.html?id=6532 [18:47:26] ah sure, rebases :) probably they were invented so that people choose little frequented areas for their patches [18:48:06] Nemo_bis: so, rebase is one thing if it's one commit. but this i want to split up into more than a few commits. then they'll all need rebasing. how many times? [18:51:09] dan-nl: if done "right", you can just commit your changes to master, and it'll roll out with the normal deployment "train" on (for Commons) Tuesday morning Pacific. [18:51:09] dan-nl: we branch installed extensions from master in most cases [18:51:29] oh, cool … [18:51:42] dan-nl: the branching happens on Thursday morning (when we roll out to testwikis), and them non-wikipedias (eg: Commons, wiktionaries, etc) get the update on Tuesday [18:51:57] Reedy: around per chance? [18:53:30] greg-g, regarding the merge of gwtoolset branches into gwtoolset master … what +1's do i need? [18:53:30] for example Reedy requested a few changes which i've made but he hasn't been available ( maybe he's on holiday? ) … but tgr has given them +1's [18:54:11] dan-nl: whoever has +2, really [18:54:31] greg-g: i do at times [18:55:32] so, if i feel okay with the change and it has passed my local tests … just +2 it? [18:56:53] well, the policy is don't self-merge ideally [18:57:54] ja, thought might be the case … was hoping Reedy would be able to +2 it but i assume he's been away or busy [18:58:14] apparently :) [18:58:22] thanks [18:58:22] :) [18:59:55] wtf. anyone else see funky formatting at bottom of https://rt.wikimedia.org/Ticket/Display.html?id=6552 ? [19:06:37] aude: ^ ? [19:08:42] greg-g: maybe? [19:08:42] never seen anything like this [19:11:24] jeremyb: screenie? [19:11:43] i just took a bunch, was scared it would fix itself [19:17:35] <^d> jeremyb: I'm seeing it too, I assume. [19:17:48] i'm redacting [19:18:29] (03PS2) 10Nemo bis: [Planet] Update wikimedia.fi URL [operations/puppet] - 10https://gerrit.wikimedia.org/r/103047 [19:18:29] <^d> Weird indentation of the datestamps? Green #'s? :) [19:19:01] well green is normal i think [19:19:10] indentation/lack of background [19:21:51] greg-g: ^d: http://i.imgur.com/dzDlv22.png [19:22:32] <^d> Yep, that's what I'm seeing, just diff colors. [19:22:32] <^d> But borked layout like that. [19:23:53] what a weird system. sending text/html wrapped in 
 instead of just text/plain
[19:24:13] 	 Content-Type: text/html; charset="utf-8"
[19:24:13] 	 
Dear Customer
[19:24:33] 	 yada\nyada\nyada
[19:24:33] 	 

[19:24:47] <^d>	 jeremyb: So do we blame the incoming e-mail? :p
[19:25:14] 	 ^d: idk yet
[19:25:14] <^d>	 Or RT for allowing such a poorly formatted e-mail to break its UI? :p
[19:25:14] 	 i guess it's easy enough to test
[19:25:35] 	 oh, definitely RT's at fault. but what made RT break?
[19:26:21] 	 ^d: at first i thought some external stylesheet was missing and i needed to reload. but that's not it
[19:27:58] 	 hrmmmm, they're inconsistent on that ticket. (att.net vs. at-t.net; substitute att for real vendor's name)
[19:36:22] 	 so, the desktop view works well enough on mobile for me but i figured something useful may come of checking that ticket on mobile. now i notice the mobile view merges all lines in a msg to one line? (strips line breaks) ewww
[19:37:02] 	 anyway, that ticket on mobile doesn't look extra bad compared to other tickets on mobile
[19:53:31] 	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[20:07:41] 	 (03PS4) 10John F. Lewis: Redirect kr.wikimedia [operations/apache-config] - 10https://gerrit.wikimedia.org/r/101220 
[20:09:14] 	 jeremyb: ^ Better?
[20:10:50] 	 yeah
[20:11:01] 	 :)
[20:11:17] 	 JohnLewis: but where's jenkins?
[20:11:35] 	 Hm. I do not know.
[20:11:50] 	 Jenkins hasn't updated that commit since PS2.
[20:11:54] 	 *patch
[20:11:58] 	 https://integration.wikimedia.org/ci/job/operations-apache-config-lint/ looks bad
[20:18:33] 	 (03Abandoned) 10John F. Lewis: Redirect m.wikidata to www.wikidata.org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/101787 (owner: 10John F. Lewis)
[20:20:34] 	 (03PS2) 10Ori.livneh: Removed scap-1skins for performance [operations/puppet] - 10https://gerrit.wikimedia.org/r/103880 (owner: 10Aaron Schulz)
[20:20:42] 	 (03PS2) 10Ori.livneh: Use delete-delay in MW_RSYNC_ARGS to make it a bit more atomic [operations/puppet] - 10https://gerrit.wikimedia.org/r/103895 (owner: 10Aaron Schulz)
[20:20:54] 	 (03CR) 10Ori.livneh: [C: 032 V: 032] Use delete-delay in MW_RSYNC_ARGS to make it a bit more atomic [operations/puppet] - 10https://gerrit.wikimedia.org/r/103895 (owner: 10Aaron Schulz)
[20:21:55] 	 (03PS3) 10Ori.livneh: Removed scap-1skins for performance [operations/puppet] - 10https://gerrit.wikimedia.org/r/103880 (owner: 10Aaron Schulz)
[20:22:15] 	 (03CR) 10Ori.livneh: [C: 032 V: 032] Removed scap-1skins for performance [operations/puppet] - 10https://gerrit.wikimedia.org/r/103880 (owner: 10Aaron Schulz)
[20:22:35] 	 hey, have you thought of moving scap outside of ops/puppet?
[20:23:16] 	 it has its own life mostly
[20:23:26] 	 what would be the benefit?
[20:23:56] 	 expand the set of +2ers?
[20:24:03] 	 (or limit them, although I presume expanding is more important here)
[20:24:37] 	 there are other ways to expand the set of +2ers :)
[20:24:37] 	 but, joking aside --
[20:24:37] 	 not without expanding the set of roots :)
[20:24:57] 	 scap is a horrible, tangled mess
[20:25:18] 	 extracting it to its own repo would be tantamount to a refactor / cleanup
[20:25:18] 	 ok
[20:25:18] 	 that's a good thing, obviously, but it's some work
[20:25:18] 	 it's also mixed between puppet and a debian package
[20:25:49] 	 unless someone removed the need for the deb
[20:26:18] 	 i think i may have a while back, i can't remember
[20:26:18] 	 i may have just moved things from one to the other
[20:27:02] * ^d  forks scap to github
[20:27:16] 	 heh
[20:47:19] 	 (03PS1) 10Ori.livneh: Explicitly close streams in serve_xml() [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104132 
[20:51:37] 	 (03PS2) 10Ori.livneh: Explicitly close streams in serve_xml() [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104132 
[20:53:38] 	 (03PS1) 10Manybubbles: Cirrus for more wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104134 
[20:56:00] 	 (03PS2) 10Manybubbles: Cirrus for more wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104134 
[20:56:40] 	 (03CR) 10Ori.livneh: [C: 032 V: 032] Explicitly close streams in serve_xml() [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104132 (owner: 10Ori.livneh)
[21:16:20] 	 (03PS8) 10BryanDavis: [WIP] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 
[21:17:11] 	 (03CR) 10jenkins-bot: [V: 04-1] [WIP] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis)
[21:39:02] 	 (03PS9) 10BryanDavis: Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 
[21:43:53] 	 (03CR) 10BryanDavis: "This requires the new deb from I748d45d" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis)
[21:45:37] 	 (03CR) 10Ori.livneh: Logstash puppet class (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis)
[21:46:47] 	 (03PS3) 10Ryan Lane: Enable a labs site override option for nova config [operations/puppet] - 10https://gerrit.wikimedia.org/r/102185 
[21:58:22] 	 (03PS3) 10Ori.livneh: Revert "Hack: cron job to clean up tifs from /tmp on app servers" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103390 
[22:09:17] 	 (03PS10) 10BryanDavis: Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 
[22:16:23] 	 (03CR) 10BryanDavis: Logstash puppet class (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis)
[22:28:34] 	 !log Stopping uWSGI on Tungsten to debug Graphite issue
[22:28:50] 	 Logged the message, Master
[22:28:51] 	 PROBLEM - Graphite Carbon on tungsten is CRITICAL: CRITICAL: Not all configured Carbon instances are running.  
[22:29:13] 	 that's me, as noted.
[22:41:33] 	 (03CR) 10Ori.livneh: [C: 032] Revert "Hack: cron job to clean up tifs from /tmp on app servers" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103390 (owner: 10Ori.livneh)
[22:47:44] 	 (03PS1) 10Ori.livneh: Print the task name as the event name, not the full entry key [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104154 
[22:48:55] 	 (03CR) 10Ori.livneh: [C: 032 V: 032] Print the task name as the event name, not the full entry key [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104154 (owner: 10Ori.livneh)
[22:49:52] 	 RECOVERY - Graphite Carbon on tungsten is OK: OK: All defined Carbon jobs are runnning.  
[22:54:31] 	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[22:55:07] 	 ori: https://gerrit.wikimedia.org/r/#/c/103872/ is trivial
[23:04:43] 	 (03PS2) 10Ori.livneh: Made scap cdb scripts only show output if anything happened [operations/puppet] - 10https://gerrit.wikimedia.org/r/103872 (owner: 10Aaron Schulz)
[23:04:55] 	 (03CR) 10Ori.livneh: [C: 032 V: 032] Made scap cdb scripts only show output if anything happened [operations/puppet] - 10https://gerrit.wikimedia.org/r/103872 (owner: 10Aaron Schulz)
[23:06:10] 	 are there any graphs for jobqueues?
[23:06:18] 	 (03CR) 10Aaron Schulz: "This was added in https://gerrit.wikimedia.org/r/#/c/103895/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103870 (owner: 10Aaron Schulz)
[23:06:44] 	 AzaToth: like https://gdash.wikimedia.org/dashboards/jobq/ ?
[23:06:55] 	 (03CR) 10Ori.livneh: "OK, let me know when." [operations/puppet] - 10https://gerrit.wikimedia.org/r/103768 (owner: 10BryanDavis)
[23:07:09] 	 or http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&r=hour&z=default&jr=&js=&st=1388185606&v=5203746&m=Global_JobQueue_length&z=large
[23:07:20] 	 thanks
[23:08:27] 	 have been some big bumps lately
[23:12:24] 	 (03PS1) 10Ori.livneh: report.py on noc.wm.o: use tungsten, not professor [operations/puppet] - 10https://gerrit.wikimedia.org/r/104158 
[23:13:07] 	 I seem to recall there was some old bug report about report.py
[23:13:32] 	 let me know if you unearth it
[23:14:00] 	 i am finally done setting up tungsten, argh
[23:14:12] 	 hm https://bugzilla.wikimedia.org/show_bug.cgi?id=16716
[23:14:20] 	 i have to wait a bit for it to accumulate enough metric data that flipping graphite.wikimedia.org doesn't make the graphs empty
[23:15:03] 	 Nemo_bis: then we can apply your log scale patches
[23:16:22] 	 what I had in mind was https://bugzilla.wikimedia.org/show_bug.cgi?id=44134 though, no idea what happened to that URL
[23:19:04] 	 ah I thought that migration had already happened, I'm sad it was more effort than expected
[23:22:20] 	 (03PS1) 10Ori.livneh: Make report.py set the default system encoding to UTF-8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104159 
[23:22:51] 	 (03CR) 10Ori.livneh: [C: 032] report.py on noc.wm.o: use tungsten, not professor [operations/puppet] - 10https://gerrit.wikimedia.org/r/104158 (owner: 10Ori.livneh)
[23:23:14] 	 (03CR) 10jenkins-bot: [V: 04-1] Make report.py set the default system encoding to UTF-8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104159 (owner: 10Ori.livneh)
[23:24:15] 	 (03PS2) 10Ori.livneh: Make report.py set the default system encoding to UTF-8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104159 
[23:25:22] 	 (03CR) 10Ori.livneh: [C: 032] Make report.py set the default system encoding to UTF-8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104159 (owner: 10Ori.livneh)
[23:26:04] 	 Nemo_bis: fixed
[23:26:49] 	 wow :)
[23:59:56] 	 !Log doing some sync script timing