[00:25:14] !log olivneh synchronized php-1.21wmf8/extensions/GettingStarted [00:25:21] Logged the message, Master [00:58:01] New patchset: RobH; "narrowing scope of regex node matching for mw servers, added mw1161-mw1200" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48056 [00:58:51] Change abandoned: RobH; "included with other changes in I74bd06b4" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47886 [01:00:21] New review: RobH; "related to RT4390" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/48056 [01:03:15] !log reedy synchronized php-1.21wmf9/languages/classes/ [01:03:16] Logged the message, Master [01:10:41] RobH, you can merge 4490 into 3617 - https://rt.wikimedia.org/Ticket/ModifyLinks.html?id=4490 [01:11:41] or ottomata who's on RT duty :) [01:12:21] New patchset: Jdlrobson; "Enable Watchlist schema in config file (EventLogging)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48060 [01:16:08] New patchset: Jdlrobson; "Enable Watchlist schema in config file (EventLogging)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48060 [01:24:29] !log disabling notifications on icinga and nagios [01:24:32] Logged the message, Mistress of the network gear. [01:30:49] nginx doesn't want to let me view Obama [01:31:38] nginx voted for romney [01:34:38] !log deactivating european bgp sessions - may cause some connectivity issues as connections reset [01:34:39] Logged the message, Mistress of the network gear. [01:36:19] !log cr2-knams reloading after upgrade [01:36:20] Logged the message, Mistress of the network gear. [01:48:14] !log cr2-knams is back up on the new code [01:48:15] Logged the message, Mistress of the network gear. [01:48:26] New patchset: Tim Starling; "Disable $wgNewUserMessageOnAutoCreate on metawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48067 [01:51:46] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48067 [02:02:30] !log tstarling synchronized wmf-config/InitialiseSettings.php [02:02:31] Logged the message, Master [02:03:19] New patchset: Reedy; "Disable NewUserMessage on meta" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48068 [02:04:12] Change abandoned: Reedy; "(no reason)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48068 [02:30:42] !log LocalisationUpdate completed (1.21wmf9) at Fri Feb 8 02:30:42 UTC 2013 [02:30:44] Logged the message, Master [02:54:50] !log LocalisationUpdate completed (1.21wmf8) at Fri Feb 8 02:54:49 UTC 2013 [02:54:51] Logged the message, Master [02:59:22] New patchset: CSteipp; "Enable Global Abuse Filters on test, mediawiki, wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48070 [03:35:49] :o [03:35:56] global abusefilter :D [05:47:50] !log tstarling synchronized php-1.21wmf9/includes/context/RequestContext.php [05:47:52] Logged the message, Master [08:08:04] Hmpf, if puppet run why wasn't the cron updated to make planet's fetches hourly? http://nagios.wikimedia.org/nagios/cgi-bin/extinfo.cgi?type=2&host=zirconium&service=Puppet+freshness [12:21:32] hello [14:39:34] New patchset: Hashar; "contint: php-luasandbox to latest version" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48127 [14:39:54] !log gallium installed php-luasandbox manually. Gerrit change is {{gerrit|48127}}. [14:39:57] Logged the message, Master [14:40:40] New review: Hashar; "installed manually already." [operations/puppet] (production); V: 1 C: 0; - https://gerrit.wikimedia.org/r/48127 [15:07:51] paravoid, sorry to so pokey, but my nits are itchy: [15:07:52] https://gerrit.wikimedia.org/r/#/c/48041/ [15:07:52] :) [15:07:53] hahah [15:08:03] pick my nits! [15:08:22] 'sorry to be'* [15:08:26] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: srwikibooks back to 1.21wmf9 to test variant fixes [15:08:28] Logged the message, Master [15:09:30] New patchset: Reedy; "Revert "Roll back ku/sr wikis to 1.21wmf8 - langconverter breakage"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48128 [15:09:57] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48128 [15:10:34] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [15:10:35] Logged the message, Master [15:11:31] !log Set cr2-eqiad VRRP prio to 50 to transition to backup role [15:11:32] Logged the message, Master [15:12:19] New patchset: Diederik; "Replace FAKE_CACHE_STATUS with FCS to save 14 bytes per log line" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48129 [15:51:13] New patchset: Ottomata; "Logging to geoipupdate.log when running geoipupdate command." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48130 [15:52:56] New patchset: Ottomata; "Logging to geoipupdate.log when running geoipupdate command. Ensuring that geoip packages are present, not latest." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48130 [16:06:07] New patchset: MaxSem; "Advanced Solr monitoring script" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47111 [16:09:05] !log Upgraded and rebooted cr2-eqiad (both REs) [16:09:06] Logged the message, Master [16:10:15] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48130 [16:10:20] mark: would you be there next monday to finish up the review of the varnish hacks for beta cluster? :) [16:11:35] i should be [16:11:41] :-] [16:11:52] I will ping you if you are!, ,, [16:19:01] eh, am I the only one who can't SSH into fenari? [16:19:19] yes, it's just you [16:19:32] (not actually sarcastic, I checked and it worksforme fine) [16:19:46] grrr, connected after several attempts [16:20:23] weird but at least you're in [16:32:41] !log updated vumi packages in apt to vumi_0.5.0~a+143-0~ppa3 and vumi-wikipedia_0.1~a+14-0~ppa3 [16:32:43] Logged the message, Master [16:34:58] New patchset: Ottomata; "Updating to newer version of Vumi packages with metrics support." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48134 [16:37:13] New patchset: Ottomata; "Updating to newer version of Vumi packages with metrics support." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48134 [16:40:10] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48134 [16:51:40] New patchset: Ottomata; "Adding metrics config line to vumi/wikipedia.yaml" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48137 [16:51:40] New patchset: Ottomata; "Ensuring /var/log/vumi/metrics.log is writeable by udp2log" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48138 [16:52:01] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48137 [16:52:34] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48138 [16:54:54] New patchset: Ottomata; "Fixing mode of /var/log/vumi/metrics.log" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48140 [16:55:17] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48140 [17:02:05] New review: Silke Meyer; "Ping... Andrew, is this okay?" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/47585 [17:23:02] apergos: around? [17:23:13] yes, for a little bit yet [17:23:15] what's up? [17:23:24] 281 15 10.0.6.213 6000 sdn1 33.00 517 -0.16 [17:23:27] 339 14 10.0.6.212 6000 sdi1 33.00 518 0.03 [17:23:30] that's ms-be11 & 12 [17:23:36] but they're on the same rack according to racktables [17:23:40] so why a different zone? [17:24:12] I need to find out what rack it's in actually (ms-be11) [17:24:27] C2 as I see it [17:24:31] I was under the impression that old hosts were coming out and new ones slotted in place [17:24:38] but I'll check with chris on that [17:26:32] for now it won't hurt anything but it it needs to be updated I'll get that the next ring update [17:28:13] there's a reason I'm asking [17:28:30] mark wants to reboot cr2-pmtpa at some point for a junos upgrade, which means that we'll lose pmtpa for a while [17:29:02] right now that's not an option, but I was thinking of depooling ms-be1 for a while as to get a third copy somewhere [17:29:20] when did he want to do this? [17:29:24] rough guess [17:29:42] there's no hurry [17:29:46] ok [17:29:49] but with 4 zones it's worse [17:30:04] so let's get it down to three, then we can possibly do that [17:30:08] sure [17:30:23] it'll probably be a week before the next round of updates, if we're lucky sooner [17:30:49] if you plan on taking a host down then, let's coordinate so we just do that [17:31:15] I'll ping you when it's coming up on the date [17:31:31] so ms-be1 needs to be replaced, right? [17:31:34] anyway I mean [17:31:51] yes but we don't pull out hosts 100 -> 0 all at once [17:31:58] if you want we could .. what's the next one [17:32:09] ok, so here's my proposed plan [17:32:16] well ms-be12 [17:32:17] :-D [17:32:21] change ms-be11/12 to be on the same zone [17:32:29] so in a week + ms-be12 will be powered off [17:33:40] wait, ms-be11/12 are about to be depooled? [17:33:44] 12 [17:33:51] not 11 [17:33:51] or are you rumping them up? [17:33:53] they're both at 33 [17:33:58] 12 is on its way out [17:34:02] 11 is on its way back in [17:35:10] ok, so either depool both of them, upgrade cr2-pmtpa, pool both of them back in [17:35:45] or, pool both of them back, wait until they sync, depool ms-be1, wait until the cluster gets more copies, upgrade cr2-pmtpa [17:35:50] then pool back ms-be1 [17:35:58] taking them both out will be quicker [17:36:38] so next round I'll put them both to 0, wait for obj rep to clear and then the upgrade can happen [17:36:56] k [17:36:59] great, thanks [17:37:03] yw [17:38:26] apergos: you are correct..the new host are replacing the old host in the rack w/ the exception of ms-be2...that will move over 1 rack to c2-sdtpa [17:38:38] ok [18:16:31] New review: RobH; "new apaches \o/" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48056 [18:16:32] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48056 [18:25:28] cmjohnson1: morning, how goes last day in tampa? [18:27:59] Reedy: Hmmm, language variant code should surely be tested on test or test2. [18:29:01] <^demon> Hard to test for all users unless the content language is set to a language with variants. [18:29:09] <^demon> You can easily test on any wiki by swapping your language. [18:36:19] ^demon: I meant that testing on test or test2 seemed better than syncing to the live sites and then reverting. [18:41:06] Change merged: preilly; [operations/debs/python-jsonschema] (master) - https://gerrit.wikimedia.org/r/48044 [18:41:07] Change merged: preilly; [operations/debs/python-jsonschema] (master) - https://gerrit.wikimedia.org/r/48043 [18:41:14] Change merged: preilly; [operations/debs/python-jsonschema] (master) - https://gerrit.wikimedia.org/r/48042 [18:42:26] Change merged: preilly; [operations/debs/python-jsonschema] (master) - https://gerrit.wikimedia.org/r/48045 [18:50:14] robh: hi [18:50:49] heyas [18:50:49] the loop you sent me for the mac address didn't work for me [18:51:08] oh? i copy pasted it from my console =/ [18:51:25] Hrmm, I just spun up 4 new jobrunners [18:51:32] trying to see where the hell they get pooled. [18:51:50] does RoanKattouw know? [18:54:04] robh: i want to kill of srv226-234 today and replace w/remaining 9 [18:54:15] Job runners? [18:54:20] They don't get pooled explicitly [18:54:22] You just run em [18:54:31] so if i have them spun up, they are in service? [18:54:39] They don't get stuff assigned to them, they grab it themselves [18:54:42] cuz yea, i dont see any file anywhere that has the pool [18:54:43] They should be, yeah [18:54:43] cool [18:54:46] good times [18:54:48] thx =] [18:54:59] Run top and see if you see a bunch of php processes happily churning away at near-100% CPU [18:55:22] well, they started spiking cpu use as well [18:55:27] in ganglia [18:55:33] That's probably a good sign [18:55:46] yea, i see the jobs-loop.sh [18:55:54] more apache capacity going online, huzzah! [18:56:05] Yay [18:56:05] gonna need it come monday. [18:56:52] What happens on Monday? [18:57:20] wikidata crap [18:57:28] Oh right [18:57:31] not sure exactly, but it may have a spike on our apache use [18:57:32] And then Lua the Monday after [18:57:36] which is overburdened in eqiad [18:58:23] Oh BTW Rob [18:58:27] Lemme show you something funny [18:58:37] https://gerrit.wikimedia.org/r/#/c/48072/1 [18:58:44] Turns out, Parsoid is still running out of Tampa [18:59:04] oops [18:59:14] Heh yeah [18:59:20] RoanKattouw: i thought so cuz the parsoid varnish server in tampa is busy [18:59:25] but wasnt sure [18:59:32] RobH: No, Parsoid Varnish is running out of eqiad [18:59:38] Celsus is "busy" because of the power_saving bug [18:59:42] ohhhh [18:59:45] yea. [18:59:50] But the Varnish server in eqiad has a VCL pointing to Tampa [19:00:01] bleh [19:00:10] Yeah :( [19:00:20] I was looking for Sue's request and didn't see it in any of the eqiad logs [19:00:28] So then I started reading pmtpa logs, and they made me sad [19:02:09] New patchset: Brion VIBBER; "Add Wikipedia Zero IP ranges for Morocco" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48149 [19:02:10] ya need me to do something to get it moved to eqiad, or is it puppet stuff? [19:02:21] (if its puppet stuff i can review a change) [19:02:57] !log mw1161-1164 in service as eqiad jobrunners [19:02:59] Logged the message, RobH [19:03:43] It's just puppet stuff [19:03:55] Parsoid in eqiad is up, the VCL is just misconfigured [19:04:16] My puppet change will regenerate the VCL to point to the right place, and traffic will magically move, and unicorns will rejoice and see rainbows [19:04:18] (hopefully) [19:04:30] rainbow unicorns yay! [19:05:24] https://gerrit.wikimedia.org/r/#/c/48072 [19:07:47] * ^demon got pinged for rainbows and unicorns. [19:08:02] haha [19:08:09] Do you ping on unicorn? [19:08:46] <^demon> I was kidding, heh. [19:10:37] Too bad. That's stuff that needs prompt reaction. [19:11:02] which one, the rainbows or the unicorns? [19:12:48] <^demon> Both! [19:13:25] I could see it for ponies [19:13:38] unicorns > ponies [19:13:52] so not true [19:13:55] narwhal + pony = unicorn [19:14:22] apergos: http://www.weebls-stuff.com/songs/Narwhals/ [19:14:37] in a bit (doing a timing test) [19:22:16] RoanKattouw: So just to reinterate, i can take down and fix celsus yes? [19:22:28] Yes [19:22:38] As long as you don't want it for investigation purposes, go ahead [19:22:55] nah, we know exactly the issue and we disable it now on most systems [19:23:00] OK [19:23:01] any that have it enabled were simply missed [19:23:12] !log celsus rebooting to fix power mgmt cpu spike issue [19:23:13] Logged the message, RobH [19:23:25] It would also be nice if https://gerrit.wikimedia.org/r/#/c/48072 was deployed. That will affect cerium to send traffic the right way, and will also affect celsus slightly [19:23:34] (It'll regen the file on celsus but it shouldn't actually change there) [19:24:50] RoanKattouw: uhhhh, wait, you want that live before celsus has downtime? [19:24:56] cuz i already sent the command when i logged it ;] [19:26:42] No it's fine eitherway [19:26:48] cool [19:27:10] Just saying that celsus is one of the boxes that should have puppet kicked when that is deployed [19:27:14] But the most important one is cerium [19:28:16] RoanKattouw: just fyi, the new dells power mgmt comes set to DAPC (dell active power control) [19:28:25] when it needs to be os controlled power control [19:28:32] so now chris and steve change them as they come in [19:28:40] Ah OK [19:28:52] cmjohnson1 & sbernardin ^ (i hope you do ;) [19:29:18] robh: yep! [19:29:26] we change that setting [19:29:26] coolness [19:29:32] celsus fixed now, its booting back up [19:30:00] Got it... [19:33:42] !log celsus back online [19:33:43] Logged the message, RobH [19:43:12] New patchset: Cmjohnson; " Adding macs for mw85-125" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48154 [19:43:13] !log mw1165-1188 os installed for apache use, initial puppet runs in progress [19:43:14] Logged the message, RobH [19:43:48] cmjohnson1: want review? [19:44:18] not on this...unless you really want do ;-] [19:44:33] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48154 [19:44:43] cmjohnson1: you can handle merge on sockpuppet [19:44:48] k [19:44:51] RobH: this video you linked.. it never ends right? [19:45:02] pretty sure it never ends [19:45:05] its just a silly song. [19:45:29] cmjohnson1: https://rt.wikimedia.org/Ticket/Display.html?id=4495 [19:45:32] is that on a foundry or juniper? [19:45:35] if its foundry i can handle it. [19:45:37] foundry [19:45:42] ok, i'll steal and handle in a bit [19:46:16] I know you only like it cause of the gratuitous cthulhu reference [19:46:42] New review: preilly; "Please have someone from operations review this change-set." [operations/puppet] (production); V: -1 C: 0; - https://gerrit.wikimedia.org/r/46672 [19:51:10] robh: have you given any thought on how to break these up? [19:55:13] cmjohnson1: break up the assignments? Not yet, I did it for eqiad first [19:55:29] have to take inventory of how many are there total, and what percentage of each are what service group [19:55:42] and then ensure none are overloaded compared to the others [19:55:52] and distribute them evenly to keep same ratio in place [19:56:14] example: https://rt.wikimedia.org/Ticket/Display.html?id=4491 [19:56:19] my distribution for eqiad. [19:56:32] don't add anymore for bits, for the record [19:56:45] notpeter: in tampa or both? [19:56:50] i have slated to add one to bits in eqiad. [19:57:06] I would say don't bother [19:57:07] I think [19:57:08] lemme check [19:57:24] bits is 4 sytems in eqiad currently [19:57:27] at a load of 6% [19:57:33] i only added 1 to it so bits has a single row C server [19:57:35] yeah, 4 in both is fine [19:57:43] in case we lose a rack with one of the 2 server sets [19:57:49] but i can easily just roll it into something else [19:57:53] meh, up to you [19:58:00] I understand your reasons [19:58:04] i'll toss in image instead [19:58:08] cuz its overtaxed compared to bits [20:00:45] Is there any objection to me taking srv226-234 offline today? looks like they are all apaches [20:01:32] umm, lemme see [20:04:59] cmjohnson1: So, if you want to bring them down, that is fine, but you need to do all the footwork first. [20:05:18] decomissioning.pp, removal from all node lists in /usr/local/dsh/node_groups [20:05:21] yeah i know [20:05:25] removal from site.pp, annnnnd [20:05:27] hrmm. [20:05:34] site.pp [20:05:41] cmjohnson1: If any of those are the ganglia aggregator for that group, you need to change it to another set of apaches [20:05:50] so that means changing site.pp and ganglia's config [20:05:53] ganglia.pp [20:06:24] but yes, if they are all general apaches, you can pull them offline to replace them [20:06:27] okay..it doesn't appear they are but I will check [20:06:33] (also dont forget the pybla files in home/w/conf/pybal [20:07:53] * hashar adds "pybla" to the list of typos to watch for [20:08:41] notpeter: so if i change site.pp to make the new jobrunners into general apache [20:08:45] they dont need reinstall? [20:10:36] !log decommissioning srvs226-234 [20:10:37] Logged the message, Master [20:10:53] cmjohnson1: first thing of that is remove from node lists [20:11:00] New patchset: RobH; "new jobrunners/bit/image apaches should be apache/api per py" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48157 [20:11:05] as if they arent, any sync scripts will throw errors for them [20:11:17] RobH: make sure that the jobrunner shit is uninstalled [20:11:44] might just be easier to reinstall, tbh. you're right [20:13:11] !log mw1161-1164 offline for reallocation from job to general apache [20:13:12] Logged the message, RobH [20:13:35] i have multiple tabs running puppet runs in for loops for all my new apaches =P [20:17:28] RobH [20:17:35] ? [20:17:41] i recently discovered iterm2 multiple terminal input [20:17:46] not sure if you use iterm2 [20:17:52] but it made my multi machine puppet runs much nicer [20:18:05] i just use stock os x terminal [20:18:35] I meant to commit my change and sign the puppet keys last night [20:18:40] so they would sit and spin overnight and get updates [20:18:43] but i ran out of time. [20:18:45] yeah, i used to use it, by iterm2 is so goooood [20:18:50] free? [20:18:52] yup [20:18:54] uhh [20:18:54] thikn so [20:19:02] yeah free [20:19:17] split screens on it are really nice too [20:19:26] bookmarked, i used iterm a long time ago [20:19:34] when os x terminal did some odd crap that bugged me [20:19:42] but i migrated off it slowly back to stock and have not bothered to update [20:20:21] New review: Andrew Bogott; "I'm working with S on this as part of his levelup for this quarter. We've been discussing things of..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/46672 [20:20:36] !log disregard errors for mw1161-1164 [20:20:37] Logged the message, RobH [20:21:07] yeah iterm is terrible [20:21:17] i never understood when people praised it [20:21:20] but they were talking about iterm2 [20:21:39] it may have been iterm2 i was using, this was pre-wiki [20:21:43] 2005 [20:22:01] when i made the transition from my personal mac to wmf mac, i didnt do a time machine move [20:22:06] just reinstalled crap i needed and skipped iterm [20:22:10] New patchset: Cmjohnson; "Decommissioning srv226-234 removing from site.pp/dhcpd file and adding to decommission.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48160 [20:23:37] cmjohnson1: if i +1 its good, i just dont +2/merge so you can handle. [20:23:43] i +1'd it [20:23:51] cool [20:23:52] thx [20:24:55] cmjohnson1: wait [20:25:01] you are overwriing my change [20:25:23] what? [20:25:45] New review: RobH; "i take it back, look at current site.pp and this version, and you are undoing my recently merged cha..." [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/48160 [20:26:03] cmjohnson1: You are going to have a merge conflict as it stands now, or entirely undo my recently merged changes [20:26:13] you need to rebase the site.pp change cuz its off. [20:26:35] (i put 1161-1164 into apaches regex and 1199-1200 into api) [20:26:49] now, im not sure if you should bother attempting a merge and correction [20:26:53] or what. [20:27:10] we should ask someone who knows more about git than me. [20:27:13] ^demon: ! [20:27:26] i tried to rebase and i get an error that says change is up to date [20:27:29] so chris has an old version of site.pp and a newer version is committed and merged [20:27:42] <^demon> Rebase against origin. [20:27:47] <^demon> `git rebase origin/production` [20:28:05] cmjohnson1: cherry pick out your change, then do that (i think) [20:28:06] ? [20:28:22] ^demon: since he already git review'd, he should chrry pick out right? [20:28:32] <^demon> You can cherry pick it locally yeah. [20:28:45] cmjohnson1: welcome to git for realz. [20:28:45] <^demon> Rebase, fix conflicts, then submit back to gerrit. [20:28:56] New patchset: Ottomata; "Allowing udp2log instances to configure recv-queue in puppet." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48161 [20:29:05] k [20:29:38] when i have git questions, i hunt down ryan or chad. [20:30:53] RobH: [20:30:54] http://www.youtube.com/watch?v=kkMLMYT6y7w [20:30:56] :) [20:30:57] hehe [20:34:07] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48161 [20:38:16] ^demon so i hosed it up pretty good and can't even cherry pick [20:38:31] <^demon> What do you get when you try cherry picking? [20:39:10] cherries? [20:39:17] New patchset: Ottomata; "Lowering receive queue for Vumi udp2log instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48163 [20:39:51] well i done f'd it up...i cherry picked initially and than rebased but did not see rob's changes so i blew out my repo and tried it again and now i get error could not apply "cherry pick" [20:40:25] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48163 [20:44:54] hey notpeter - I got bored and wrote a completely new monitoring script at https://gerrit.wikimedia.org/r/#/c/47111/ [20:46:02] LeslieCarr: Whats the command to pull serial # info out of a juniper? [20:46:13] (I need to get them off all the 4500s, as we have not populated racktables with it) [20:46:28] i have to tie asset tags to every line item on the juniper warranty renewal quote/order for accounting [20:46:44] show chassis hardware ? [20:47:22] robh: just going to abandon the change and do it again [20:47:49] New patchset: Ottomata; "Using IP rather than fqdn to send vumi metrics" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48165 [20:49:22] hashar: that works ;] [20:49:24] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48165 [20:49:26] thx dude [20:49:47] RobH: you can get more details [20:50:00] but I can't remember the exact sub commands off hand [20:50:04] Change abandoned: Cmjohnson; "messed it up pretty good...." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48160 [20:50:15] the juniper CLI is very nice, you even get man pages for each commands [20:50:18] hashar, would love a review of this when you have a sec: [20:50:18] https://gerrit.wikimedia.org/r/#/c/48041/ [20:50:25] yeah sure [20:50:29] oh [20:50:34] THERE IS NO UNIT TESTS!! [20:50:43] beehawwaawwww? [20:50:51] puppet unit tests?! you are cRzyman! [20:50:52] let me drink some water and have a cigarette :-] [20:50:58] k [20:50:59] ottomata: that is on my goal list :-] [20:51:02] COOL [20:51:03] let's do it [20:51:05]