[00:18:47] New patchset: Asher; "varnish3 pkg wants to restart with the varnish3 init, while puppet installs to just init.d/varnish breaking pkg upgrades on hosts that reference geoip libs in their vcl this is a hackish workaround that should work on a second puppet run, until the pkg is" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1821 [00:19:04] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1821 [00:19:04] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1821 [00:19:40] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [00:23:03] !log mw1002 coming down for hw testing rt 1656 [00:23:04] Logged the message, RobH [00:23:48] !log correction for typo, mw1102, not mw1002 [00:23:50] Logged the message, RobH [00:26:16] notpeter: ping! [00:26:30] notpeter: could you help OrenBochman out in -labs ? [00:27:30] PROBLEM - Host mw1102 is DOWN: PING CRITICAL - Packet loss = 100% [00:36:10] RECOVERY - Host mw1102 is UP: PING OK - Packet loss = 0%, RTA = 26.45 ms [00:38:56] !log db1004 pd8 set to offline per rt 2140, will place call to dell for replacement [00:38:58] Logged the message, RobH [00:45:50] PROBLEM - RAID on ms1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:52:10] PROBLEM - Disk space on mw68 is CRITICAL: DISK CRITICAL - free space: /tmp 51 MB (2% inode=95%): [00:57:31] im goin home. [01:11:36] New patchset: Asher; "use varnish 3.0.2 on all hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1822 [01:11:51] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1822 [01:11:51] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1822 [01:16:29] what's the command for the menu on our serial console ? [01:21:17] how do I restart apache ? [01:21:25] LeslieCarr: you mean ctrl-\? [01:21:26] service apache2 restart [01:21:34] that gets you out of console and back to the prompt. [01:21:43] maplebed: i mean the equivalent of ts_menu if you've ever used cyclades [01:22:11] ah. that I don't know. sorry. [01:31:08] !log all varnish servers have been upgraded to 3.0.2 [01:31:10] Logged the message, Master [02:02:32] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Tue Jan 10 02:02:25 UTC 2012 [02:14:13] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [04:16:35] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:21:45] RECOVERY - Disk space on es1004 is OK: DISK OK [04:39:21] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:00:41] RECOVERY - Disk space on mw68 is OK: DISK OK [06:23:17] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [06:50:28] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [06:53:38] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [10:01:16] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 457027 MB (3% inode=99%): [10:02:56] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 448463 MB (3% inode=99%): [10:25:16] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:11:37] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [12:23:48] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [13:28:05] 24 CPUs in the new mw* servers? Wow, pretty nice [13:28:16] Logical CPUs that is [13:41:38] Ok, why can't I ssh into mw1001 [13:42:12] Stupid thing [13:42:44] yup [13:42:59] johnduhart, x5650 @ 2.67GHz [13:46:36] I suspect mw1001 isn't serving MW then [14:22:29] Reedy: `nc -v $host 80` ? [14:23:24] can you even ping it? [14:23:44] yeah [14:24:00] it asks me for a password [14:24:05] oh [14:24:08] huh? [14:24:24] neither of those things should ever ask for a password ;-P [14:24:32] Hence me saying oh immediately after :p [14:24:45] i guessed... :) [14:24:46] no webserver seemingly running on it [14:25:01] quite possible they are still idle [14:34:54] hi Reedy, i can login to mw1001.. any problems? [14:35:08] You've not got a restricted account ;) [14:35:23] it's asking for a password, which means it's not accepting my ssh key [14:35:42] I'm guessing as it's not in production, it's not got the user groups for mortals etc on it [14:41:48] Reedy: yeah, it does not seem to appear in site.pp [15:58:44] robh: r u around [15:58:50] yep [15:59:05] regarding srv199 http://rt.wikimedia.org/Ticket/Display.html?id=2209 [16:00:04] what he is asking for you cannot run [16:00:17] how can you run DEST when it wont go past boot? [16:00:30] you have a usb key right? [16:00:34] yes [16:00:35] you can save the report with that [16:00:42] so basically, boot off the DEST cd [16:00:50] it comes with servers, or you can download the newest version [16:00:54] in face, he links old version [16:00:56] fact [16:01:37] oh, wait, nm [16:01:44] in fact that is 6.5 i have 6.4 [16:01:50] so basically, run the DEST [16:01:59] k [16:02:06] it will save the report to /tmp/data/DEsoandso [16:02:30] i knew how to do this, but I asked the tech to provide me with the full instructions nicely written for you actually [16:02:34] check out ticket 1549 [16:02:44] its my ticket for a server in eqiad, but it has the instructions on how to do this [16:02:53] okay..cool that helps [16:02:55] basically you insert the usb stick and mount it, and copy the zip file over [16:03:03] its what I do for all DEST tests [16:03:19] lets you get the copy locally to your laptop to attach in RT [16:03:48] i would get comfortable doing this if i were you, becasue its basically how you tend to have to do half the dell returns [16:03:54] on another dell note...srv199 new hdd should be here tomorrow [16:03:58] they are getting more and more a pain in the ass, requiring the dest report [16:04:08] and the damned thing takes 15 to 20 to run [16:04:25] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1470s [16:04:28] i ordered those labels for ya [16:04:34] maybe the output of "racadm gettracelog" or similar [16:04:34] i got more than you asked for, 5 carts of each kind [16:04:45] great thx [16:04:46] mutante: the dest report is huge [16:04:51] pulls all the sysinfo [16:05:09] usually they used to not require it, and just having the basic drac log was enough [16:05:20] and if the server is live, and we dont want to bring it down, i will insisit that they skip the DEST [16:05:32] but if the system is offline, its usually easier to just pull the report than argue with them [16:05:36] that is how i get around the hdd replacement issue [16:05:46] i tell them it is an active server and we cannot take it down [16:05:53] yep, just saying log output from mgmt via racadm is an option if it doesnt boot anymore [16:05:56] yea, if its a hdd and the system is stil lup, then they will just accomodate us [16:06:07] but no mac means not working [16:06:08] mutante: ahh [16:06:14] cmjohnson1: well, it sounds like this sytem isnt booting though [16:06:20] so i have no idea how you are going to get it to run DEST [16:06:30] it is not making it past mpt boot process [16:06:39] then how are you going to run a boot cd? [16:06:57] if you cannot run it, they need to correct how they ask for the info. [16:07:04] idk yet...but i believe i have to at least give it the ol' college try! [16:07:10] no you dont [16:07:16] if it wont boot to pxe or hdd boot [16:07:18] it wont boot to cd [16:07:24] dont waste time downloading and burning one for that [16:07:33] if you kept a cd out of a delivery, it has this app on it [16:07:39] but its a waste of time [16:07:41] its not booting. [16:07:51] you dont have to try everything they tell you [16:07:55] they either didnt listen [16:07:59] or they didnt understand what you said [16:08:09] no i told him [16:08:23] exactly what it does [16:08:31] if you have another dell cd there rather than wasting time burning another [16:08:34] then try to boot off it [16:08:42] but if its not posting, your basically wasting time. [16:08:43] i have a few of the new ones [16:09:04] go ahead and try if you want to be able to tell them you tried, but i would call them back and insist that what they are asking for wont work [16:09:08] that we can run them some drac output [16:09:11] as mutante pointed out [16:09:14] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1760s [16:09:21] but asking for us to boot off a cd when the system wont boot off anything is silly [16:10:07] agreed [16:28:30] RobH: are the c series expected to be easier to do these diagnostics on remotely? e.g. the DEST thing above [16:28:42] nope [16:28:47] ;( [16:28:49] dest is just dell utility cd [16:28:54] boot and run, cannot get easier than that ;] [16:29:23] the c series is mostly that its more standardized things [16:29:24] like intel nics [16:29:40] and chipsets tend to run to more linux compatible stuff [16:29:59] intel over broadcom, standard lsi controllers that allow the software to access them rather htan just hardware raid [16:30:05] R series costs more [16:30:20] since it has things built in to do windows controlling and deployment [16:30:30] and one screen to many type items, but again need windows [16:32:22] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [16:36:31] so c series seems like it is marginally cheaper with less of the features that we will never use. [16:36:59] also the one we are using for swift is pretty normal config, and other folks are using it for exactly what we want to use it for [16:47:00] anyone want to look at my rsvg package? Tim said he couldn't compile it but it WFM [16:48:01] mutante: could you look at it? I want to get it deployed asap. [16:52:23] robh: can you recommend a good way for me to get MAC addys on the new ms-fe and no ident servers using mgmt [16:52:51] what ticket? [16:53:45] 2211 [16:54:15] no name for ticket ;p [16:54:41] http://rt.wikimedia.org/Ticket/Display.html?id=2211 [16:56:06] hexmode: _look_ at, sure, where? i dont know what it's doing though at all yet [16:56:15] cmjohnson1: btw, you can use !rt :) [16:56:21] ok, the best way to get this info is via drac [16:56:32] cmjohnson1: so in drac, racadm getsysinfo [16:56:36] it shows the MAC for all nics [16:56:38] mutante..cool thx [16:56:56] !rt 2211 [16:56:56] http://rt.wikimedia.org/Ticket/Display.html?id=2211 [16:57:00] eww [16:57:03] have it link https! [16:57:19] !rt is https://rt.wikimedia.org/Ticket/Display.html?id= [16:57:19] Key exist! [16:57:35] ehm..:) [16:57:37] !help [16:57:37] want docs? ask for "!wm-bot". all keywords? try "@regsearch .*" [16:57:53] !wm-bot [16:57:53] http://meta.wikimedia.org/wiki/WM-Bot [16:58:15] !rt del [16:58:16] Successfully removed rt [16:58:20] !rt is https://rt.wikimedia.org/Ticket/Display.html?id= [16:58:20] Key was added! [16:58:25] mutante: https://rt.wikimedia.org/Ticket/Display.html?id=2190 ... only look at my diff, not the tarball . the diff should be applied to svn deb [16:59:13] mutante: tim gave instructions for making sure it was acceptable, so if you could test it and deploy, that is what I really need [16:59:20] !rt del [16:59:21] Successfully removed rt [16:59:24] !rt is https://rt.wikimedia.org/Ticket/Display.html?id=$1 [16:59:24] Key was added! [17:00:17] !rt 2190 [17:00:17] https://rt.wikimedia.org/Ticket/Display.html?id=2190 [17:00:21] :) [17:00:23] robh: duh! I don't know why I didn't think of that [17:01:14] huzzah [17:01:19] https forever! [17:04:45] hexmode: hmm.ok. i see the instructions Tim left.. i can try ..but that might also take a while and if Tim couldnt build it on lucid..uhm.. somehow expecting problems. it might keep me from working on the bugzilla sitemap problem and puppetizing bz. is "as soon as possible" = really urgent ? [17:05:35] mutante: I'm confused about why tim couldn't build it as I used lucid to build it [17:06:35] hexmode: with your compiled version, could you pass the test Tim suggests? [17:07:13] mutante: regarding "really urgent"... there is a problem on commons with svg that was introduced when we upgraded to lucid, but, yes, if you have other priorities the problem has been there a few months. I just think we should fix the problem we introduced. [17:07:32] hexmode: i can try that for you if you want me to, with your binary though [17:08:16] mutante: I'll fire up my lucid instance and try... shouldn't take too long and the binary is there anyway [17:09:15] hexmode: ok, i suggest to just append the test result to the ticket, to investigate compile problems we would need a problem description or error from Tim anyways..right [17:10:18] hexmode: and you're right about "both compiled on lucid", so if it worked for you... [17:11:30] mutante: I'm doing 50 things here, so I'm confused now: do you want me to test tim's file? [17:12:26] hexmode: no, test your binary vs. the unpatched svg. "In the unpatched SVG, the output image should show the included image. With the patched SVG, the image should be missing in the output and strace should show no attempt to open the file." [17:13:37] Tim said if it does that he is suggesting to deploy it [17:17:38] about the bot alias, you can also do this: !rt | [17:25:19] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [17:27:09] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [17:31:29] RECOVERY - RAID on ms1002 is OK: OK: State is Optimal, checked 2 logical device(s) [17:39:08] mutante: tested stock and patched... got expected results :) [17:39:17] cool:) [17:39:38] robh: Dell is sending out a new board with a tech for tomorrow [17:39:47] a tech? [17:39:57] do you think you need a tech? [17:40:07] (if you want one thats fine, but know its not required) [17:40:21] but its included if you want it, doesnt cost anything [17:40:28] just means you haver to be on site with them and such [17:40:37] cannot leave them alone even for a moment [17:41:06] (seriously, not even a second, since we dont know them and they could access things they shouldnt) [17:41:20] right..i told them I do not want the tech. [17:41:56] yea have them dispatch the part fedex overnight [17:42:01] its what they owe us for the warranty [17:42:15] if its a new mainboard, also tell them to include new cpu thermal paste and wipes and such [17:42:22] they will then include all that to make it easy [17:45:01] done. [17:48:56] RobH: i was thinking maybe an option to boot off ISO from nfs/http. then you could run their "CD" remotely if you also have remote kvm. anyway, the most recent DRAC i'm familiar with is 4+ years old i think. and i've never done a return. (didn't have to IIRC but doesn't matter because they were mostly out of warranty) [17:49:17] there is an option to connect virtual media [17:49:25] but i have not messed with it [17:49:31] its been on drac for a long time [17:49:39] i think i maybe only tried that option once [17:49:43] in fact, if you turn it on it will mess your boot order (obvoiusaly) [17:49:53] so we ensure its off of course [17:50:00] I've done it with HP stuff before now [17:50:03] but its a flag in drac, can bring online and such, but yea, never messed with it [17:50:17] at other datacenters we just had the pxe boot option for utitlities [18:19:36] cmjohnson1: we dont have the old serial console server from row A do we? [18:19:43] i dont think we do, and if not its not racked [18:23:41] no, it has been re-used [18:23:50] i have a 24 port serial console available [18:24:22] robh: [18:24:54] cmjohnson1: reused? huh, where? [18:24:58] i really dont want to use that thing [18:25:04] we have an opengear in both datacenters, no reason to [18:25:15] opengear is what we use elsewhere now as well [18:25:40] right..i have a ticket for a new one [18:25:42] cmjohnson1: https://racktables.wikimedia.org/index.php?page=object&object_id=233 shows it reused [18:25:47] not reused i mean [18:26:06] it was in row a so i thought it was gone. [18:26:12] i dont want it in a rack, is it? [18:26:20] let's sell it on the street corner :) [18:26:41] we could get a dollar for it :-P [18:26:51] let me check... [18:27:06] i am hoping its not reused, since racktables doesn't say its being used. [18:30:30] robh: no...that is not being re-used...i was thinking it was something different [18:30:36] i don't even see that here [18:30:41] good good =] [18:30:46] New patchset: Lcarr; "Pushing ganglia aggregator correction for misc pmtpa" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1823 [18:30:49] glad its gone [18:31:25] hrm, might be able to trade it on craigslist for a sandwich [18:31:28] then at least you'd get lunch [18:31:35] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1823 [18:31:35] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1823 [18:32:11] !log restarting gmetad on nickel [18:32:13] Logged the message, Mistress of the network gear. [18:33:08] sandwich sounds good! [18:56:08] RECOVERY - Puppet freshness on storage3 is OK: puppet ran at Tue Jan 10 18:55:58 UTC 2012 [19:21:28] Ryan_Lane: can we have one more machine for deployment in labs for search? [19:22:46] make it yourself ;) [19:22:57] I don't know how! [19:23:24] calling notpeter: you are wanted in -labs! [19:25:02] hexmode: it's about time to learn how, then eh? [19:25:18] and please use -labs for asking these questions [19:25:38] Ryan_Lane: np... was just trying to find you [19:25:42] * hexmode goes back [19:25:58] heh. I'm the one who made the channel, you'd think I'd be in there :D [19:35:35] :-D [19:45:08] hey guys, i am going to shut down the ganglia collection on spence [19:45:36] !log stopping gmetad on spence and unmounting the tmpfs drive [19:45:38] Logged the message, Mistress of the network gear. [19:55:54] LeslieCarr, can we remove the /ganglia from http://ganglia.wikimedia.org/ganglia/ ? [19:56:31] dislike it ? I did it so we can have multiple ganglia version running at once easily ? [19:57:07] Why has formey got a load average of 21.56, 26.55, 20.42 o_0 [19:57:30] http://ganglia.wikimedia.org/ganglia/?c=Miscellaneous%20pmtpa&h=formey.wikimedia.org&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [19:57:33] Oh, it's swapping [19:58:13] LeslieCarr, just looks a little neater IMHO, but if there's reason for doing it, sure [19:58:56] rats!! [20:00:41] formeys load looks sane now... [20:02:46] LeslieCarr: maybe ganglia.w.o/v3 by default? or something else that's short but not the root [20:04:10] length isn't even that bad. i think it's just the repetition that's annoying [20:04:22] ah [20:12:56] hi binasher! [20:13:38] hey diederik [20:13:55] New patchset: Lcarr; "changing name of ganglia redirect making the tmpfs mount puppetized" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1824 [20:14:13] binasher: would you have some time to deploy the glam filter? [20:14:27] Reedy/jeremyb that config change look nicer to you ? [20:15:50] LeslieCarr: no compalaint here. just make sure the root redirects :) [20:16:19] diederik: i want to benchmark the python vs. a few other simple non-c possibilities (mawk, perl) first, since its intended to be deployed indefinitely [20:16:26] LeslieCarr: but i hate teh way the gerrit bot leaves no indication that there was a \n in commmit msg [20:16:49] binasher: is this upload squids? (re diederik) [20:17:00] binasher: sure, we can also use pypy to speed up the script [20:17:13] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1824 [20:17:14] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1824 [20:18:05] jeremyb: yeah, udplog filter from upload squid data [20:18:30] binasher: woot. will be in labs and/or puppet? [20:18:57] (before or in addition to prod) [20:19:45] jeremyb: can we run filters in lab that listen to upd2log? [20:20:44] diederik: i guess... certainly we've proved that UDP works in labs. it's just vanillaish lucid virtualized [20:21:00] jeremyb: will be in puppet, won't be in labs [20:21:40] i thought that labs was shielded away and couldn't listen to that port [20:22:29] diederik: labs doesn't get the production log traffic [20:22:50] i see [20:23:42] diederik: you can have both ends of the UDP traffic (the squids and the log daemons?) in labs and then they can definitely talk to eachother [20:24:17] and what kind of traffic can we log? [20:25:09] i don't follow... maybe you're not familiar with labs [20:25:54] it's a virtualization cluster. (based on nova) you can make a linux instance or 5 instances and model prod machines [20:32:41] I know, but binasher said that there is no production log data available in labs, so my question is, what kind of data can be logged in the labs environment [20:32:57] oh, i have no idea about that [20:33:14] and now i just misread what you just said :) [20:37:29] diederik: if you setup squids and apaches and whatever daemon listens to the UDP all in labs then you can send reqs to teh labs squids. maybe with ab or whatever way you like. [20:37:45] diederik: anyway, you don't *have* to use labs... i was just curious [20:37:53] i love labs! [20:57:20] binasher: super thanks! [20:57:53] np :) i'll ping you tomorrow when there's a glam log being written in prod [21:04:50] woot [21:17:51] robh: srv199...did eventually go through post but it took forever. About an hour to get my boot option window. I am not 30 minutes into trying to read the optical drive. [21:18:03] Just FYI...getting a new board tomorrow. [21:20:57] i wouldnt bother troubleshooting it with the old board [21:21:03] its pretty much waste of time [21:22:13] no...i am not...i just left it running w/out shutting down and noticed it made it to boot options so I thought I give it a go to produce the dist report [21:28:10] New patchset: RobLa; "Telling the world mediawiki rocks cuz it duz" [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/1826 [21:32:34] New review: Catrope; "Clearly, this breaks lots of things :)" [test/mediawiki/core] (master) C: -1; - https://gerrit.wikimedia.org/r/1826 [21:36:08] New review: Catrope; "This also needs to support logging to different channels for different projects. I just noticed revi..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/1794 [22:09:12] New patchset: Lcarr; "Fixing redirect on nickel" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1827 [22:10:13] doh i broke ganglia redirect, fixing now [22:10:25] !log broke ganglia redirect on nickel, fixing with next push [22:10:26] Logged the message, Master [22:11:01] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1827 [22:11:02] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1827 [22:16:22] join #wmfstaff [22:16:41] * Jamesofur sighs [22:16:55] i think nagios is responding a little bit more quickly now that it doesn't have ganglia also taking resources [22:21:29] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [22:25:59] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Tue Jan 10 22:25:50 UTC 2012 [23:22:04] RobH: did you hook up the management port on psw1 ? [23:22:08] i am not seeing it up [23:29:59] New patchset: Asher; "initial carrier tagging code for wikipedia zero" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1828 [23:30:10] preilly: ^^^ [23:30:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1828 [23:30:20] binasher: thanks! [23:30:46] New review: preilly; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/1828 [23:39:13] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1828 [23:39:14] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1828 [23:43:24] preilly: should be live now [23:43:34] binasher: okay, cool