[00:09:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:14:45] New patchset: Tim Starling; "Send fatal errors to mxircecho on nfs1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19408 [00:15:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/19408 [00:15:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.643 seconds [00:15:51] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19408 [00:50:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:26] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [01:04:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.016 seconds [01:06:32] !log restarting apaches to get new wmerrors configuration [01:06:40] Logged the message, Master [01:11:14] PROBLEM - Apache HTTP on srv196 is CRITICAL: Connection refused [01:36:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:31] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 216 seconds [01:40:39] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [01:47:06] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 612s [01:47:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.012 seconds [01:51:27] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:52:03] RECOVERY - Apache HTTP on srv196 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.041 second response time [01:55:03] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [01:58:39] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 17s [01:59:15] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 18 seconds [02:14:33] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [02:21:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:56] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.015 seconds [02:44:36] any op around? need to send a big file [02:46:59] TimStarling: can I send it to you? [02:47:36] isn't there a procedure for that? [02:47:47] don't know [02:52:04] you want to upload it to commons? [02:54:27] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [02:58:46] TimStarling: no, it is a bed file [03:02:39] TimStarling: Most large file uploads involve a Bugzilla ticket and a shell user slurping the content from a public repo, I think. [03:02:55] I don't think there's any formal procedure, though. [03:04:10] what's a bed file? [03:04:52] *deb [03:05:18] how big is big? [03:06:25] 25mb [03:09:27] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [03:10:36] is this a file that you created? or did you download it from opensiddur.org? [03:12:23] New patchset: Tim Starling; "Install LuaSandbox extension on MW apaches" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19425 [03:13:05] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19425 [03:20:42] TimStarling: I built a package for the sources there [03:20:55] https://bugzilla.wikimedia.org/show_bug.cgi?id=39324 [03:22:00] can you just send me the debian directory? [03:22:45] sure [03:23:06] any address/server? [03:24:27] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [03:24:46] TimStarling: scp would be best [03:25:05] it should only be a few KB, you can send it by email [03:25:16] 8kb [03:25:24] ts@wikimedia.org? [03:26:35] tstarling@wikimedia.org [03:28:25] sent [03:31:58] you used Open-Siddur-Project-Hebrew-Font-Pack-1.8.zip ? [03:33:39] yes [03:34:33] I removed one non-free font, and updated version for some [03:39:26] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [03:42:52] have you tested it? the directory layout looks unconventional [03:44:28] also the "GNU FreeFont" fonts are duplicates of fonts that are already available on the system [03:45:04] I installed it locally [03:45:41] they aren't duplicate, there are updated version (in case of using 10.04) [03:53:45] so how would rsvg know which copy to use? [03:55:38] TimStarling: re wikimedia-fonts pkg: wouldn't it be better to just get everything we need into debian/ubuntu and use our local repo until they get in upstream? [03:55:51] or is wikimedia-fonts just a meta package? [03:56:35] TimStarling: I prepared a package, you may use it or not. you are the sysadmin [03:57:03] wikimedia-fonts was just in our local repo until recently, when it was removed because the relevant fonts were upstream [03:57:49] so what would be the verdict? [03:58:05] normal fonts packages seem to have a /etc/fonts/conf.d configuration file [03:58:36] can you stick the debian dir in git somewhere? [03:59:02] for wikimedia-fonts? or hebrew-fonts? [03:59:02] * jeremyb can't really look at it today though. maybe in 36 hrs [03:59:07] TimStarling: the new one [03:59:19] I'm not sure if it's usable [03:59:53] I can put it in git but I would need to create a project for it under operations/debs [03:59:56] my point was that you're the only one that can see that right now ;P [03:59:59] and that means knowing what the package will be called [04:00:10] I can attach it to the bug [04:00:14] anyway, bbl [04:00:15] that works [04:00:23] (-> bug) [04:11:11] New review: Tim Starling; "The package seems to be broken on Lucid. Someone installed it on mw8, and it failed to configure, wi..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/7349 [04:15:13] Who should I CC on tickets such as https://bugzilla.wikimedia.org/show_bug.cgi?id=39322 ? [04:20:32] depends on whose fault it is [04:23:30] It sounds IPv6 related, so I was guessing network people. [04:23:47] I just couldn't remember who besides Mark was networking related. [04:24:00] Then I wondered if there was a chart. Then I went back to my show. [04:24:53] well, Leslie is also a network person, but it's not really clear if that is a network issue [04:25:04] it would be useful if someone else could confirm it [04:27:06] It isn't clear to me how well IPv6 is supposed to be supported right now on Wikimedia wikis, so I wasn't sure what priority it had. I recall it being mostly experimental. [04:27:18] Though I think https support is also still allegedly experimental... [04:27:38] "unsupported" [04:39:38] Brooke: https is not unsupported [04:39:41] secure is [04:39:45] https is fully supported [04:40:03] AFAIK ipv6 should also be fully supported [04:40:19] I filed a bug. [04:40:51] It's listed as "https services (unsupported)" at http://status.wikimedia.org [04:40:54] are we using apache as our IPv6 frontend? [04:40:55] that's secure [04:40:58] i think watchmouse says that HTTPS is unsupported? [04:41:09] in fact, I'd say we don't support secure at all [04:41:23] hmmm, varnish I guess [04:41:36] TimStarling: nginx is ipv6 frontend [04:41:36] TimStarling: no, it's nginx [04:41:36] Ryan_Lane: You mean secure.wikimedia.org? [04:41:40] Brooke: yes [04:41:50] secure can die in a fire as far as I'm concerned [04:41:52] Via: 1.1 varnish [04:41:56] Ryan_Lane: Right. I'd say that's the case as well. But I'm mostly concerned with status.wikimedia.org. [04:42:04] TimStarling: well, nginx is an ssl termination proxy [04:42:05] Which doesn't make the distinction, as far as I can see. [04:42:07] maybe varnish behind nginx? [04:42:08] it should be transparent [04:42:28] Brooke: and here's the reason I never wanted to list that damn service on that page at all [04:42:33] anyway I get apache errors if I try to use the IPv6 address as a browser proxy [04:42:50] Squid converts them into proper requests with Host headers [04:42:52] Ryan_Lane: Well, maybe it should be removed. :-) But listing it incorrectly is a pretty terrible option to land on. [04:42:54] I guess varnish is not doing that [04:43:04] Brooke: I don't disagree [04:43:15] it should have been listed as secure (unsupported) [04:43:19] And https is not unsupported. You're funny this evening. ;-) [04:43:34] if there's a problem with https, I'll fix it [04:43:53] if there's a problem with secure, I'm not going to put in a ton of effort [04:43:57] Right. I think there's just a labeling problem with status. [04:44:06] Anyway, I filed a bug. Someone will look at it at some point. It's very minor. [04:44:16] which bug? [04:44:29] oh, with status? [04:44:36] https://bugzilla.wikimedia.org/39325 [04:44:38] Yes. [04:45:06] That site (status.wikimedia.org) is only going to grow in prominence. It's important that it be corrected at some point. [04:45:15] Ryan_Lane: have you got a browser with an IPv6 connection handy? [04:45:18] We don't want more and more people to think https is unsupported. [04:45:25] TimStarling: hm. maybe [04:45:32] I guess I could install X on my VPS... [04:45:43] * jeremyb runs away [04:45:55] ugh. seems I don't [04:45:59] damn US ISPs [04:46:10] I thought comcast was supposed to enable it? [04:46:29] just looking for someone to confirm bug 39322 [04:47:08] I tried proxying through an ssh tunnel, but like I say, I just get 404 errors, probably due to the lack of a Host header [04:48:19] and maybe it wouldn't work anyway, maybe it is a problem with the browser's IPv6 support [04:49:03] You could ask orzel if he's having the problem with any other sites. He's in #wikimedia-tech. [05:08:32] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [05:08:32] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [05:38:18] mornin [06:08:48] yes it is [06:13:08] :) [06:14:33] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [06:46:57] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [09:25:56] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [09:36:53] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [09:39:53] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [09:39:54] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [09:39:54] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [10:57:53] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [11:51:54] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:15:53] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [12:54:54] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [13:09:53] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [13:24:53] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [13:39:54] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [14:25:12] PROBLEM - Host virt1005 is DOWN: PING CRITICAL - Packet loss = 100% [14:30:45] RECOVERY - Host virt1005 is UP: PING OK - Packet loss = 0%, RTA = 36.28 ms [14:34:39] PROBLEM - SSH on virt1005 is CRITICAL: Connection refused [14:49:20] PROBLEM - Host virt1005 is DOWN: PING CRITICAL - Packet loss = 100% [14:51:04] morning mr. notpeter, you up and about? [14:51:14] did you get to check out the analytics dell partman stuff at all? [14:59:14] RECOVERY - SSH on virt1005 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:59:23] RECOVERY - Host virt1005 is UP: PING OK - Packet loss = 0%, RTA = 35.32 ms [15:09:54] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [15:09:54] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [15:10:23] morning RobH! [15:10:31] i'm on to figuring out how to install stat1001 [15:10:50] do you know how to connect to console or boot for install? [15:10:54] i'm looking at t his page right now:http://wikitech.wikimedia.org/view/Dell/PowerEdge_1950_2950#Connecting_to_serial_console [15:13:52] i know that it is an R510 [15:13:59] but there isn't that much info on the R510 page [15:16:54] all the same [15:17:00] connect to root@stat1001.mgmt [15:17:03] console com2 [15:17:08] but... [15:17:16] let's first make sure to put it in the analytics subnet [15:17:26] ok, well i've only done c2100s thus far [15:17:29] and that is ipmi [15:17:31] this is ssh, right? [15:17:32] connect com2 the old ones, console com2 the new ones [15:17:37] i just tried sshing from sockpuppet, and from my local [15:17:57] what is stat1001 going to be used for? [15:18:12] kinda like stat1, various analytics stuff, but also for hosting some sites [15:18:19] the new reportcard will go there [15:18:33] current stats.wikimedia.org will be moved there (I think it is currently on spence :p ) [15:19:29] so hm, this will have a public ip [15:19:43] so foward dns in wikimedia.org? [15:20:00] then you're already set [15:20:02] it already has a public ip [15:20:06] perfect [15:20:06] great [15:20:31] so networking should be ready to auto config [15:20:33] PROBLEM - NTP on virt1005 is CRITICAL: NTP CRITICAL: Offset unknown [15:20:41] need to make sure partman stuff is set up? [15:21:11] or you do it manually [15:21:53] oh that's fine [15:21:54] PROBLEM - Host virt1008 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:57] i can do that during installer? [15:22:04] since there is only one of these? [15:22:22] yes [15:22:24] k [15:22:28] use LVM [15:22:32] can do [15:22:32] and you have lots of flexibility later [15:22:34] for / too? [15:22:38] just do whatever was done for stat1 really [15:22:46] aye [15:22:49] not necessarily, get 30 GB for / and you're good [15:22:55] ok cool, yeah that's what i'd do [15:22:58] cool [15:23:06] PROBLEM - Host virt1007 is DOWN: PING CRITICAL - Packet loss = 100% [15:23:47] will do mirrored 30G raid on /, and maybe raid 10 for rest of /a with lvm [15:24:10] according to the r510 wikitech page, this has 12 drives so ja [15:24:12] um, so, ok [15:24:20] yeah, how do I access the console, or tell it to boot installer? [15:24:22] ssh? [15:24:29] and always keep free space in LVM [15:24:32] yeah [15:24:36] at least 10%, but normally, just use what you need [15:24:37] will keep like 20% free or something [15:24:38] yeah [15:24:39] increasing is easy, decreasing is not [15:24:42] aye yeah [15:24:49] yes, ssh [15:24:54] from sockpuppet? [15:24:54] RECOVERY - Host virt1008 is UP: PING OK - Packet loss = 0%, RTA = 35.81 ms [15:24:58] from anywhere [15:25:17] hm, oh host key changed on sock puppet, hang on [15:26:29] ah cool in [15:27:00] RECOVERY - Host virt1007 is UP: PING OK - Packet loss = 0%, RTA = 35.39 ms [15:27:48] hm, so how do I tell it to boot install? [15:28:31] racadm serveraction powercycle [15:28:36] then F12 in bios [15:28:43] hmk [15:28:44] there's also a command to have it pxe boot once, I forget what it is [15:28:50] rob would know [15:29:42] is that from the admin cli or from console after console com2? [15:29:46] console [15:30:17] hmmm, console seems to not really respond much [15:30:23] it goes into console mode [15:30:29] but doesn't respond [15:36:10] hm, i would rather tell it to boot [15:36:14] since my F buttons are all mapped to stuff [15:36:19] not sure how to send a real F132 [15:36:22] F12 [15:36:23] but [15:36:31] console doesn't respond for me at all [15:36:57] oh, btw, I don't think I have a wikitech account [15:37:02] should I? [15:38:40] ottomata: use escape-shift-2 [15:38:48] yea [15:39:06] k, i'm out [15:39:07] ~. [15:39:10] i think worked [15:39:18] RECOVERY - NTP on virt1005 is OK: NTP OK: Offset -0.03916990757 secs [15:42:48] hm [15:42:51] yeah i just got a black screen [15:42:54] couldnt' type anything [15:44:32] should I try again? [15:46:38] nope [15:46:52] whats happening? [15:47:32] no [15:47:36] i get to the admin cli [15:47:49] and then do [15:47:51] console com2 [15:48:13] I see 'connected to serial device...' [15:48:15] for a second [15:48:16] then it clears [15:48:33] or, rather, a buncha blank lines [15:48:39] but then doesn't accept input [15:48:58] racadm from admin cli? [15:49:56] those are my boot commands, no? [15:50:07] racadm config -g cfgServerInfo -o cfgServerBootOnce 1, etc [15:50:20] cool [15:51:13] ok cool it is booting [15:51:23] are those commands in wikitech somewhere? [15:51:52] ah yup! [15:51:53] build a new server [15:51:55] perfect [15:51:55] thank you [15:52:35] cool, sorry, should've referred to that doc when I started a bit ago, all this info is there [15:52:47] i had seen it before, but had been using the c2100 specific pages for the analytics machines [15:52:56] so was looking for dell r510 specific pages for this [15:53:03] ottomata: the normal R series are a lot easier to deal with than the C series [15:53:14] ayyye cool [15:53:21] +1 [15:53:40] i mean, i dont hate them as much as say, our old supermicros [15:53:42] great, and the remaining analytics are r310s [15:53:48] ottomata: yep, all your misc [15:53:57] 'misc' hosts, or whatever you guys called them [15:58:44] hm, ok [15:59:27] its hanging now [15:59:57] https://gist.github.com/3350483 [16:00:06] been sitting at that last bit for 5 mins now [16:05:03] it's not doing anything [16:05:12] "no boot device available" [16:05:16] so you didn't select pxe boot [16:08:18] ottomata: the boot you set with [16:08:22] racadm config -g cfgServerInfo -o cfgServerBootOnce 1 [16:08:22] racadm config -g cfgServerInfo -o cfgServerFirstBootDevice PXE [16:08:26] but that ONLY does the next boot [16:08:35] if you reboot twice, the second reboot goes back to the defaults (disk first) [16:09:10] hard to read the output since it writes over itself [16:09:40] i tend to ctrl+k to clear backlog a lot in serial console [16:09:57] since it doesnt scroll the terminal, just writes over itself [16:15:53] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [16:19:44] New patchset: Pyoungmeister; "fixing transitional nagiso breakage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19497 [16:20:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/19497 [16:21:04] RobH: apple-r works for that [16:21:25] mark: during the actual serial console output, your screen scrolls down? [16:21:37] i know after i exit serial console apple+r sends a reset to terminal to work right [16:21:50] but i will try during post next time =] [16:21:52] i press it multiple times during boot [16:21:58] especially when the colors fuck up [16:22:02] but i believe it also works for scrolling [16:22:49] coolness, will have to give it a shot, would be nice to keep a scrolling backlog of the serial redirection [16:27:36] sorry, was grabbing a sammy [16:27:44] um, i think I did only tell it to pxe boot once [16:27:46] will try again [16:27:50] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19497 [16:38:54] hm so [16:39:00] i was getting more dots and spinning slashes! [16:39:01] DHCP.....\ [16:39:07] but it has now stopped moving [16:39:25] mark,RobH,cmjohnson1 ^ [16:39:48] is this wrong then? [16:39:49] Nopboothdevice1available.nteltCorporation v6.0.11 [16:39:49] CurrentmbootemodeeisXset.tovBIOS [16:39:52] so it never loads up with a ip address ? [16:39:57] i guess not [16:40:04] what machien ? [16:40:11] i can check out the tubes and stuff [16:40:20] leslietubechecker on duty! [16:40:48] oh, shit i did promise i would only work on fundraising stuff today [16:40:53] let's pretend it's a fundraising machine [16:40:56] ;) [16:42:00] ahha, is it 78:2b:cb:35:c4:f7 ? [16:42:44] so, if you want to know what's up, i went to brewster (the installation server) and checked out /var/log/syslog [16:43:30] when you see a line like "dhcpd: DHCPDISCOVER from 78:2b:cb:35:c4:f7 via 208.80.154.131: network 208.80.154.128/26: no free leases" then it usually means that machine's mac address isn't configured in one of the "linux-host-entries" files (ottomata ^^) [16:43:38] hmmm [16:43:55] there is an entry [16:43:56] but it is [16:44:00] host stat1001 { [16:44:01] hardware ethernet 78:2b:cb:1e:79:dd; [16:44:27] ok, so how do I check MACs on the r510? [16:44:41] on the management you can check out i think it's "racadm getsysinfo" [16:44:55] k [16:44:58] (and was the mobo changed on this machine to fix it?) [16:46:16] weird, I don't see 79:dd in the sysinfo at all [16:46:19] dunno where that came from [16:46:29] i do se c4:f7 [16:46:32] will chage in host entries [16:47:53] it could have been a motherboard swap ? [16:48:00] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [16:48:07] New patchset: Ottomata; "Fixing MAC addy for stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19499 [16:48:14] aye oh yeahhh [16:48:19] they did have hw issues with this machine [16:48:20] cool [16:48:24] think they did a swap [16:48:40] can you approve that one? [16:48:44] https://gerrit.wikimedia.org/r/19499 [16:48:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/19499 [16:49:13] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19499 [16:49:18] approved :) [16:49:23] danke! [16:49:29] are you going to merge on sockpuppet then update brewster ? [16:52:47] done already [16:52:58] booting now [16:53:23] i hope this works! [16:53:30] i've been trying to install OSes on servers for a week now! [16:53:34] haven't yet had a success! :p [16:55:44] looking better! [17:00:23] should I see more than just sda when I am doing manual partitioning during boot? [17:00:48] ottomata: so that system is a raided system [17:00:52] it has 12 disks, but in a raid10 [17:01:08] if you all dont need raid10 performance, you can get quite a bit more space by going raid6 [17:01:08] already? [17:01:10] oh hw raid! [17:01:12] ohhhhhh [17:01:30] 6TB as is [17:01:32] that should be fine [17:01:35] so you may wanna change the raid to raid6 if you dont need the high performance writes for raid10 [17:01:35] cool even easier [17:01:41] but otherwise raid10 is fastest [17:02:45] New patchset: Pyoungmeister; "adding real password lookups for vumi" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19500 [17:03:27] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/19500 [17:06:30] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19500 [17:09:30] mark, i see on stat1, root is LVM [17:09:36] is that ok? [17:09:43] should I do that for stat1001 [17:09:44] ? [17:09:46] New patchset: Ryan Lane; "Rework openstack manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19501 [17:09:48] paravoid: ^^ [17:09:54] I'm going to −2 this for now [17:10:21] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/19501 [17:10:27] New review: Ryan Lane; "Glance's configuration is wrong between diablo and glance. This needs to be fixed before being pushe..." [operations/puppet] (production); V: 0 C: -2; - https://gerrit.wikimedia.org/r/19501 [17:10:49]