[19:12:32] only because of how the disks were lined up... [19:12:43] well, before it was on a riad [19:12:51] that raid is gone, so needs reinstall yes? [19:13:4] the install worked fine when it was a regular ES host. [19:13:11] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Thu Dec 15 19:12:20 UTC 2011 [19:14:11] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Thu Dec 15 19:13:14 UTC 2011 [19:14:25] well, it launched back into installer since no os disk, installer is running. [19:14:32] since the os partition was in a raid that doesnt exist now [19:14:55] maplebed: you dont want to test with failing disks out in swift or anything befoer we do this order? [19:15:7] cuz this will be a special order [19:15:28] it'd be nice, but I'd have to set up 3 of them like that. [19:15:41] hm. [19:15:42] that means we have to steal parts from 6 other servers [19:15:43] =P [19:15:57] the sas cable b works, not sas a, due to how they put the connectors on there [19:16:6] which are just for cable routin gin the r610, otherwise seems to be same cable [19:16:11] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Thu Dec 15 19:15:15 UTC 2011 [19:16:11] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Thu Dec 15 19:15:20 UTC 2011 [19:16:18] (a cable was thicker than b, but seems to not matter so far) [19:16:36] I think I can do the disk-fail tests on the thumpers in pmtpa. [19:16:52] well, its already got differnt items in the installer [19:16:57] │ Activate Serial ATA RAID devices? │ [19:17:9] so i am going to update a ticket, and leave this for when i am not on site for doing the installer stuff [19:17:17] unless you snag before me, but otherwise bios shows disks [19:18:22] I think I'm stuck in my puppet work for now till mark gets back, so I'll try and get it to boot now. [19:18:55] ok, i am in here if you need hands on [19:19:0] routing cables for new ciscos [19:19:9] !log removed mw* and virt* yaml files from stafford in order to clear up broken files and make puppet run again [19:19:11] RECOVERY - Puppet freshness on virt1 is OK: puppet ran at Thu Dec 15 19:19:02 UTC 2011 [19:19:11] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Thu Dec 15 19:19:04 UTC 2011 [19:19:18] Logged the message, Mistress of the network gear. [19:19:26] hopefully we should see a bunch of "RECOVERY" from nagios [19:19:27] :) [19:20:11] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Thu Dec 15 19:20:06 UTC 2011 [19:20:39] * apergos peeks in [19:21:28] cmjohnson1: have not gotten to it yet, though apergos may beat me to it [19:21:40] apergos: dataset1 mainboard is in, it just needs reinstall and testing to see iff issues are gone [19:21:50] sweet [19:22:15] I planned to work on the backup code for a few days, since I have a backlog [19:22:20] s/backup/dumps/ [19:22:35] LeslieCarr: They dont even pay for shipping on this? [19:22:39] our juniper warranty fucking sucks [19:22:43] seriously [19:22:45] :( [19:23:34] i also dont have the original box [19:23:41] just a medium sized box, and lots of paper packing. [19:24:11] RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Thu Dec 15 19:23:10 UTC 2011 [19:25:0] that will have to do RobH - or i guess you can take it to the local fedex/etc ? [19:26:0] heh, i can print a label and schedule a pickup [19:26:4] just sucks that they arent paying is all [19:26:11] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Thu Dec 15 19:25:12 UTC 2011 [19:26:52] LeslieCarr: uhh, there is no address in those tickets [19:26:55] where the hell do i ship it? [19:27:9] lemme look... [19:27:10] (there is nothing in RT, i just have the emails you cced me into, without that info) [19:27:23] normally i tie open cases to the rt email so they email that but some vendors dont do that [19:27:27] dell does, but i had to beat them into it [19:27:54] holy shit, the juniper case doesn't have it [19:27:55] awesome [19:27:55] the ticket info online may have it, but i cannot login to that yet [19:27:59] they fucking succccck [19:29:0] the tickt info online doesn't [19:29:7] emailed an update to the ticket [19:29:24] i will go ahead and box it up and write rma info on outside [19:29:49] cool [19:29:58] you should be cc'ed on the reply that will hopefully have the address [19:31:26] yep, i have it boxed up to go back and sitting in our cage. Once we get that info i will either come back to datacenter, or drop a ticket to EQ with all the info and label to do the shipment [19:31:30] since its all ready to go now =] [19:31:54] cool [19:31:56] thank you :) [19:32:19] thx for getting the rma [19:43:44] RobH: the installer keeps asking me to log into iSCSI. [19:43:47] any idea what that's about? [19:46:40] =/ [19:47:5] no, i saw it want to activate serial ata devices [19:47:7] also not normal [19:47:16] so it may be it sees the disks, but we cannot use it like we are [19:47:20] dell hasnt answered me yet [19:48:2] drdee: welcome aboard ;] [19:48:15] now i dont have to feel bad about you doing puppet work as a volunteer! [19:48:28] * RobH laughs as evilly as possible [19:50:1] apergos, I just noticed it seems to make the connection to the database arbitarily [19:50:8] it doesn't use it [19:50:18] oh, wait [19:50:34] * apergos waits [19:50:43] you have been eaten by a grue. [19:51:0] haha [19:51:26] it just doesn't pass the link identifier back in anywhere [19:52:32] RobH: I'm going to leave es1002 in your capable hands (for when you get home). [19:53:14] maplebed: thats cool, i will take a stab at it later today or tomorrow morning =] [19:53:24] so we can still do order tomorrow if it works [19:53:25] thanks. [19:53:31] +1 [19:57:6] New patchset: Jgreen; "fix typo in exim-ganglia collector, open throttles more on exim config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1566 [19:57:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1566 [19:59:31] PROBLEM - Puppet freshness on ssl3004 is CRITICAL: Puppet has not run in the last 10 hours [19:59:31] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: Puppet has not run in the last 10 hours [19:59:31] PROBLEM - Puppet freshness on ssl2 is CRITICAL: Puppet has not run in the last 10 hours [20:0:8] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1566 [20:0:9] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1566 [20:5:47] back [20:11:51] New patchset: Mark Bergsma; "Correct class inheritance syntax" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1568 [20:12:1] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1568 [20:12:22] apergos, now it's shouting in me in German :D [20:12:36] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1568 [20:12:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1568 [20:14:28] nice [20:19:34] New patchset: Mark Bergsma; "Get rid of the noop hack by using virtual resources in swift::proxy::config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1569 [20:19:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1569 [20:21:25] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1569 [20:21:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1569 [20:28:27] New patchset: Mark Bergsma; "Fix requirements, Definition[] does not work" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1570 [20:28:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1570 [20:28:56] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1570 [20:28:56] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1570 [20:35:52] New patchset: Platonides; "Remove unneeded global" [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/1571 [20:40:42] New review: Platonides; "(no comment)" [test/mediawiki/core] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1571 [20:41:50] New review: Platonides; "(no comment)" [test/mediawiki/core] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1571 [20:41:51] Change merged: Platonides; [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/1571 [20:53:05] Platonides: watch out, I am not sure your code will be kept [21:19:01] New patchset: Mark Bergsma; "Remove useless $roles variable" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1573 [21:19:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1573 [21:19:20] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1573 [21:19:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1573 [21:23:17] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1319 [21:23:17] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1319 [21:23:47] PROBLEM - Puppet freshness on ssl3003 is CRITICAL: Puppet has not run in the last 10 hours [21:26:24] New review: Mark Bergsma; "I don't like these all-encompassing "cleanup" commits. It's a recipe for conflicts, breaks blame." [operations/puppet] (production); V: 0 C: -2; - https://gerrit.wikimedia.org/r/1494 [21:28:25] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1497 [21:28:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1497 [21:42:50] New patchset: Platonides; "Revert "Remove unneeded global"" [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/1574 [21:44:37] New review: Platonides; "Partial review" [test/mediawiki/core] (master); V: 1 C: 1; - https://gerrit.wikimedia.org/r/1574 [21:44:46] New review: Platonides; "Partial review" [test/mediawiki/core] (master); V: 1 C: 1; - https://gerrit.wikimedia.org/r/1574 [21:57:43] argh, autoconfirmed [21:58:22] your new account? [21:59:34] yeah [21:59:40] images seemed to be silently failing to upload [21:59:53] instead of telling you? that's a drag [22:00:00] check the html spewed (it's posting against Special:Upload, ewww), and the error is in the middle [22:00:11] blah [22:00:27] https://commons.wikimedia.org/w/index.php?limit=50&tagFilter=&title=Special%3AContributions&contribs=user&target=Reedy+RotateBot&namespace=&tagfilter=&year=&month=-1 [22:00:30] Working now [22:00:34] yay [22:00:41] Upload seems slow though [22:00:56] Going to try and get some rough speed stats [22:01:51] this only does requested bots right? [22:01:52] er [22:01:54] requested images! [22:02:09] yup [22:02:21] but it's staying constant about 970 images, due to the speed they're being tagged... [22:02:46] great [22:06:40] download is fast, as expected [22:15:07] apergos, Special:Contributions would suggest 2-3 minutes for a 3MB file :/ [22:15:18] 2-3 minutes? [22:15:36] to upload [22:16:11] 480 images a day? [22:16:40] didn't it do better than that from the toolserver? :-P [22:16:45] Yeah [22:16:54] I was just thinking, this seems to suck even more locally [22:17:54] seems nuts [22:17:54] Hmm [22:18:04] There's bug 31607 saying similar [22:18:09] 0-20-40 KB/s [22:18:18] https vs secure.wm.o [22:18:31] I thought that had been fixed up [22:18:38] Yeah, it was for a time [22:18:43] It would seem to have regressed at some point [22:18:58] can you use regex in the ServerName field for apache ? [22:19:18] doesn't the bug discuss what the "fix" was? I mean it wasn't changes to mw [22:19:22] or even to nginx [22:19:55] does the bot use https? [22:20:01] IIRC it was a router issue [22:20:44] so I don't see how we can have a regression [22:21:01] I'm not saying you're imagining it, I just wonder what the heck the cause could be now [22:23:34] Content-length is number of bytes, right? [22:23:40] yep [22:23:50] for the last, 388205 took 89 seconds [22:24:17] no [22:24:24] 3882805 [22:24:42] ~ 3.7MB taking 89s [22:25:05] About 40KB/s? [22:25:22] does the bot use https? [22:25:43] nope, it's doing it to port 80 [22:25:48] ok [22:26:02] I think i should have the headers to see what dealt with it [22:26:30] you want to do an upload to testwiki? same image? [22:26:31] time it? [22:26:59] New patchset: Lcarr; "Rewriting planet ssl to go to http planet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1575 [22:27:12] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1575 [22:27:17] maplebed: have time for a quick review? I want to make sure I got this syntax correct - https://gerrit.wikimedia.org/r/1575 [22:27:20] http://noc.wikimedia.org/~reedy/log.txt [22:27:21] Sure [22:27:27] LeslieCarr: looking. [22:27:52] wow, there is a real chance I will wake up tomorrow with no achy shoulder [22:27:54] I don't think you need to repeat the servername in the serveralias line [22:27:56] this would be amiazing [22:28:01] :) [22:28:05] -i [22:29:13] maplebed: my hope is that everything will just go to 80 instead of 443 since it is not working correctly…. I figured a * would be easier than making 1 entry per site, but I'm not sure if it will work correctly [22:29:15] LeslieCarr: does singer host any sites on https? [22:29:19] yes [22:29:37] maplebed: techblog and contacts are on ssl [22:29:47] oh, nevermind on the servername / serveralias comment - I misread wikipedia as wikimedia.o [22:29:52] techblog https *should* work [22:30:11] Ryan_Lane: it does, at least from my quick "go to it and see if it works" [22:30:30] does anyone know how I can get sun java JDK installed ? :) [22:30:41] it does not seem to be available from the apt sources list we have [22:30:56] hashar, sun java!? [22:30:59] oracle java! [22:31:02] heh [22:31:04] ahh [22:31:09] I think it's been dropped from ubuntu's repos [22:31:09] Though, you realise that's swearing, right? [22:31:12] worse than evah [22:31:23] snoracle java [22:31:26] :) [22:31:28] anyway [22:31:35] reedy, the suspense is killing me... [22:31:46] an "apt-cache search jdk" does not returns anything [22:31:58] I guess it is only available from a non-free apt or something else [22:32:00] i'm just making it directly do to commons, then to test afterwards [22:32:07] ok [22:32:07] 4008298 124 seconds to commons... [22:32:30] LeslieCarr: (when you return) I think you want 301, not 302. Other than that, the syntax looks fine. just make sure to test non-planet ssl on singer and make sure it still works. ;) [22:32:30] ah no wonder I am sleepy [22:32:43] it's after midnight [22:33:59] 128 seconds for commons [22:34:49] I thought you said 124 for commons [22:34:58] oh, test even [22:34:59] sorry [22:35:01] ah [22:35:08] ok this is different behavior than the bug iirc [22:35:28] it does seem very weird that from toolserver it's faster [22:35:40] do you have an account there you can try a timed test from? [22:35:59] (I don't... unrenewed for at least 2 years now) [22:36:06] I do, yeah [22:36:08] ok [22:36:18] 30086 was the old upload slowness [22:36:25] LVS not liking jumbo packets [22:36:38] right [22:37:01] if toolserver is in fact faster I would open a new bug report (point to the other ones but they might not be relevant) [22:37:09] and sic mark on it [22:37:16] maplebed: i always mix up 301 and 302 -- perhaps i should print out the lolcats [22:37:29] ... lol! [22:37:40] yeah, me too, but the rfc is a short google away. [22:37:51] 301 is permanent; 302 temp. [22:38:35] apergos, just looking at the Special:Contributions.. it's about a minute a file [22:38:46] so toolserver is around twice as fast [22:38:57] the fie sizes are the same? [22:39:14] ok well I think that's grounds for asking mark to take a look [22:39:31] they're all smaller [22:39:31] *file! [22:39:39] fie, fie on files [22:39:50] um, how much? [22:40:16] low as 133 KB, most of the recent ones were under a meg by the look of it [22:40:43] ok well then that's not faster [22:40:47] yeah [22:40:53] can you run one that's about the same size? [22:41:04] just seemed faster due more small files [22:41:06] New patchset: Lcarr; "Rewriting planet ssl to go to http planet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1575 [22:41:19] I can run it on TS as the user, and just hack in my debugging output for timing [22:41:25] ok [22:41:31] maplebed: how about now ? https://gerrit.wikimedia.org/r/1575 [22:42:06] if there's not an appreciable time difference then... gotta look at the upload code I guess [22:42:24] looks fine. why are we squashing ssl on planet? [22:42:37] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1575 [22:42:44] it doesn't work correctly http://rt.wikimedia.org/Ticket/Display.html?id=1854 [22:44:08] I'm going to afk and try to get a full night's sleep, might actually happen.... [22:44:20] apergos: good night and good luck! [22:44:35] in the meantime, running the bot form two places at least gets twice as many images in :-D [22:44:40] see folks tomorrow [22:44:49] !log removing https support from planet.wikimedia.org [22:44:57] thanks for the luck, I can use it! [22:44:58] Logged the message, Mistress of the network gear. [22:44:58] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1575 [22:44:58] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1575 [22:56:53] New patchset: Lcarr; "Making planet.wikimedia.org owned by puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1576 [22:57:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1576 [22:57:16] maplebed: got time for another review ? someone had made a planet class in puppet but didn't actually make singer use that class :) [22:58:53] looking. [23:00:07] LeslieCarr: +1 [23:00:22] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1576 [23:01:35] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1576 [23:01:36] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1576 [23:03:42] !log puppetized planet.wikimedia.org on singer [23:03:51] Logged the message, Mistress of the network gear. [23:04:00] let's hope planet doesn't break now… :) [23:06:28] New patchset: Lcarr; "Closed VirtualHost" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1577 [23:06:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1577 [23:07:04] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1577 [23:07:05] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1577 [23:09:43] maplebed: so it looks like i do have to make a redirect for each language, as planet.wikimedia.org just redirects to an info page about planet [23:10:11] or you could just send all https traffic to the planet front page. [23:10:19] maybe the english one ? [23:10:30] by default as most likely ? [23:10:42] you'll be accused of cultural elitism. [23:10:46] yes i will be [23:10:59] is that better or worse than sending people to the meta wiki? [23:11:58] i'll say it is better than redirecting people to meta wiki when they hit the https site - and it's better than the previous state of affairs [23:12:52] actually, you can probably redirect them to the correct language. [23:13:12] what would the syntax for that look like ? [23:13:27] it's been a while - I'm looking for it here: http://httpd.apache.org/docs/current/mod/mod_rewrite.html [23:13:33] New patchset: Sara; "Testing lvs for labs." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1579 [23:13:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1579 [23:14:17] ah maplebed it looks like here might work - just in reverse http://www.cyberciti.biz/tips/howto-apache-force-https-secure-connections.html [23:14:51] yeah, something like that. I thought it would be SERVER_NAME rather than HTTP_HOST, but I haven't found the definitons of each yet. [23:15:14] this page uses SERVER_NAME [23:15:22] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1579 [23:15:23] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1579 [23:15:24] that's probably using whatever was passed in by the Host: header rather the name the server uses to refer to itself (which is what we want, so http_host is probably right) [23:18:15] maplebed ooo apache has documentation that is actually nice - https://httpd.apache.org/docs/2.0/misc/rewriteguide.html [23:19:00] yeah - it's pretty good. [23:19:01] server_name may be set to whatever Host: header had, or not [23:19:23] there's a setting for that, which may fall back to Host: [23:20:26] New patchset: Lcarr; "Redirect all https requests for planet to http" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1581 [23:20:35] maplebed: want to check it out ? https://gerrit.wikimedia.org/r/1581 [23:20:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1581 [23:22:15] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1581 [23:22:30] lgtm. [23:22:42] !log pushing rule to planet.wikimedia.org which should redirect all https to http [23:22:50] Logged the message, Mistress of the network gear. [23:22:52] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1581 [23:22:52] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1581 [23:26:51] maplebed: got a minute ? [23:28:01] that was less than a minute. [23:28:03] :D [23:29:03] is this going to work with HTTPS Everywhere? [23:29:40] let me look int he https everywhere rules [23:29:56] Nemo_bis: sadly, https on planet is currently broken, which is why I'm doing the redirect [23:30:03] yes, I know [23:30:18] although it wasn't compltely tragic AFAICS [23:30:38] seems to work [23:31:51] https everywhere is working for me [23:33:02] woosters: who could we get to take a look at this http://rt.wikimedia.org/Ticket/Display.html?id=2147 .. sun java sdk ? [23:33:14] its important for the android app market release [23:33:19] and it'll be part of our nightly build system