[00:04:33] notpeter: still about? [00:06:01] RobH: I had officially stopped working, but is it importante? [00:06:15] just running into partman error on cp installs [00:06:28] was wondering if you had tweaked squid-raid1.cfg or whatever [00:06:40] seems we need some of the new cp servers online tonight to cover mobile [00:06:44] spinning up mobile cps? [00:06:49] yea [00:07:24] RobH: I could try checking the partman's out -- though it sounds like by hand might be the quickest for now… [00:07:42] by hand is awful, i want to try to fix this [00:07:48] its not acceptable for it to remain broken [00:08:08] when i did the cp1044/45 install, it wasnt broken [00:08:16] so something since then has made it inoperable, which needs fixing. [00:08:35] RobH: I spent a lot of time trying to make a working partman config. I'm not sure partman can do that. might need a post-install script [00:08:47] I did it by hand [00:08:49] it used to work though. [00:09:01] not for mobile CPs [00:09:03] not that long ago, on cp# [00:09:23] the installer for this is no different for mobile versus non [00:09:30] partman is same for all squids [00:09:45] unless it was recently changed, and broken. [00:10:27] notpeter: so when you went to do it, it was broken? [00:10:28] RobH: I don't think I changed it [00:10:43] was it giving an error during the installer? [00:11:02] New patchset: Asher; "ugly hack to get varnish purges working while w3/wp sends broken purge reqs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1941 [00:11:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1941 [00:11:42] RobH: when I went to make mobile CPs, I didn't use squid-raid1.cfg [00:11:46] I went to making a new one [00:11:57] so mobile is using varnish? [00:12:00] that has two xfs partitions [00:12:00] yes [00:12:06] because if so that directly conflicts with what woosters told me to install these for [00:12:08] god dman it. [00:12:27] well, nice to know i wasted a couple of hours on this already. [00:12:37] ugh, I'm sorry [00:12:44] so varnish needs a different partitioning setup than normal squid? [00:12:47] want help on it? [00:12:47] yes [00:13:11] /dev/sda5 /a/sda xfs nobarrier,noatime 0 2 [00:13:11] /dev/sdb5 /a/sdb xfs nobarrier,noatime 0 2 [00:13:19] RobH: building as squids are fine.. the actual disk partitioning is the same, except that sda5 and sdb5 are mounted with xfs [00:13:20] that's what it wants to look like [00:13:32] instead of being used as raw devices as squid does for coss [00:15:02] Ryan_Lane: https://gerrit.wikimedia.org/r/#patch,sidebyside,1941,1,templates/varnish/blog.inc.vcl.erb === lulz [00:15:16] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1941 [00:15:17] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1941 [00:15:24] hahaha [00:36:25] New patchset: RobH; "added cp103X to mobile varnish range" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1943 [00:36:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1943 [00:37:39] notpeter: check that please? ^ [00:37:43] being good, not checking my own shit [00:38:13] notpeter: argh, it rebooted back in to the installer [00:38:16] so bios may not be set right [00:38:34] yeah, but I had to flip mine too [00:38:45] i didnt, so setting it on cp1040 now [00:39:13] RobH: that will only do 31-34 and 41-44 [00:39:26] bugger =P [00:39:33] decline it and i redo [00:39:39] well, i have not had a mistake yet [00:39:46] i guess i should do some odd command to append to that one? [00:39:59] yeah, you can just append [00:40:02] er [00:40:02] amend [00:40:16] New patchset: Lcarr; "tryuing to see where ganglia chokes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1944 [00:40:31] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1944 [00:40:33] Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1944 [00:40:50] notpeter: so i did my change locally, how exactly do i amend it? [00:41:05] git commit -a --amend [00:41:09] then push again [00:41:10] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [00:41:31] notpeter: sound like we dont need as many as i thought, so puppet change is fine for that many when i fix it [00:41:56] but otherwise we will only install down to cp1036, thats 5 servers when they wanted 2 [00:42:11] down to 36 soudns good [00:42:22] New patchset: RobH; "fixed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1946 [00:42:30] meh, abandon and redo is faster. [00:42:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1946 [00:43:48] notpeter: i am doign the even installs, figure we will do post os install stuff in a moment after these are done [00:43:50] sound good? [00:44:22] RobH: sure [00:44:40] I already statred doing some crap on 39, so I'm going to finish that [00:44:43] but after that, yes [00:44:58] New patchset: Lcarr; "another ganglia test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1947 [00:45:00] thats cool [00:45:11] wrong pfr- doh [00:45:14] i have cp1038 installer started, once its into the software part i move on to 36 [00:45:14] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1947 [00:45:22] Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1947 [00:47:06] New review: RobH; "once more with feeling" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1946 [00:47:53] notpeter: when you are in bios [00:47:57] turn off the logical processor under cpu [00:48:06] thats hyperthreading, misrepresents cpu [00:48:11] its not end of world, but its crappy [00:48:33] kk [00:48:58] heh, it used to break ganglia back in the day [00:49:07] it had no idea where to put the ton of cpu core apaches [00:49:23] hah [00:51:09] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.043 second response time [00:53:57] wtf cp1038 [00:54:10] RobH: while we're in making filesystems and shit, we should also run puppet [00:55:05] RobH: want me to look at it if you finish up changes to site.pp? [00:55:26] site.pp is checked in, making live now [00:58:18] Change abandoned: RobH; "redid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1943 [01:00:42] New patchset: Asher; "use mod_rpaf on blog server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1950 [01:00:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1950 [01:01:08] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1950 [01:01:09] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1950 [01:01:44] Change abandoned: RobH; "tired of dealing with it" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1946 [01:12:41] ok, so cp1036 is in the installer now [01:12:55] New patchset: Pyoungmeister; "this one is for comrad robh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1953 [01:13:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1953 [01:13:21] New review: RobH; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1953 [01:13:32] Your change could not be merged due to a path conflict. [01:13:32] Please merge (or rebase) the change locally and upload the resolution for review. [01:13:34] wtf [01:13:41] notpeter: reads that on the change. [01:14:12] RobH: woops... [01:14:22] aint ever easy ;] [01:14:27] can you abandon? [01:14:35] wrong branch.... [01:14:51] Change abandoned: Pyoungmeister; "wrong brach" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1953 [01:15:31] heh, you beat me to it [01:16:31] New patchset: Pyoungmeister; "for comrade robh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1954 [01:16:37] ok, that one won't be fucked up [01:16:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1954 [01:16:57] New review: RobH; "once more with feeling!" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/1954 [01:17:26] New review: RobH; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1954 [01:17:26] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1954 [01:18:36] its live [01:19:33] ok [01:19:50] so, 37, 39, and 40 have file systems all squared away [01:19:55] so I'm going to run puppet on them [01:20:08] 38 should be ready to have opst install [01:20:26] ugh. need to specify puppetmaster [01:20:27] ugh [01:21:40] yea, want me to do that [01:21:44] and you do post install on 38 [01:22:08] 37, 39, and 40 all have keys waiting to be signed on sockpuppet now [01:22:11] will do 38 [01:23:09] signed all three [01:23:12] running on 37 now [01:23:40] hrmm [01:23:46] 37 must have hit the other server first [01:23:51] cuz its not hitting right server now [01:23:58] wiping its keys and trying again [01:24:28] kk [01:24:33] I might have fucked on up [01:24:42] thought I fixed it, though [01:24:49] if it runs and hits the toher puppetmaster [01:24:55] ya have to do the remove keys on the client [01:25:09] I did so [01:25:10] if you just run the test with the right server, it will hit it to sign [01:25:15] huh [01:25:17] eh [01:25:18] well, trying again [01:25:19] whatevs [01:25:23] indeed [01:25:37] 36 good to go? [01:26:10] its rebooting post install [01:26:11] right now [01:26:21] so will be in 30 seconds or so [01:27:00] kk [01:27:43] I signed cert for 38 [01:27:47] goin to 36 now [01:29:41] puppet is odd [01:29:50] its making me specify the server a second time after the initial one [01:29:57] which returns iwth a different error [01:30:01] then puppetd --test works normal [01:30:05] so odd. [01:30:37] going to run initial puppet run on 36 [01:30:56] i think you will see what i mean [01:31:07] wtf? [01:31:38] weird [01:31:55] I'm glad m_ark emailed us good instructions. very counterintuitive [01:33:18] !log cp1037, cp1038, cp1039 os installed, varnish partitions mounted, and puppet run [01:33:21] Logged the message, RobH [01:33:33] binasher: ^ those are ready for you I do think, notpeter and i are finishing up cp1036 and cp1040 now [01:33:38] thank you! [01:33:55] puppet run on 36 [01:33:57] quite welcome [01:34:07] puppet is finishing run on cp1040 [01:34:10] RobH: want to run puppet on 40 and then we call it done? [01:34:18] yep, near done [01:35:01] !log cp1040 and cp1036 ready for use [01:35:03] Logged the message, RobH [01:35:05] notpeter: all done [01:35:08] binasher: they are all yours [01:35:59] they look good, thanks again [01:36:11] that leaves you like an hour to make them work ;] [01:36:13] heh [01:37:13] wha? blackout doesn't start for 3 hours+ [01:37:28] although, at my rate of whiskey consumption, I might beat it =P [01:37:32] I kid [01:38:22] hah [01:39:21] i wonder how many people will actually switch to the mobile site… and when. if it gets hammered, it might not be til tomorrow morning [01:49:24] black the planet!!! [02:04:55] binasher: hey [02:05:09] sorry, my hands were full of dinner stuff, had my bf transcribe [02:05:20] LeslieCarr: i think i just found the problem, but not why it exists [02:05:38] what's the issue ? [02:05:50] LeslieCarr: gmond.conf on the cp aggregator had deaf = yes [02:05:57] oh [02:06:03] i made a ticket for that and forgot [02:06:13] for some reason puppet isn't parsing the names when they're in a variable [02:06:22] so have to split that out [02:06:35] i'll grab that since i should have gotten to it today anyways [02:11:57] LeslieCarr: after staring at the puppet files and gmond template.. [02:11:59] New patchset: Lcarr; "fixing cp hosts so they will properly alert" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1955 [02:12:08] binasher: want to approve/deny ? [02:12:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1955 [02:12:42] for some reason it isn't always looking at the internal "if host =~ blah, then true" bits [02:12:46] i think the problem is just that $ganglia_aggregator = "true" is after the includes [02:12:53] ah [02:12:57] that could be it [02:13:47] Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1955 [02:13:57] i'll try that first :) [02:15:48] New patchset: Lcarr; ""fixing cp hosts so they will properly alert"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1956 [02:15:57] binasher: check that out ? [02:16:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1956 [02:16:17] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1956 [02:16:17] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1956 [02:17:35] binasher: are you sockpuppeting ro should i ? [02:18:35] i'll give it a shot [02:18:45] doh i hadn't heard you so i just did [02:18:47] :-/ [02:19:05] well the fetch bit, looks like you did the merge bit :) [02:19:18] i'm running a puppetd --test now on cp1044 [02:20:04] LeslieCarr: that fixed it, thanks! [02:22:04] w00t [02:22:19] text me if anything else comes up [02:22:43] i'll be back online at 8pm [02:22:43] will do, thanks for hopping on [02:22:48] no prob [02:25:03] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1799s [02:29:03] New patchset: Asher; "this adds cp1039 and cp1040 to the varnish backend pool (add to pybal conf to also add frontend)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1957 [02:29:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1957 [02:29:33] PROBLEM - Memcached on marmontel is CRITICAL: Connection refused [02:30:05] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1957 [02:30:05] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1957 [02:34:53] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:35:29] !log cp1039-40 are now in service for mobile wikipedia [02:35:31] Logged the message, Master [03:49:05] okay, i'm back online :) [04:16:03] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:24:53] RECOVERY - Disk space on es1004 is OK: DISK OK [04:39:53] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:23:31] is it expected that hitting "stop" on your browser results in circumvention of the blackout? [05:24:29] looks like the blackout was implemented in javascript, overwriting the page content [05:24:38] so yes pressing stop will stop script execution [05:24:50] sorta odd :) [05:25:09] they didn't exactly have the luxury of a long time to prepare the code :) [05:25:47] well, it's the thought that counts. :) [05:52:58] Ryan_Lane: you might appreciate this: http://torrus.wikimedia.org/torrus/CDN?path=%2FSquids%2FTotals%2FAll_squid_client_requests [05:55:57] heh [05:56:03] that's a pretty quick jump [06:13:53] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [07:27:42] RECOVERY - Squid on brewster is OK: TCP OK - 0.000 second response time on port 8080 [08:01:52] PROBLEM - DPKG on db43 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [08:35:43] PROBLEM - Puppet freshness on db1045 is CRITICAL: Puppet has not run in the last 10 hours [09:52:21] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 452125 MB (3% inode=99%): [09:59:41] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 423807 MB (3% inode=99%): [10:34:22] https://bugzilla.wikimedia.org/33509 could use a look. reedy RT'd it at least several days ago [10:40:38] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:48:13] PROBLEM - Puppet freshness on mw1096 is CRITICAL: Puppet has not run in the last 10 hours [12:58:02] PROBLEM - Auth DNS on ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:03:47] !log restarted pdns on ns0 [13:03:48] Logged the message, Master [13:10:21] RECOVERY - Auth DNS on ns0.wikimedia.org is OK: DNS OK: 6.668 seconds response time. www.wikipedia.org returns 208.80.152.201 [13:21:20] New patchset: Hashar; "gallium: allow postgre restart" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1958 [13:30:01] !log Starting mailman migration [13:30:02] Logged the message, Master [13:30:44] !log Set hold_domains = lists.wikimedia.org on lily, to hold new lists mails on the queue [13:30:45] Logged the message, Master [13:34:35] !log Stopped mailman on lily and sodium [13:34:36] Logged the message, Master [13:35:15] !log Stopped lighttpd on lily [13:35:16] Logged the message, Master [13:37:19] !log Created /var LVM snapshot on lily [13:37:20] Logged the message, Master [13:37:29] !log Removed all test messages on the exim4 queue on sodium [13:37:31] Logged the message, Master [13:38:20] !log Started rsync of selected mailman directories under /var/lib/mailman from lily to sodium [13:38:21] Logged the message, Master [13:58:13] Hi! I have a press inquiry in OTRS. Is anyone from the technical staff online? [13:59:34] <^demon> Lots of people are. What's up? [14:01:03] I have an inquiry from a Norwegian journbalist who wants to know what impact the blackout has on other language editions, especially the Norwegian one, in terms of hits. [14:01:21] please do this in #wikimedia-tech [14:02:11] :) [14:02:15] Thx. Will do. [14:02:44] thanks :) [14:21:51] !log rsync complete. Running dpkg-reconfigure mailman on sodium [14:21:52] Logged the message, Master [14:28:25] !log Setup lily to route lists.wikimedia.org mails to sodium [14:28:26] Logged the message, Master [14:35:28] New patchset: Mark Bergsma; "Disable holding all mail on sodium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1959 [14:36:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1959 [14:36:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1959 [14:40:40] !log Disabled hold_domains on sodium and lily [14:40:42] Logged the message, Master [14:45:07] !log Changed service IP addresses of lists.wikimedia.org in DNS to US prefixes [14:45:09] Logged the message, Master [14:52:24] mark: did you bounce a msg to wikitech-l without changing the date? [14:54:02] I guess I did [14:54:11] oh well, done now [14:55:48] mark: well "today" is relative. anyway, forwarding on to the same place i saw sumanah forward the original (wikitech-ambassadors) [14:56:44] nice, google is now blocking some mail because it's coming from a new ip address [14:56:56] grey? [14:57:03] no, after data [14:57:18] mark: also, i mailed the TS announce list yesterday and it was held for moderation and then dab mailed the same thing (independantly) to the list. i think i'm still in the queue? can you reject me? and maybe river should not be the list admin any more? [14:57:44] that's not for me to decide [14:57:56] if the toolserver admins want that changed, they can file a request [14:58:05] sure, i can tell them to do that [14:58:09] thanks [14:58:14] but in the mean time cna you reject me? :) [14:58:17] can* [14:58:22] i'll have a look [14:59:30] mark: see bottom of http://mail.python.org/pipermail/mailman-i18n/2012-January/001765.html for some stats in case you care ;) [14:59:52] I don't see your mail [14:59:57] lunch [15:00:45] hrmmmmm [15:01:34] does this help? 17 Jan 2012 17:14:39 [15:01:37] UTC [15:01:53] no, it's simply not there [15:02:26] i guess someone got to it then. i never got a reject msg [15:03:43] (i mailed another non wikimedia list a few months ago and was moderated ~10-15 days after sending the msg... i think they just moderate all new ppl. anyway it was about an event and the msg got through a week after teh event ;( ) [15:15:03] New patchset: Mark Bergsma; "Google and possibly others are rate limiting our new ip, so use the old server(s) for delayed messages (for now)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1960 [15:15:33] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1960 [15:15:34] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1960 [15:17:38] PROBLEM - Host srv278 is DOWN: CRITICAL - Plugin timed out after 15 seconds [15:22:37] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [15:23:48] j #wikimedia-labs [15:48:02] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [15:59:54] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [16:11:38] PROBLEM - Recursive DNS on 208.80.152.131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [16:22:27] RECOVERY - Recursive DNS on 208.80.152.131 is OK: DNS OK: 6.189 seconds response time. www.wikipedia.org returns 208.80.152.201 [16:23:44] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [17:09:36] apergos: dataset1001 is here [17:10:09] yay [17:21:40] all kinds of shit came in today actually [17:25:07] RECOVERY - Host dataset1 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [17:31:36] Can someone add DNS for wikimedia.pl ? Or should I just put it in RT? [17:31:37] LeslieCarr: the sfp+ modules came in today [17:31:45] yay RobH !!! [17:31:49] now we can get that psw up [17:31:50] and then nag you guys about RT? [17:31:59] hexmode sure, gimme the rt # [17:32:05] That's already in RT [17:32:07] 2277 [17:32:22] LeslieCarr: the safest place for all the spares is in psw2 with the rest of the SFP? [17:32:33] but it's not DNS they want [17:32:35] figured its less likely to be stolen and the like [17:32:36] ah that's much more difficult [17:32:41] Reedy: please put rt tickets in the bz comment :) [17:32:53] I was asked to directly log it on irc [17:32:57] not on the bug [17:33:05] I did reference the bug on RT though :p [17:33:11] New patchset: Lcarr; "adding in curl package" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1961 [17:33:12] heh [17:33:13] k [17:33:25] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/1961 [17:34:53] LeslieCarr: so i need to replace every connection i made for you the other day right? [17:35:14] yep [17:36:45] (I'll be a little more excited about it tomorrow, I'm still mostly in sopa land today) [17:37:04] New patchset: Lcarr; "adding in curl package" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1961 [17:38:13] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1961 [17:38:14] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1961 [17:38:18] LeslieCarr: think i can bump hurricanes port a moment? [17:38:26] the sfp module lever isnt swung up properly [17:38:29] blocking the port below it. [17:38:40] RobH: let me see the traffic on it [17:38:45] might have to drainthe port, then you can bump [17:38:58] if we could do that it would be great [17:39:03] its not end of world, but annoying [17:40:29] !log Draining HE to perform maintenance on the physical port [17:40:30] Logged the message, Mistress of the network gear. [17:42:32] RobH: go for it [17:43:51] LeslieCarr: can pull he sfp? [17:43:56] the other conections are swapped [17:44:43] LeslieCarr: i assume ya meant go for it on pulling sfp, but gonna wait to confirm [17:47:17] RobH: yes [17:47:20] you can pull the sfp [17:47:21] sorry [17:47:25] ok, also, store these sfp+ in switch? [17:47:36] where would you be able to find them best ? [17:47:43] that's where to store them :) [17:47:48] HE pulled and fixed [17:47:58] well, we had some walk off a few years ago [17:48:11] so storing them in switch means you can see if/when they are removed and walk off [17:48:14] okay [17:48:16] cool [17:48:19] wasnt in this facility mind you [17:48:22] but meh, its ok habit [17:48:31] so should i shove the rest of these in psw1? [17:51:46] sure [17:51:56] we'll move everything off psw2 in the nearish future [17:52:54] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Wed Jan 18 17:52:36 UTC 2012 [17:56:12] mutante: so the srv199 repair, chris is dropping a ticekt for it to get reinstalled [17:56:20] mainboard swap means it doesnt know the nic and such [17:56:23] so reinstall is the best route [17:56:36] normally i drop those tickets and resolve the repairs with him, but he is doing that now [17:58:37] RobH: ok, i can re-install [17:58:39] New patchset: Asher; "fix vg naming on db builds, include new server range" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1962 [17:59:42] RobH: he can just move 2209 around [18:08:21]