[00:00:02] technically, I guess all labs systems could [00:00:12] but we don't tell people to change their passwords via instances [00:00:34] honestly we should disable that, if it isn't already [00:00:58] not disabled [00:03:26] labsconsole, gerrit, the CLI tools for making new accounts. what else? [00:03:50] that's it [00:03:55] that's 4 things, though [00:04:09] actually, gerrit doesn't modify ldap [00:04:27] labsconsole, formey, manganese, …. [00:04:33] may just be three things [00:04:39] what's labsconsole's box called? [00:04:43] virt0 [00:05:28] PROBLEM - MySQL Slave Delay on es1004 is CRITICAL: CRIT replication delay 226 seconds [00:12:34] o_O I got a "Your message to MediaWiki-CVS awaits moderator approval" bounce back email (for my commit) after doing a commit? … something seems wrong [00:13:03] mutante: ^ [00:16:12] 3am again [00:16:15] bye-bye [00:16:38] paravoid: later [00:16:44] I think that's what chad asked to be added earlier [00:18:49] RECOVERY - MySQL Slave Delay on es1004 is OK: OK replication delay 0 seconds [00:38:05] New review: Jeremyb; "We should not have this set locally in MediaWiki space on some wikis and customize per wiki in site ..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/11161 [00:56:18] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [01:11:06] hah, had a merge conflict with myself ;) [01:12:20] nb, about to do a large file upload to commons from hume [01:14:12] New patchset: Jeremyb; "make spacing consistent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5492 [01:14:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5492 [01:16:29] New review: Jeremyb; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/5492 [01:40:42] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 211 seconds [01:41:54] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 262 seconds [01:43:44] New patchset: Demon; "Allow some logs to supress comment-added notifications" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11176 [01:44:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/11176 [01:46:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [01:47:45] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 8 seconds [02:59:24] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [03:52:54] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [04:44:51] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [08:46:38] hello [08:46:52] today i have a network question :) [08:50:00] a user complained about the fact that he is only able to send small portions of data from the toolserver (esams/haarlem) to the wikis via a bot. he said it started on probably the 6th or 7th. [08:51:29] i saw you had some routing changes in esams on i think the 5th because of ipv6 but unfortunatelly i did not measure how the regular throughput to ie washington is. so i cannot test whether there was a significant change in the throughput. [08:52:16] do you have any ideas how much throughput you have regularily? did it change? [08:53:19] then we could exclude the network stuff as a cause [08:53:35] the guy opened a ticket at you site too: https://bugzilla.wikimedia.org/show_bug.cgi?id=37536 [09:01:43] is he making changes/edits? or just reading? [09:02:13] he is making edits [09:04:45] nosy: it is possible the IPv6 and IPv4 connectivity are slightly differents [09:05:16] so we are talking esams to tampa [09:05:43] I would not expect there to be a difference bth [09:05:45] *tbh [09:05:53] did anyone of you make any scp or something from one destination to the other a while ago? [09:06:08] not me [09:06:14] mutante: poke [09:06:48] I guess the workaround would be to raise the timeout :-] [09:07:25] ohh v4 indeed works [09:07:27] hrhrhr...as long as the other side does not close the connection in this time [09:07:52] hashar: do you know where the edits finally land on the wmf side? [09:07:59] do small edits works over IPv6 ? [09:08:09] then i could also look whether we travel via v6 or v4 [09:08:23] if they use the api, then one of the api servers [09:10:01] from fenari to willow: IPv4 pass through tele2, v6 through hurricane electric [09:10:29] both paths look fine right now, that might be another issue entirely [09:13:08] yay reviews done whew! [09:13:14] hashar: ok thank you for testing this [09:13:51] hashar: how did you test? can you upload from willow to fenari? [09:14:06] if you do, do it to /tmp :-P) [09:14:08] I just did a traceroute from the US to willow [09:14:19] I don't have an access on willow :-] [09:14:24] oh i see [09:14:31] so i will poke dabpunkt for a test [09:14:34] I have no toolserver access either, not for years now [09:14:41] apergos: I will just use my ~ over NFS, seems better than using local disk *grin* [09:14:53] :-P :-P [09:15:42] you know a local /home with sym links to /home/username/nfsdir might keep people from doing silly things (though it's gotten much better) [09:15:45] anyways... [09:15:45] grr http://dumps.wikimedia.org/ is not v6 awareè [09:15:56] are we not proxied? [09:16:20] oh, right, no need, just need to set up lighty [09:16:28] I forget if it's been given an address though [09:16:47] I vaguely think not [09:16:54] * apergos adds it to list ot ask about [09:17:53] nosy: one possible test would be to download some long article using curl [09:18:08] nosy: the curl command can be forced to v4 or v6 (with --ipv4 or --ipv6) [09:18:10] oh golly I hate gerrit already this morning [09:18:14] ahah [09:18:18] use the command line ? [09:18:35] want to add reviewer, don't know their gerrit name, no gneral list of reviewers that's clickable... why not?? [09:18:44] ohh [09:18:52] well that is an issue for Ryan [09:19:10] the LDAP schema is messy, you need to know the Labs login :-( [09:19:15] who are you looking for? [09:19:28] and as promised yesterday I have started a list of issues with that as the first one! [09:19:33] hashar: ok, good idea. do you think the routing path is the same back and forth? [09:19:44] no guarantee about that [09:19:45] i think you said its not [09:19:47] :D [09:19:50] we often have assymetric paths [09:20:00] nosy: I have no idea, from the traceroute it looks like the paths are symmetrics right now [09:20:13] but anyway its probably really sensible to test both ways [09:20:16] but hard to know for sure without having a trace from willow to USA :-] [09:20:21] I don't see how you can tell from the traceroute one way [09:20:28] you want a trace? [09:20:44] apergos: you can if the paths have a different number of routers. The TTL will be differents [09:21:07] it gets through tele2, then equix-dc2.wmf [09:21:37] nr 9 is fenari in my traceroute [09:22:26] you really want to do a download over v4 and another one over v6 [09:22:33] root@willow:~# traceroute -A inet6 fenari.wikimedia.org [09:22:33] traceroute: getaddrinfo: no address for the specified node name [09:22:38] that would let you know if the bandwidth iran issue [09:22:41] ok i will do so [09:24:27] the times are the same [09:26:29] i asked dabpunkt to test the other way [09:32:44] so that is most probably not a bandwidth issue [09:34:27] well that was painful (finding brion vibber's user name: not actually easy) [09:34:51] brion ? [09:34:53] davibber [09:34:55] vbrion [09:34:57] brionv [09:35:00] well ... [09:35:04] yeah that is cumbersome :-( [09:35:07] no, for the reviewer it is Brion VIBBER [09:35:12] ahahah [09:35:16] yeahright [09:35:24] you have to poke Ryan about it :-D [09:35:26] so a simple list... sure would be nice [09:35:32] I shall, I shall! [09:35:35] need to update the LDAP schema to add anew field [09:35:47] I am ranting about it from time to time [09:35:51] Chad does too [09:35:58] so one day, maybe we will fix that :-] [09:36:25] I would love a listtle link that would let me choose to see: list of projects. list of user emails. list of reviewer names. list of project owners. [09:36:46] (select one from dropdown list on any gerrit page) [09:40:19] New review: ArielGlenn; "and that's "unused thumbs" purge helper, not " [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11202 [09:40:21] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/11202 [09:54:05] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [09:54:06] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [09:54:06] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [09:58:56] maerlant & professor [09:58:58] lovely names [09:59:02] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [10:00:23] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [10:03:27] New review: Nikerabbit; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 1; - https://gerrit.wikimedia.org/r/10756 [10:04:48] New review: Nikerabbit; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/10707 [10:45:34] New patchset: ArielGlenn; "with --nooverwrite, don't overwrite tarballs at the dojob stage either" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/11276 [10:47:05] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11276 [10:47:07] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/11276 [10:57:03] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [11:21:11] Logged the message, Master [11:21:16] Logged the message, Master [11:21:21] Logged the message, Master [11:29:51] New review: SPQRobin; "@Hashar: The first one was a small change and was already marked OK. Recommitting them separately no..." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/10756 [12:00:13] apergos: about gerrit lacking list of users, projects … We have two bugs open for that : https://bugzilla.wikimedia.org/show_bug.cgi?id=35508 & https://bugzilla.wikimedia.org/show_bug.cgi?id=35510 [12:00:29] checking... [12:01:53] good it's not just me that feels the lack [12:07:43] New review: Hashar; "So I will do it." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/10756 [12:21:54] New patchset: Hashar; "(bug 33809) redirect unknow projects to a meta page" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11277 [12:21:55] New patchset: Hashar; "(bug 36033) unknown wikisource redirected to multinlingual one" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11278 [12:22:02] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11277 [12:22:04] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11278 [12:23:25] New review: Hashar; "I have split this change in two new Gerrit changes:" [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/10756 [12:24:29] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11277 [12:24:32] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11277 [12:27:36] Change abandoned: SPQRobin; "Oh, okay, thanks anyway for splitting :)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/10756 [12:28:01] New review: SPQRobin; "(no comment)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11278 [12:32:41] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11278 [12:33:54] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11278 [12:33:56] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11278 [12:34:00] going to deploy that [12:34:52] best thing about listening to house music [12:35:03] you can stroke your keyboard keys using the same rythm [12:35:11] aka 64 keystrokes per minutes :-] [12:35:17] or maybe just 128 [12:35:18] hmm [12:35:29] * hashar finds out a keystrokes counting software [12:40:22] New review: Hashar; "Applied to live site." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11277 [12:40:24] New review: Hashar; "Applied to live site." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11278 [12:42:18] PROBLEM - Puppet freshness on stat1 is CRITICAL: Puppet has not run in the last 10 hours [12:43:27] hashar: http://en.wikipedia.org/wiki/Frets_on_Fire [12:43:43] ohh [12:43:47] I have the non-free / for pay version [12:43:57] oh no [12:43:58] gmm [12:44:01] that is guitar hero [12:44:19] hmm [12:44:21] that article is wrong [12:44:29] winner of the Assembly, but yet written in Python [12:47:03] bah http://download.wikimedia.org/mediawiki/1.19/ is redirected to http://dumps.wikimedia.org/mediawiki/1.19/ [12:47:03] ;) [12:52:58] can someone check why there is no http://download.wikimedia.org/mediawiki/1.19/mediawiki-i18n-1.19.1.patch.gz [12:53:13] it was linked on download page before I removed it [12:53:18] the link [12:53:24] file should exist [12:54:39] petan: isn't there a bug report for that? [12:54:48] Sam should be able to tell [13:00:18] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [13:03:21] hashar: I don't know [13:10:21] ottomata: what do you mean might be a problem with /etc/issue on stat1? [13:12:28] Ubuntu 10.04 [13:12:29] right? [13:12:32] New patchset: Ottomata; "files/nagios/check_udp2log_log_age - I renamed the Zero filter logs, need to do this here as well." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11281 [13:12:34] that's not precise, is it? [13:12:56] yesterday Leslie took down the machine to reinstall precise [13:12:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/11281 [13:13:08] ottomata: precise is 12.xx something [13:13:11] right [13:13:18] oh [13:13:20] I just finished restoring after the supposed upgrade reinstall to precise [13:13:26] there is that website with lot of stuff on it let me find [13:13:28] the link [13:13:48] ottomata: http://tinyurl.com/79blsmo [13:13:58] those guys are great at listing stuff [13:14:14] aye [13:14:14] cool [13:14:19] uhm [13:14:39] does it say anywhere in there that they decided to keep /etc/issue saying 10.04 even if you upgrade? :p [13:14:53] lol [13:17:28] could someone approve this for me? [13:17:28] https://gerrit.wikimedia.org/r/11281 [13:17:48] mutante, maybe could do me a kindness? [13:17:50] :) [13:19:02] New review: Dzahn; "sure, we have done these filter changes for mobile before" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11281 [13:19:04] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11281 [13:25:43] ottomata: i was a bit worried about /a , how did you restore it since it failed in amanda? [13:25:53] i heard about a flash drive? [13:26:05] ottomata: what? leslie reinstalled it with lucid? [13:27:24] you gotta be kidding me [13:27:27] not precise? [13:28:40] yup :) [13:28:47] shall I reinstall it right now? [13:28:47] mutante, yeah we had a usb drive plugged in [13:28:50] cp -a [13:28:51] ummmm [13:28:54] yes, should be fine [13:28:59] everything is still on the usb drive [13:29:09] i'm sorry about that [13:30:49] erik and andre are currently logged in [13:30:51] let me email them [13:30:53] but yeah it should be fine [13:33:31] ok then i'll move manutius first... [13:37:52] ottomata: let me know when I can start [13:39:54] RECOVERY - udp2log log age for oxygen on oxygen is OK: OK: all log files active [13:40:47] mark, go ahead whenever you are ready [13:41:00] ok, will do so now [13:45:44] leslie gets to wear the clowns nose today ;) [13:46:53] PROBLEM - Host stat1 is DOWN: CRITICAL - Host Unreachable (208.80.152.146) [13:47:49] poor leslie... [13:51:05] mark: i have the mc servers here in tampa to rack [13:51:12] but they have the SFP type cards, where did you want htem? [13:51:42] RobH: I honestly don't know [13:51:49] i've lost track with all the purchases and rack assignments [13:51:58] they will need an EX4500 in the same rack, that's for sure [13:52:13] ottomata: would it help to keep the existing data on /a? [13:52:17] I don't need to format that [13:52:26] RECOVERY - Host stat1 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [13:53:38] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [13:53:48] ottomata: if permissions matter in /a and you just said "cp" you might want to copy them again (with rsync -avp or so) [13:54:20] i did cp -a [13:54:31] huh there's a /dev/sdb which is not the external drive [13:54:33] some raid array [13:54:35] rsync -a for /home [13:54:36] checking what that is before I reinstall [13:54:58] um mark, yeah! [13:55:05] that would be easy [13:55:28] dunno why we didn't just think of that before :p [13:55:43] couldn't do that before, the previous install wasn't LVM I think [13:55:49] (or was it) [13:55:53] PROBLEM - SSH on stat1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:36] i think it was when i looked at size of /a for backup [13:56:54] dunno [13:59:17] ok [13:59:21] mark: ok, i have empty sdtpa d-3 [13:59:24] so will use that [13:59:29] PROBLEM - Host stat1 is DOWN: CRITICAL - Host Unreachable (208.80.152.146) [13:59:38] according to the raid bios, stat1 has one virtual disk of 10 TB (12 drives raid6) [14:00:20] whoa really [14:00:21] 12 drives? [14:00:31] 12x 1TB [14:01:36] wish we had that on the analyitcs cluster machines :/ [14:02:10] don't you? [14:02:17] New patchset: ArielGlenn; "script to copy the dump deployment files around to the cluster hosts" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/11285 [14:02:18] 8 drives, right? [14:03:20] 8 drives [14:03:24] not 1TB either [14:03:32] yeah, 8i think ~300G? [14:03:35] though they are SAS [14:03:39] das true! [14:03:41] ah! [14:03:47] it makes the usb drive /dev/sda [14:03:49] that's silly [14:03:54] oh when it rebooted? [14:03:55] lemme disable that temporarily in bios [14:03:57] yeah [14:03:59] aye [14:04:10] ye, hence Ciscos start with sdc in partman [14:06:46] "User-accessible USB ports: All ports OFF" [14:06:50] I so don't trust chris & rob ;-) [14:09:21] mutante: weird that some do and some don't, though [14:09:53] indeed, the first 10 analytics bunch did not seem to be different though [14:10:11] or no wait, 1 one of them [14:10:18] the first afair [14:10:35] RECOVERY - Host stat1 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [14:10:40] and all from same donation date [14:10:41] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/9109 [14:10:44] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9109 [14:10:50] going to be annoyed if that breaks gerrit ^^ [14:11:58] that said, I can't complain to devs about not reviewing and merging my code if I don't review/merge theirs :) [14:12:28] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/10956 [14:12:28] I so can [14:12:33] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/10956 [14:12:45] yeah, I'm full of it. I can too [14:12:53] I review for them way more than they review for me [14:17:07] ottomata: who runs less.ly? do you still need that external git? i heard you got a wmf repo to replace it if im not mistaken. in the pending change 11042 would be good to update that before. seems like a good time to talk about it now because it is just being blocked by "precise upgrade" which happens momentarily [14:17:47] PROBLEM - Host stat1 is DOWN: CRITICAL - Host Unreachable (208.80.152.146) [14:17:51] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/11176 [14:18:11] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/11176 [14:18:18] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11176 [14:18:19] ^^ who wants to bet that's going to break the hooks? [14:18:20] (this is all for "newer nodejs" package) [14:18:33] and installing "reportcard" [14:20:06] ha, well, sort of [14:20:07] i mean [14:20:13] mark really wants us to upgrade [14:20:22] we tried to do this 6 weeks ago because of that, but [14:20:28] and indeed the fucking hooks are broken [14:20:33] at some point gave up because analytics didn't really care if we upgraded [14:20:35] of course there are more reasons to upgrade [14:20:56] RECOVERY - SSH on stat1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:21:03] but now it'd be nice so we can have the newer nodejs packages available, without having to add more .debs to the wikimedia apt repo [14:21:05] RECOVERY - Host stat1 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [14:21:11] PLUS mark wants us to upgrade [14:21:23] if it was just nodejs alone, we probably wouldn't do it [14:21:24] all fine, just wondering about the git repo [14:21:30] oh yeah, that is going to change [14:21:38] that commit is not ready yet, just wanted to put it up [14:21:46] kk [14:21:55] I minus oned it so no one would bother with it yet :) [14:22:04] ok [14:22:31] was going to offer to create a gerrit project for it if needed [14:22:38] but i heard you had one [14:22:45] thanks! I have create powers now! [14:22:51] alright:) [14:23:05] so, we are actually hosting the main bit of the reportcard repo on wikimedia's github, but I think there will be a mirror to gerrit [14:23:09] which is where we will clone from [14:23:16] i thiiiink, not sure about that [14:23:34] The framework that david wrote for reportcard is reallly awesome and very generic [14:23:37] so we wanted to open source it [14:24:15] sounds like a good thing for sure [14:26:53] running puppet for the first time [14:28:33] it can't install udp-filter [14:28:37] probably not in the precise apt repo yet [14:29:06] hrmm, i wanna ditch db21-db24 (old suns) and use the reclaimed space for 5 new dbs. [14:29:14] i guess we unpack and stack to chat with asher later [14:29:38] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Thu Jun 14 14:29:09 UTC 2012 [14:29:41] also mysql-client [14:30:07] !log Reinstalled stat1 with Ubuntu Precise [14:30:08] New review: Ryan Lane; "test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11176 [14:30:12] ugh, this python was really fucked up [14:30:13] Logged the message, Master [14:30:22] ottomata: stat1 is rebooting, when it's back up you can have it [14:30:40] puppet has a few problems, let me know how we can help in getting them resolved [14:30:49] (e.g. by putting a udp-filter package for precise in our apt repo) [14:31:58] ok cool [14:32:11] hmm, that would be good [14:32:14] i'm sure we will need that [14:32:56] who built it last time? [14:33:05] New patchset: Ryan Lane; "Fix gerrit hooks. Follow up to Ieabdf1ae." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11286 [14:33:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/11286 [14:33:41] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/11286 [14:33:44] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11286 [14:35:00] New review: Ryan Lane; "test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11286 [14:35:08] well, that fixed that [14:36:56] 0 review requests! \o/ [14:43:11] RobH: do you remember specifics for logging into the remote management of the IBMs? [14:43:16] I'm trying to go onto the manutius console [14:43:22] mark@fenari:~$ ssh root@manutius.mgmt [14:43:22] Received disconnect from 10.1.6.51: 2: Too many authentication failures for root [14:45:20] maybe it's another one that uses admin? :) [14:45:29] or maybe it uses the old mgmt password? [14:45:41] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [14:48:55] admin asks for a password at least [14:49:00] but doesn't take the one I give it [14:49:06] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11285 [14:49:08] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/11285 [14:49:27] i'll just reboot the box and change the vlan while it boots [14:49:28] easier [14:50:57] the ibm stuff is on wikitech that i know of [14:51:08] that doesn't say anything about this [14:51:13] except "you must login over ssh" [14:51:16] (well I could guess that part) [14:54:18] gah [14:54:24] I log in on one of our foundry core switches [14:54:30] and can't find any of the tampa servers on there [14:54:38] took me a minute to realize I'm logged into csw1-esams [14:55:47] um, diederik [14:55:51] or maybe me [14:55:53] build udp-filter [14:57:25] RobH: so… bout those ciscos in tampa.... [14:59:42] PROBLEM - Host manutius is DOWN: PING CRITICAL - Packet loss = 100% [15:01:14] New patchset: Jeremyb; "Added php-htmlpurifier to go w/ I2ddf3e2ab00584de" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8348 [15:01:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8348 [15:01:56] so mark, curious as to why I can't install mysql-server-5.1 on precise [15:02:28] hmm, precise upgrades to 5.5? [15:02:36] or is 5.1 in the lucid wikimedia repo? [15:02:42]