[13:32:16] hey all [13:32:21] hi [13:32:30] someone need anything :P [13:32:37] * aude wonders if i can log into my new instance now [13:32:42] "new" = setup yesterday [13:32:56] probably yes [13:32:57] :P [13:33:02] !access [13:33:02] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [13:33:24] you definitely should be able to do that, unless it's broken [13:33:38] !console [13:33:39] in case you want to see what is happening on terminal of your vm, check console output [13:33:43] check that [13:33:53] you should be able to see if it's ok or not [13:34:09] !instances this is where you find it [13:34:09] need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance want resources? use !resource [13:34:15] it's been doing dhclient: bound to 10.4.0.229 -- renewal in 55 seconds. [13:34:19] repeatedly since yesterday [13:34:23] hm that's ok [13:34:27] no i can't login [13:34:31] that means it renew its IP address [13:34:35] ok [13:34:35] ok, what is error you get [13:34:55] Permission denied (publickey). [13:35:00] but my other instances work [13:35:21] it probably broke [13:35:23] e.g. ssh maps-test3.pmtpa.wmflabs [13:35:25] works great [13:35:26] puppet run didn't run I guess [13:35:30] right [13:35:37] you should delete it and create again [13:35:45] also been getting LDAP Search failed errors [13:35:47] creation doesn't take more than 2 minutes [13:35:56] yes that's a part of that error [13:35:57] i can try :) [13:36:17] sometimes puppet can't connect to master [13:36:25] seems to be the case [13:36:42] you think a new instance now might work and i delete this one? [13:36:48] seemed borked yesterday [13:37:18] * aude trying [13:37:26] ok [13:39:08] * aude back in a few minutes.... escaping the rain [13:49:21] * Damianz naws petan [13:49:47] * petan stabs Damianz [13:50:04] petan: Would you happen to know how to fix thumbs on beta? Someone wiped the symlink out via apt the other day and the only thing I couldn't figure out how to make work was the thumbnails. [13:50:19] Symlinks to the upload6 folder is there, paths look ok they just error with can't find source [13:50:38] eh [13:50:48] you definitely should make a ticket for this [13:51:09] I don't want to touch this because I don't want to conflict with something hashar might be on [13:51:14] We actually use bz? :P [13:51:18] yes [13:51:22] product labs [13:51:30] category beta [13:51:31] :P [13:51:51] fuck me [13:51:55] no thanks [13:52:00] petan: my new instance is also doing the LDAP Search failed thing [13:52:01] I can't make the damn text box work in c# [13:52:09] lol [13:52:17] this suck too [13:52:24] it makes all hashes link [13:52:25] meh [13:52:35] Jul 29 13:43:07 i-00000373 puppet-agent[3935]: Starting Puppet client version 2.7.11 [13:52:40] aude really? is there some puppet error in console? [13:52:45] Jul 29 13:43:07 i-00000373 puppet-agent[3935]: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed when searching for node i-00000373.pmtpa.wmflabs: [13:52:49] aah [13:52:50] and ldap failed [13:52:51] that suck [13:52:59] borked since yesterday [13:53:01] this mean it broke again [13:53:07] it's likely broken on side of puppet [13:53:18] aude can you create a ticket for this, then link me to it [13:53:42] where do i create a ticket? [13:53:46] !bz [13:53:46] https://bugzilla.wikimedia.org/$1 [13:53:52] i emailed labs list about it yesterday [13:53:53] !bz new [13:53:53] https://bugzilla.wikimedia.org/new [13:53:54] ok [13:54:05] eh, ok this link doesn't work [13:54:25] just open bz [13:54:26] :P [13:54:27] in wikimedia-operations there's been puppetmaster 400 errors being logged [13:54:32] ok [13:54:34] hm [13:54:44] problem is that not all of ops are watching labs on sunday [13:54:50] of course these things happen on the weekend ;) [13:55:09] they happen even on work days, but usually someone fix them [13:55:19] * aude kicks puppetmaster [13:55:25] I don't have enough powers to fix this [13:55:29] right [13:55:36] brb [13:55:56] * aude can wait until tomorrow, though i'm doing stuff on the side of my day job (e.g. labs on teh weekend) [13:59:57] oooh cool, it auto adds the cc users [14:04:05] seems it's already reported: https://bugzilla.wikimedia.org/show_bug.cgi?id=38639 [14:04:39] That was from last week IIRC [14:06:53] yeah and it's a problem again [14:07:23] seems the same problem... restarting the server seemed to work but then problem happens again after a while [14:07:37] as technically knowledgable i am about these things [14:14:40] IIRC we have 2 opendj boxes now, so loosing one shouldn't cause massive issues eventhough it seems to there [14:22:11] mh [14:22:12] back [14:22:53] Damianz my client now works even on linux, so if you wanted to switch to some proper irc client let me know :P [14:23:25] I had to write own control which draw text, but it eats 10 times less memory and is so much faster <3 [14:24:28] microsoft's components are shiny but horribly resource expensive and untweakable [14:24:33] * controls [14:27:07] hi ^demon [14:27:14] <^demon> Hello. [14:27:40] you're not a labs person (?) but i'm getting ldap errors on new instances [14:27:47] https://bugzilla.wikimedia.org/show_bug.cgi?id=38639 [14:28:04] * aude assumes ryan is the only one who can do anything about it [14:29:57] <^demon> Yeah, I can't help there. [14:30:20] ok :/ [14:30:20] petan: Whatever is wrong with irssi? [14:30:55] There's nothing to annoyg ^demon about now, gerrit is actually useable [14:30:57] Damianz it wasn't written by me [14:31:02] that's what [14:31:03] :D [14:31:14] <^demon> Then let's not ping me senselessly. I knew it was a mistake to sign into IRC on a Sunday. [14:31:22] heh [14:31:31] <^demon> aude: Not you, you had a legit question :) [14:31:47] * aude wish some volunteers had ops powers to fix things on the weekend [14:31:50] I really feel like I want to ping ^demon for some silly reason :D [14:32:03] <^demon> Man, my /ignore list is gonna get big today. [14:32:04] * Damianz wonders if chad hilights ^demon [14:32:08] :P [14:32:19] maybe he even highlight Chad [14:32:26] <^demon> Damianz: No, I only ping on my nick. I do not ping on "Chad" "Gerrit" or anything else that someone may type. [14:32:28] ^demon did it highlught you? [14:32:36] git [14:32:40] <^demon> "Unicorns" does not ping me, contrary to what you may believe. [14:32:41] gerrit gerrit gerrit ;) [14:32:45] porn [14:32:46] Oh you should totally add git and gerrit to your hilight lise :D [14:32:52] how many people I just pinged :P [14:32:53] s/lise/list/ [14:32:58] gerrit generally works for me [14:33:35] new pretty colors would be nice thought but i'm not good at picking colors [14:33:39] Gerrit is awesome now, it's like someone let it snort cocaine for a week [14:33:47] lol [14:35:21] Hmm to try and update this box from fedora 15 to 17, re-install it to ubuntu or buy some hardware and give it a real raid then re-install fedora 17/ubuntu on it. [14:37:51] <^demon> Damianz: We gave gerrit a couple of kilos of fresh colombian blow. That's why gerrit's so speedy these days. [14:38:33] Must be an expensive architecture :D [14:38:55] <^demon> You have no idea. The hamsters got addicted :( [14:39:45] Poor hamsters [14:50:11] ^demon, can you add me to the gerrit-dev project? [14:50:35] <^demon> One moment. [14:50:43] btw, that email you sent me, I don't know if you expected it to go to wikitech-l [14:51:25] <^demon> No, it was sent offlist. [14:52:00] <^demon> Added you to the project. Right now we're doing a manual install of gerrit in /var/lib/gerrit2. Don't run puppet right now, manifests are broken on labs :) [14:52:20] <^demon> Unless you're trying to fix them. In which thanks :D [14:52:32] 07/29/2012 - 14:52:32 - Created a home directory for platonides in project(s): gerrit [14:53:31] 07/29/2012 - 14:53:31 - User platonides may have been modified in LDAP or locally, updating key in project(s): gerrit [14:53:53] * Damianz hides from Platonides [14:58:05] Damianz? [14:58:20] Hi [14:58:41] hi Damianz XD [14:58:44] I am back [14:58:48] <^demon> aude: I could pick some new colors, but they might not be pretty :p [14:58:58] <^demon> How about red backgrounds with blue text? [14:59:12] black and white text pls [14:59:16] Black background, black text? [14:59:19] like in terminal [14:59:51] <^demon> My terminal is kind of an off-beige with dark dark brown text. [14:59:51] eye friendly [14:59:56] <^demon> Very easy on the eyes as well. [15:00:11] hm... white background usually hurts when u look in that for long time [15:00:14] My terminal is black with white text, but my text editor is purple [15:00:33] Damianz ur terminal is not black with black text? [15:00:39] how noobish [15:00:45] Nope, it use to be pink with black text [15:00:57] <^demon> My bash prompt: http://www.mediawiki.org/wiki/File:ChadBash.png [15:01:01] ^demon, how do you connect there? [15:01:09] my terminal is transparent [15:01:10] :P [15:01:12] I like it [15:01:13] platonides@bastion1:~$ ssh gerrit-dev.wmflabs.org [15:01:13] ssh: connect to host gerrit-dev.wmflabs.org port 22: Connection timed out [15:01:20] That's a little hmm [15:01:26] <^demon> Platonides: The instance name is "gerrit" not "gerrit-dev" [15:01:48] but has a gerrit-dev domain for the outside workd [15:01:48] ^demon that's not white [15:01:50] :P [15:01:52] that's confusing :P [15:01:57] <^demon> Platonides: Yes, I didn't set it up. [15:02:00] <^demon> Blame Roan, I think :) [15:02:21] Platonides we really need to enforce some rule for hostnames [15:02:24] :D [15:02:39] I found another bug, brb [15:02:46] * Damianz nukes the bug [15:02:48] ok, fixed at my ~/.ssh/config [15:07:11] {{fixed}} [15:07:13] :P [15:07:21] I hope [15:07:57] but I just found another one [15:07:58] :/ [15:08:23] this is like 200th time I restart ir client... [15:08:40] if I didn't have a bouncer I would be spammy [15:19:50] ^demon, what's the db password? [15:20:09] <^demon> Lemme login. [15:21:25] <^demon> Hmm, it's supposed to be in /var/lib/gerrit2/review_site/etc/secure.config [15:21:31] <^demon> But I'm having issues sudoing. [15:21:58] <^demon> Oh, prolly cuz of LDAP issues. [15:22:15] I thought it was just me [15:22:27] <^demon> Nope, seems not. [15:22:32] <^demon> aude was complaining about it earlier. [15:23:13] yep, it is also happening on other project [15:23:49] <^demon> So yeah, when we can sudo again, that's where it's at. [15:23:52] ok [15:24:15] <^demon> Gerrit has two config files, gerrit.config and secure.config. You can put values in either, but it's so you can move super-secret stuff to a file only owned by gerrit. [15:29:11] * ^demon goes back to olympic coverage [15:32:33] <^demon> Noooo, stupid nbc live stream died :( [19:39:00] <^demon> !log gerrit Restarting gerrit on labs--quick note--if you're going to start/stop gerrit, please use gerrit.sh so there's a pid. [19:39:01] Logged the message, Master [21:17:13] <'borrows' ^demon's cable bandwidth, to ensure no live olympics> [21:17:39] <^demon> Nooo. There's like 300-something hours of coverage. Gotta watch them all. [21:18:36] [22:06:23] !puppetmaster::self [22:06:23] https://labsconsole.wikimedia.org/wiki/Help:SelfHostedPuppet [22:38:09] how do you force a puppet run? i poked around the docs but can't seem to get the exact syntax [22:41:53] you run puppetd -tv [22:42:11] note that it's puppetd what you run, not puppet [22:42:35] and I'm not sure if the puppet problems were already fixed [22:51:22] Platonides: aha, thanks. [23:09:46] Platonides: they are. fixed this morning. was an ldap issue. [23:13:21] oh? [23:13:24] what happened? [23:15:09] couldn't connect to the daemon [23:15:16] iptables rules were there [23:15:37] I didn't have much time to investigate [23:15:43] so, I did a restart [23:15:51] I'm going to investigate the clients, too, though [23:15:59] since they didn't fail to the secondary [23:18:26] while at it, can we get rid of the annoying "replication failed to virt1" messages that are logged every second or so? [23:18:39] yes [23:18:44] I'm going to upgrade the servers [23:18:49] do we need an upgrade for that? [23:18:53] right now it's a pain in the ass to remove a replica [23:18:56] argh. [23:18:57] it's possible, though [23:19:07] we're using a pretty old version [23:20:49] Ryan_Lane: Will it cause problems for others, if I try and import a reasonably large postgresql database onto /mnt ? [23:21:04] what's reasonable large? [23:21:27] we need a volume service for this kind of stuff :( [23:21:36] possibly around 100Gb [23:21:37] really we need database services [23:21:39] hm [23:21:46] apmon: on a new or old instance? [23:21:53] err [23:21:56] I just created the instance this moring [23:21:59] ah ok [23:22:02] it should be fine [23:22:51] I am trying to test some things about osm2pgsql importing a europe extract [23:23:08] please let me know if it causes any issues. [23:23:10] paravoid: even though the ceph performance tests aren't going wonderful. I'm thinking it still may be a good idea just for project storage [23:23:15] apmon: ok [23:23:20] thanks [23:24:03] paravoid: one of the reasons I'm thinking this is that we can use ceph as both shared storage (via cephfs) and for a volume service [23:24:35] and if we're really lucky it won't have the bugs that gluster currently has with regards to locking an directio