[00:00:01] df -h [00:01:12] it should be at /user/local/search/ls2/bin [00:01:30] should I sudo ant ...? [00:01:33] /usr/local? [00:01:38] yes [00:01:47] if it's in /usr, it requires root [00:02:09] ok [00:05:05] where should LocalSettings.php be located ? [00:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [00:13:09] ok it's indexing [00:13:31] I hope it won't run out of space [00:17:38] hi OrenBochman [00:19:05] hi [00:19:09] wassup [00:20:09] sumanah: do you have petan and notpeter's email ? [00:20:29] OrenBochman: yes, I think so, just a moment [00:20:40] OrenBochman: you may find http://toolserver.org/~krinkle/wikimedia-svn-users.php useful as a directory [00:21:01] OrenBochman: they're both listed in there [00:21:30] sumanah: we need to coordinate some work tommorow [00:22:26] look like search-test is indexing [00:24:49] OrenBochman: in my opinion you should email wikitech-l to update the dev community of what you're up to and what your needs are [00:24:56] hexmode: did you already talk with OrenBochman about what you need? [00:25:42] sumanah: OrenBochman and I talked. He is working with petan and (I think) notpeter [00:25:44] we talked - we will need assistence from notpeter [00:26:08] OrenBochman: let me see if he is online [00:26:23] i am [00:26:55] :) Well, I pinged him in -operations :) [00:27:02] thanks [00:27:23] he isn't around. I pinged him an hour or so ago [00:27:31] hrm [00:28:00] Ryan_Lane: guess I'll have to catch him tmw AM then? [00:28:06] I guess so [00:28:15] OrenBochman: when will you be around? [00:28:49] OrenBochman: is there anything I can ask notpeter to do if you're not around when he is available? [00:29:27] I'll write him an email [00:29:34] can I cc you as well [00:30:22] OrenBochman: pm'd my email ;) [00:32:30] ok [00:33:04] what can I call this project - search deployment in .... [00:34:00] are you based in europe ? [00:34:03] That sounds fine to me [00:34:13] I'm on the East Coast USA [00:34:29] so UTC+5 ? [00:34:36] in budapest [00:34:38] * hexmode has to go look [00:34:45] it's realy late here [00:35:30] yeah, time for bed. We can pick this up tomorrow and hope notpeter shows up [00:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [00:36:45] how can I tell to run a command even after I disconnect [00:37:44] OrenBochman: Use screen [00:38:21] screen [00:40:17] clear [00:42:53] hexmode: is there going to be a deployment of the dumping [00:44:16] OrenBochman: petan and johnduhart have been managing imports. I think they're only taking a small % of the pages ... except for simple [00:45:20] yep [00:45:58] I mean the machine that runs the wikipedia dumps - from the database - is that going to be in the deployment [00:46:21] deployment-sql? [00:46:23] yes [01:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [01:25:55] i saw a mention from petan... [01:26:05] back in ~10 mins [01:33:47] got a problem [01:34:10] when running service apache2 restart I get [01:34:12] * Restarting web server apache2 [01:34:13] (13)Permission denied: make_sock: could not bind to address 0.0.0.0:80 [01:34:15] no listening sockets available, shutting down [01:34:16] Unable to open logs [01:34:18] ...fail! [01:34:30] jeremyb: could you have a look why apache wont restart [01:34:53] btw search is working on search-test [01:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [01:37:18] ok, here now [01:37:27] * jeremyb was on a bus :) [01:37:33] ;-) [01:37:59] OrenBochman: where was that apache issue? search-test? [01:38:04] I've been on a bus atram a trolibus a train and in my car [01:38:54] I modified the LocalSettings to work with search [01:39:11] 10 01:34:13 < OrenBochman> (13)Permission denied: make_sock: could not bind to address 0.0.0.0:80 [01:39:17] where is that? search-test? [01:39:37] yes [01:40:03] were you running it with sudo? [01:40:07] yes [01:40:17] anyway, worked for me... [01:40:18] $ sudo service apache2 restart * Restarting web server apache2 ... waiting ...done. [01:40:43] works now [01:41:21] but I still get results from the stabase not from the lsearchd [01:41:53] but I've tested lsearchd with curl and it works [01:41:55] let me see [01:41:58] arrrrr [01:42:20] TLAPD? [01:42:29] ? [01:43:05] I made a copy of LocalSettings before I messed with it [01:43:43] it's LocalSettings.php.bak [01:44:11] OrenBochman: https://en.wikipedia.org/wiki/TLAPD ;) [01:44:18] OrenBochman: anyway, looking [01:46:06] hillerious [01:48:53] Ryan_Lane: Does the nova database persist over a reboot? (That seems like a silly question, but mine does not. I suspect it's because devstack is wiping it clean...) [01:49:12] devstack is wiping yours clean [01:49:19] it normally persists [01:49:20] ok, i won't worry about that then :) [01:49:29] and so does the ampq [01:49:52] * Ryan_Lane is reading about the ampq right now :D [01:50:53] I think I knew that queue messages were supposed to persist over reboots but it still strikes me as unlikely. [01:50:55] I need to move virt1 into the cluster, and make a virt0 that will replace it [01:51:02] heh [01:51:16] ampq supports persistent queues and non-persistent queues [01:51:38] I think most if not all of the openstack queues are persistent [01:53:30] hmm. rabbitmq supports ldap [01:53:42] that could be interesting for use inside of instances [01:53:45] for auth? [01:53:49] yeah [01:54:20] is firewall not enough? [01:54:54] how would you avoid users messing up other user's queues? [01:55:31] i'm not seeing where it would be used [01:55:41] this would be for a user to make a queue in one project, and pick up messages in another [01:55:51] this is for e.g. jobqueue AAS? [01:55:55] for instance, I'd love to have a unified irc bot [01:56:12] where an IRC bot runs in one project, but can pick up messages from everywhere [01:56:24] and I can stick a message in from anywhere [01:56:58] it sucks to need to run a different bot for every single ircecho we need [01:57:15] sure [01:57:25] if we had a queue, all the ircecho daemons could stick messages into a queue, and one bot could spit out the messages [01:57:30] btw, only marginally related but have you seen pubsubhubbub? [01:59:20] * jeremyb sees tcp6 0 0 :::8123 :::* LISTEN 15915/java [01:59:45] that the search [01:59:48] lots of ports it's listening on [01:59:56] but i guess that's the nature of java... [02:00:03] (profiler attaching port, etc.) [02:00:18] it listeing on 8123 [02:01:41] I got to sleep [02:01:47] night [02:01:50] jeremyb: yeah, I've seen pubsubhubbub [02:01:52] layla tov [02:02:10] 10x [02:02:59] btw it could be a puppet problem [02:03:11] perhaps we need to open the port on localhost [02:03:46] i don't follow [02:03:51] it's all on one box, right? [02:03:58] yep [02:04:52] we should consult with Ryan_Lane [02:05:04] i think i've got it for now [02:05:12] eh? [02:05:18] what's the issue? [02:05:31] can we have a firewall issue [02:05:45] between instances in the same project? [02:05:50] with a port on localhost [02:05:55] or between instances in different projects? [02:05:55] no [02:05:58] same machine [02:06:00] ok [02:06:02] there are no firewalls in labs [02:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [02:06:13] excluding the security groups [02:06:24] and that's between projects [02:06:28] ok [02:06:29] well, and the outside world [02:06:36] but it's not even hitting the switch [02:06:39] for loopback [02:06:49] but in-project all incoming traffic is allowed [02:07:09] or even the host. it's all within the guest [02:08:08] ok just cheking - Once upon a time I helped write an applicative fire wall that blocked evrything and it's mother [02:08:39] heh [02:08:56] ok I'll look at it 2morrow If you don't figure it out [02:10:28] did it do genealogical research? [02:10:33] was it mormon? [02:17:24] here we go... [02:17:25] Creating titlekey table...ok. [02:17:25] Populating titlekey table... [02:19:45] and it's hitting the search daemon [02:21:46] !log search search-test: added to LocalSettings.php: error_reporting( E_ALL );ini_set( 'display_errors', 1 );$wgShowExceptionDetails = true;$wgShowSQLErrors = true; [02:21:47] Logged the message, Master [02:22:16] !log search search-test: SQL error for missing titlekey DB. ran update.php and it showed up [02:22:17] Logged the message, Master [02:22:52] !log search search-test: MWSearch now seems to be working (or at least it's definitely being used) [02:22:53] Logged the message, Master [02:25:44] Ryan_Lane: i'm confused. we allow anon edits on labsconsole? [02:25:49] (see RC) [02:26:13] also, Labslogbot should have bot flag [02:26:32] hashar created it [02:26:43] created what? [02:26:50] for some reason mediawiki fucked up and thought the user's username was an IP [02:27:02] https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-000000db [02:27:07] that was the page created by the IP [02:27:19] yeah... [02:27:27] notice the instancecreator value [02:27:49] i do now... [02:31:29] this is either an OSM bug, or a MW bug [02:31:47] at first I thought it was an LdapAuthentication bug, but the session issue I was having was a MW bug, and that's been fixed [02:32:05] I thought that would also fix this bug, but apparently not [02:32:16] maybe I'm somehow messing up wgUser [02:33:07] the really funny part is that the instancecreator_username variable is set using wgUser [02:33:21] so I don't know how MW isn't getting the right value [02:33:59] damnit, i keep thinking that's openstreetmap [02:34:03] heh [02:34:47] aude: ^^^ [02:34:54] we can share the pain :) [02:35:30] Ryan_Lane: consider HOT officially on your project naming blacklist [02:35:41] what's HOT? [02:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [02:36:10] http://hot.openstreetmap.org/ [02:36:15] heh [02:39:54] ughhh wtf [02:40:01] ? [02:40:06] johnduhart@deployment-web:/usr/local/apache/common$ php bin/sql.php [02:40:06] PHP Warning: require_once(/usr/local/apache/common/wmf-config/DectectSite.php): failed to open stream: No such file or directory in /usr/local/apache/common/bin/sql.php on line 24 [02:40:06] PHP Fatal error: require_once(): Failed opening required '/usr/local/apache/common/wmf-config/DectectSite.php' (include_path='.:/usr/share/php:/usr/share/pear') in /usr/local/apache/common/bin/sql.php on line 24 [02:40:12] the file is there [02:40:22] The website wouldn't work without it [02:40:27] why are you running that directly? [02:40:42] ? [02:40:43] sql.php? that's a fairly normal thing to run [02:40:46] is it? [02:41:04] Ryan_Lane: It takes sql files and applies them [02:41:06] I thought there was a maintenance script for that [02:41:11] yes [02:41:15] that *is* the maint script [02:41:24] That's the script with a modification to make it work [02:41:30] ah. ok [02:41:40] deployment-test? [02:41:44] Yes [02:41:45] No [02:41:48] deployment-web [02:42:00] oh [02:42:10] well... [02:42:11] stat /usr/local/apache/common/wmf-config/DectectSite.php [02:42:11] stat: cannot stat `/usr/local/apache/common/wmf-config/DectectSite.php': No such file or directory [02:42:12] I'm supposed to run maintanence scripts from -test [02:42:15] huh [02:42:31] jeremyb: Fine here [02:42:42] on -web? [02:42:46] yes [02:42:55] that seems impossible... [02:43:19] johnduhart: oh, the problem is an extra c [02:43:20] Also fine on -test [02:43:29] AHHHH [02:43:43] * johnduhart throw computer out window [02:43:54] please don't [02:44:04] * aude *waves* [02:44:12] hey aude [02:44:29] aude: please register a complaint re the naming of OSM, kthx :) [02:45:01] :) [02:50:50] petan: OrenBochman: i still didn't read scrollback yet but yes, feel free to cc me on Search mails [02:50:53] night everyone. i'll be on less for the next couple days. be sure to say my name if there's something i should see :) [02:53:46] !log bastion test [02:53:46] Logged the message, Master [02:54:10] test failed [02:56:46] !log bastion test [02:56:47] Logged the message, Master [02:57:25] damn it [02:58:45] !log bastion test [02:58:46] Logged the message, Master [02:59:06] yay [03:00:30] no more broken SAL [03:01:06] yay [03:01:09] interwiki works [03:04:11] !log deployment-prep interwiki now works [03:04:11] Logged the message, Master [03:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [03:10:08] Think I've found our slowness issue [03:10:28] SELECT * FROM `wikiset` WHERE ws_id = '1' LIMIT 1 [03:10:35] At leaast 60 queries of that [03:13:07] !r 38929 [03:13:07] on is http://www.mediawiki.org/wiki/Special:Code/MediaWiki/38929 [03:14:08] wow. rabbitmq is fucking stupid [03:14:13] oh? [03:14:26] you can *never* change the rabbitmq servername [03:14:33] lols [03:14:48] that is the most full of fail thing ever [03:15:18] "The Erlang Mnesia database is host specific (because it is a distributed DB). The simplest way to get you fixed is to clear out the database dir. That's assuming that you can stand losing any persisted exchanges etc. The persisted messages are also stored in that folder, but not in Mnesia. [03:15:18] If you need to keep any persisted exchanges, queues, message etc. then that is a bit harder." [03:15:25] I only have like 3 rabbitmq servers, it's a bit of a pain to maintain. [03:15:43] haven't they ever heard of fucking host IDs? [03:15:55] this really makes me want to stab [03:16:15] lol [03:16:25] well, when I move this, I'll be making a virt-rabbit1.wikimedia.org alias [03:16:40] and pointing at that from now on [03:16:43] virtual rabbits! [03:16:58] or, better, virtqueue1 [03:18:43] * johnduhart slaps Reedy [03:18:48] https://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/CentralAuth/WikiSet.php?r1=65348&r2=71290 [03:26:22] !log deployment-prep svn up'd live [03:26:23] Logged the message, Master [03:30:54] Updating databases [03:31:34] !log deployment-prep Updated databases [03:31:34] Logged the message, Master [03:31:56] I love how there's an entry in all.dblist for arwiki but there's no database for it [03:32:43] !log deployment-prep Last update solves an issue where CentralAuth would make 70+ queries per page [03:32:44] Logged the message, Master [03:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [03:36:43] Ryan_Lane: what's the " GPS storage and retrieval" project about? [03:36:52] dunno [03:37:05] some mediawiki project a dev is working on [03:37:49] I believe it's adding some semantic support for that attribute in MW [03:38:03] so that paged with geocoordinates can be found in a programatic way [03:38:06] *pages [03:38:18] hmm... i know about wikidata but [03:38:25] this is a different project [03:38:34] i know they are doing coordinatesstuff with the mobile app [03:38:45] I think that's the main motivation for it [03:38:49] but ought to be working together? [03:38:50] I think yuvi is working on it [03:38:58] well, this is a short-term project [03:39:06] wikidata may never see the light of day for all we know [03:39:11] its goals are way more abitious [03:39:35] well, geocoordinates is one place to start on wikidata [03:39:41] *important* [03:40:20] they are looking at implementing SMW for wikidata, last I heard [03:40:43] geocoordinates support for mobile is planned to be that single thing [03:40:49] maybe wikidata can transition it later [03:40:52] nah, not sure SMW is the answer for wikidata and [03:41:02] other options are being considered [03:41:03] just repeating what I heard :) [03:41:16] s/heard/read in an email/ [03:41:28] i'll talk to yuvi maybe [03:41:52] I'm pretty sure they plan on having this done within months [03:42:04] wikidata isn't even going to be set up for months [03:42:48] I doubt they'll even have something demoable by the time mobile would have this live [03:43:10] yeah, but [03:43:32] wikidata and SMW are the eternal cookie lickers for anything semantic going in [03:44:10] some coordination might be nice, and having interwikilinks + geocooridnates more accessible via api and such are reasonable [03:44:41] objectives, in the shorter term [03:44:43] I have a feeling it'll likely just derail the mobile stuff. meh. not my project though :) [03:45:33] or duplicate efforts but i hope not [03:46:18] i'll hop on wikimedia-mobile and learn more [03:56:12] cool [03:56:56] looks like maxsem is working on it [04:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [04:17:26] Ryan_Lane: Still around? [04:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [04:37:27] !log deployment-prep unmounted deployment-nfs-memc:/mnt/export on /mnt from deployment-web [04:37:28] Logged the message, Master [05:00:35] !log deployment-prep Created nfs export /mnt/upload on deployment-nfs-memc [05:00:36] Logged the message, Master [05:00:48] !log deployment-prep Mounted that export onto -web [05:00:48] Logged the message, Master [05:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [05:11:32] !log deployment-prep Made a quick stab at upload config [05:11:33] Logged the message, Master [05:11:49] !log deployment-prep Adding apache config for upload.deployment.wmflabs.org [05:11:50] Logged the message, Master [05:30:39] !log deployment-prep Installed imagemagick on web [05:30:40] Logged the message, Master [05:32:12] !log deployment-prep thumbnails now working [05:32:13] Logged the message, Master [05:35:39] yep [05:35:42] ish [05:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [05:36:21] what's up? [05:47:37] Nevermind, gotit [05:47:50] Uploads! http://commons.wikimedia.deployment.wmflabs.org/wiki/File:Empire_State_Building_Top.jpg [05:48:00] * johnduhart is done for tonight [05:48:09] Try not to breal anything ;) [05:54:48] heh [06:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [06:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [07:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [07:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [08:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [08:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [09:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [09:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [10:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [10:15:37] can we ACK nagios in labs? [10:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [11:01:45] oh for fucks sake [11:01:53] petan: Permissions are broken again [11:03:23] petan: No I'm reverting you're last 2 changes, that's not how you do it [11:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [11:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [12:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [12:19:51] !log deployment-prep Fixed wmf-config permissions on -nfs-memc [12:19:52] Logged the message, Master [12:20:45] d-wx-wx-wx 16 petrb depops 4096 2012-01-09 19:59 live [12:20:50] what the fuck [12:21:30] depops? [12:21:37] Wait, -wx ? [12:21:50] chmod 333? [12:26:22] I know [12:26:34] wtf [12:26:52] RoanKattouw: And depops is just the group we use for deployment-*" stuff [12:27:28] !log deployment-prep Running updatedata (very slowly) [12:27:29] Logged the message, Master [12:28:22] 1.90 load on -sql, yay [12:32:18] Haha http://ganglia.wikimedia.org/ganglia/?c=Virtualization%20cluster%20pmtpa&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [12:32:27] There's yer problem! [12:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [12:38:36] yay central notice works [13:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [13:11:31] RoanKattouw: hehe http://en.wikipedia.deployment.wmflabs.org/wiki/Main_Page [13:11:36] (Central notice) [13:14:23] lolo [13:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [13:52:59] OrenBochman: hey, sorry I was afk for the night. ping me when you get this [14:06:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [14:08:14] PROBLEM Disk Space is now: CRITICAL on puppet-lucid puppet-lucid output: DISK CRITICAL - free space: / 16 MB (1% inode=35%): [14:13:14] PROBLEM Disk Space is now: WARNING on puppet-lucid puppet-lucid output: DISK WARNING - free space: / 48 MB (3% inode=35%): [14:36:04] PROBLEM host: puppet2 is DOWN address: puppet2 PING CRITICAL - Packet loss = 100% [14:43:21] Mmm pizza [14:43:26] OrenBochman: around? [14:43:30] * johnduhart-phone noms [14:43:39] johnduhart-phone: stop teasing us [14:44:01] Hehe [14:45:24] Have you seen the central notice? :p [14:45:49] link? [14:45:59] I just got here [14:46:20] One sec [14:47:03] none when I'm logged in... [14:47:16] http://en.wikipedia.deployment.wmflabs.org/wiki/Main_Page [14:47:26] er, not logged in [14:48:14]