[00:24:07] :-/ can anyone help with access to bastion.wmflabs.org ? I've uploaded my ssh public key, set up ssh-agent forwarding, but am still getting "Permission denied (publickey). [00:24:10] " [00:24:27] maybe I'm using the wrong username? ;_; unsure. [00:26:03] trying to login as grey@bastion.wmflabs.org or grェ neither seems to be working (though I'm pretty sure the UTF8 mixed mode one is probably not it, it's at least my username within the wikitech portion; shell access that was granted presumably is just grey) [00:32:38] grey-: let me take a look [00:33:37] eh [00:33:38] err [00:33:39] heh [00:33:47] your public key's name is funny [00:33:49] anyway.... [00:34:44] your key is accessible on bastion, and your username is grey [00:35:13] it just says public key failed [00:35:23] is your key in your agent? [00:35:25] ssh-add -l [00:36:29] *nod* should be let me double check; and yeah the name is funny - why do your instructions specify -t for email address? Better to use -b 4096 for better key strength. :) [00:36:54] blah, apparently my ssh-agent didn't retain what I told it. ;_; let me try again. [00:36:55] what instructions are these? [00:37:43] huh [00:37:44] ssh-add ~/.ssh/id_rsa.pub [00:37:44] Enter passphrase for /home/grェ/.ssh/id_rsa.pub: [00:37:44] grey@f10x86cur:/usr/ports/irc % ssh-add -l [00:37:44] The agent has no identities. [00:37:47] that's annoying. [00:37:53] :D [00:38:22] https://www.mediawiki.org/wiki/Git/Tutorial#Set_Up_SSH_Keys_in_Gerrit [00:38:24] ah. those [00:38:32] -C is any comment you want [00:38:36] it doesn't matter what you put there [00:38:42] as you've noticed ;) [00:38:50] *nod* :) [00:39:14] -t doesn't actually matter either [00:39:20] *nod* -b does though. :) [00:39:46] somewhat [00:39:50] hmm, why isn't ssh-agent taking this key? let me try without it maybe? [00:40:07] well, that worked. [00:40:13] * grey- stabs ssh-agent. [00:40:20] you don't necessarily need to use an agent [00:40:24] though it's nicer to do so [00:40:36] you'll need to use proxycommand if you don't [00:40:42] it can be... I don't trust it, but not many people get into dumping that process memory. [00:40:50] https://wikitech.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option [00:40:56] cool, thank you! :) [00:40:58] yw [00:41:12] one step closer to testing openstack + kvm. :) [00:41:17] heh [00:41:19] well… kind of [00:41:21] (or well, openstack, I've monkeyed with kvm elsewhere) [00:41:39] you'll need to use qemu in labs [00:41:47] since kvm in kvm doesn't work [00:41:58] heh. [00:42:11] you're going to think I'm insane... but I actually want to see if FreeBSD's bhyve will work in it. [00:42:19] :D [00:42:21] not sure though, it requires support for nested page tables. [00:42:23] I think you had mentioned this :) [00:42:34] I don't think it's crazy [00:42:35] *nod* then got distracted, now back with some free time on the weekend. [00:42:57] I prefer to go the easier route :) [00:43:03] kvm supposedly incorporated IBM's turtles project ELI stuff. [00:43:21] yeah... I'm a BSD die hard, it's rarely easier, but more satisfying personally in the long run. [00:43:43] anyway, thanks for your assistance, going to read up on this proxycommand stuff. [00:43:47] yw [00:44:12] I'm not even sure what instancename I'll be connecting to ultimately, but at least I'm able to connect to a bastion now, yay! [00:44:23] heh [00:44:29] well, we'll need to add you to a project [00:44:37] likely the openstack one, if you want to do openstack work [00:44:48] you're username is hard to type ;) [00:44:52] *your [00:45:05] *nod* most likely; albeit this all started as part of trying to learn more about openstack for an interview I had on Tuesday, but the interview went well and hopefully this work will transfer even if I don't get the job. [00:45:18] haha; yeah - I use a special character that's not meant to be easy to type. :) [00:45:35] that makes it harder for folks to add you to projects and such ;) [00:45:41] *sigh* sorry. [00:45:42] or do searches for your commits in gerrit [00:45:44] no worries [00:45:47] people will copy/paste [00:46:05] *nod* that should work. :) And it's more uniquely identifiable(or indemnifiable) than grey. :) [00:46:34] all of our tools should work with it. we've fixed any utf issues we know of [00:46:35] UTF8/unicode/CJKV is a ... side interest? [00:46:58] awesome. Mediawiki is one of the first projects to support UTF8 in URIs well that I've noticed. [00:47:05] heh. well, our software is probably some of the most translated around [00:47:14] for good reason! [00:47:55] vertically alligned right to left in-browser stuff is still pretty rarely implemented, but it helps to have a decent start with UTF8, and mediawiki is a better example than many projects in that regard. [00:54:32] hmm, though conceivably instead of ssh-agent forwarding or proxycommand config stuff, I could have a separate ssh key on the bastion? Is that inadvisable due to potential private key snooping? [08:34:35] Warning: There is 1 user waiting for shell: JdBAlten (waiting 0 minutes) [08:47:55] Warning: There is 1 user waiting for shell: JdBAlten (waiting 13 minutes) [09:01:20] Warning: There is 1 user waiting for shell: JdBAlten (waiting 26 minutes) [09:14:48] Warning: There is 1 user waiting for shell: JdBAlten (waiting 40 minutes) [09:28:08] Warning: There is 1 user waiting for shell: JdBAlten (waiting 53 minutes) [09:41:29] Warning: There is 1 user waiting for shell: JdBAlten (waiting 67 minutes) [09:54:50] Warning: There is 1 user waiting for shell: JdBAlten (waiting 80 minutes) [10:08:11] Warning: There is 1 user waiting for shell: JdBAlten (waiting 93 minutes) [10:21:35] Warning: There is 1 user waiting for shell: JdBAlten (waiting 107 minutes) [10:34:55] Warning: There is 1 user waiting for shell: JdBAlten (waiting 120 minutes) [10:47:36] !rq JdBAlten [10:47:36] https://wikitech.wikimedia.org/wiki/Shell_Request/JdBAlten?action=edit https://wikitech.wikimedia.org/wiki/User_talk:JdBAlten?action=edit§ion=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/JdBAlten [11:12:23] !ping [11:12:23] pong [12:23:44] petan, care to look an an interesting process on tools-login? pid=28708 [12:24:11] addshore: what is it doing? [12:24:30] its an instance of nano... [12:24:38] using 90+% cpu [12:24:47] o>o [12:24:50] o.o [12:25:17] :D [12:49:00] addshore hi [12:50:25] !log tools petrb: vul alias viewuserlang [12:50:27] Logged the message, Master [12:50:38] !log tools nvm previous line [12:50:39] Logged the message, Master [12:55:44] !log tools petrb: local-hawk-eye-bot /bin/nano /tmp/crontab.d4JhUj/crontab eat too much cpu [12:55:45] Logged the message, Master [12:56:55] meh [12:59:04] !log tools petrb: changed priority of nano process to 19 [12:59:06] Logged the message, Master [13:02:04] addshore idk if I should kill it or not :/ who knows what is he doing in it... [13:02:32] I don't like to kill unknown processes of someone else [13:05:43] Coren|Sleep we should probably set up some rules on tools project for running cpu expensive processes on -login? [13:06:06] I give that guy 30 minutes [13:40:42] !log tools petrb: killing that nano process seems to be some hang and unattached anyway [13:40:43] Logged the message, Master [14:25:37] heya. [14:25:42] Right call to kill the nano. [14:26:13] The TOS we're working on will have a clause about unreasonable use of resources et al. [14:28:46] 'only emacs and vim are allowed as editors' [14:32:59] No, people are welcome to inflict nano on themselves. :-) [14:33:19] Besides, if there was such a rule, I'm not sure I'd allow emacs. :-) [16:10:50] Coren, so you are saying replication setup is complete, but you're just making sure it's working? [16:12:54] Cyberpower678: No, I'm saying it's being built and I'm testing the bits that are working as we go. :-) [16:13:14] What bits are working? [16:14:03] Coren, ^ [16:14:59] All the internal stages; first instance replication, triggers to strip out PII, initial views into it. [16:15:24] What server? [16:15:35] Cyberpower678: hidden server ;p [16:19:24] addshore, tell me. I can at least setup adminstats. [16:23:39] I have no idea ;p [16:24:39] It's not accesible from the projects yet. [16:28:05] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 0 minutes) [16:29:15] damnit [16:29:38] Coren: when it is available, how will we access it from projects? [16:30:01] We'll set up DNS entries for the shards. [16:30:40] * addshore cant wait ;p [16:31:15] * Cyberpower678 can't wait. [16:31:41] You'll have to. :-P [16:31:59] I'll give it a week. ;p [16:32:35] so what will that entail? We will use a dns entry to point to the DB and the project will have its own credentials? [16:35:23] greenrosetta: The credentials will be in the project's .my.cnf (like it is now for the local DB); the DBs willl be accesible through something along the lines of enwiki.replica or whatever. We might actually try to do something close to the TS, even if only as aliases. [16:36:16] * Cyberpower678 is setup for the most part. [16:37:06] Coren: so will you use puppet to modify the existing .my.cnf in the projects home directory? [16:38:06] Not quite; there's a daemon that manages replica access that will create the .my.cnf if it is absent, or reuse the current one if it exists (and matches correctly) [16:38:13] Cyberpower678: did you ever look at my stub application? I spent a lot of time making it so the average coder could get a starter app running in a few minutes [16:38:26] ok, im gonna make a copy of mine now [16:38:30] no. [16:38:47] Cyberpower678: https://wikitech.wikimedia.org/wiki/Stub [16:39:19] greenrosetta, I don't do python. [16:39:37] But I might make one for PHP and Peachy. [16:39:59] i didnt do python either, but it really is a great little framework [16:40:32] It's too much resource hog. [16:40:35] Don't like it. [16:40:59] Besides I'm happy with Peachy and PHP. [16:41:13] Peachy makes writing a wiki bot a breeze. [16:41:39] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 13 minutes) [16:41:49] yeah, my thing isn't bots as much as a GUI tool [16:42:01] I like bots. [16:42:11] SVN? ewww [16:42:17] Adminstats will be under my care for a long time. [16:42:25] greenrosetta, what? [16:42:31] you use SVN for a repo [16:42:49] greenrosetta, no. That's old junk I haven't removed yer. [16:42:49] go DCVS my friend [16:42:51] ah [16:43:10] I don't have an active repo yet. I plan to use GitHub. [16:43:33] the two technologies that have changed my life in the past 10 years are Windows Home Server and DCVS [16:43:37] greenrosetta, why are you in my folder? :p [16:43:49] Cos I'm a fucking busy body [16:44:20] greenrosetta: dont [16:44:21] Maybe you can help me build a sockbot I'm thinking about [16:44:29] dont what? [16:44:53] 1 0 -1 0 1 0 -1 0 1 0 1 [16:44:58] go into others directories [16:45:08] its on-wiki [16:45:30] http://mw-peachy.googlecode.com/svn/trunk/ [16:45:46] Betacommand, I would have it locked, but then apache can't access it. [16:45:53] -.- [16:46:18] yeah, well when someone puts an External links section on a project page, it is there to be visited [16:46:39] greenrosetta, that Peachy is old and extinct. [16:46:43] ah [16:46:56] greenrosetta: I thought you where messing with his directories on the server [16:47:03] I have a newer one in the works that is currently approaching it's third alpha release. [16:47:37] Out of the gazillions to come. :p [16:48:00] The project's are still colloborative, but one should be considerate of the project creator [16:48:30] greenrosetta: you still shouldnt be in their directories without permission [16:49:01] * Cyberpower678 goes to Betacommand's directory. [16:49:04] Just kidding [16:49:05] * greenrosetta too [16:49:08] there are times where files may contain passwords or other non-public information [16:49:39] if a user forgets to set proper chmod it can cause issues [16:49:48] Betacommand, I have those sealed up in my folder. Only accessible to me and the group members. [16:50:24] Cyberpower678: its still not nice, and rude to snop without permission [16:50:27] I'm assuming wikitech is the same CC license [16:50:56] not that I disagree with the politeness factor [16:50:58] Betacommand, I know. I don't go looking at people's stuff. [16:51:31] Although, when I took over X!'s bot, I obtained his password to his account. [16:51:43] greenrosetta: actually that may not always be true [16:52:44] Putting nonpublic information in any project isnt a great idea [16:52:52] greenrosetta, Betacommand: My passwords are stored in the Configs folder of /data/project/cyberbot/Peachy [16:53:11] greenrosetta: sometimes it is required [16:53:12] but that is accessible to anyone that becomes peachy [16:53:24] my configs are stored in /data/project/addshore/getthefuckout ;p [16:54:06] wait, I mean /data/project/addshore-dev/get/the/fuck/out/getthefuckout.cfgyoushouldntbelookingat :D [16:54:08] addshore, that directory doesn't exist. :p [16:54:17] Cyberpower678: for you it doesnt ;p [16:54:33] :D [16:54:56] Cyberpower678: so did you change permssions on that folder so only you have access to it? [16:55:00] I invite anybody to access Cyberbot's folder directory stated above. [16:55:09] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 27 minutes) [16:55:13] greenrosetta, you tell me. [16:55:16] greenrosetta: any bot that uses pywikipedia has its login tokens stored on the server [16:55:26] they are also on git under https://github.com/addshore/addbot/tree/master/ihopenobodylookshere/allmystuffs.cfg [16:55:30] I guess, I can't see it [16:55:52] someone who wants to be evil could use them to hijack you account [16:55:55] what did you do so that people can't "become peachy" [16:56:06] exactly. It's got the GNU license which means it should be freely visible. [16:56:07] yeah, thats what I'm concerned about [16:56:08] greenrosetta: its a user rights issue [16:56:26] so how would I do the same thing on "my" project [16:56:40] greenrosetta: just use standard linux file permissions [16:56:49] greenrosetta: people cannot just become your project unless they are a member [16:57:12] isnt peachy a member of tools? [16:57:28] greenrosetta: peachy is a service group [16:57:30] its on tools [16:57:34] Cyberpower678: is a member of tools [16:57:35] greenrosetta, if you were a member of Cyberbot, you would have access to my config files and the ability to "become cyberbot" [16:57:47] Cyberpower678: is the only member of the peachy servicegroup [16:57:54] so can someone try and "become common-interests" and see what happens? [16:58:21] addshore@tools-login:~$ become common-interests [16:58:21] sudo: sorry, a password is required to run sudo [16:58:22] sudo: sorry, a password is required to run sudo [16:58:30] I plan to addshore to peachy soon. [16:58:42] *add addshore [16:58:47] Ill have more time in 2 days time :D [16:58:56] addshore: try again pls [16:59:09] sudo: sorry, a password is required to run sudo [16:59:18] addshore, that is once he starts using the framework and learns it inside out. [16:59:24] no, I added "addshore" to my service group [16:59:35] greenrosetta: I need to login and out of my user account [16:59:39] and then I would be able to access [16:59:45] ah, nvm.. thx [16:59:58] You can add me. [17:00:04] My interface is open. [17:00:19] added [17:00:21] yep greenrosetta now I can access it [17:01:22] greenrosetta, success. I am now common-interests. [17:01:50] so if I want to create a protected folder, what chmod permissions would I use so that only group members and the running process has access? [17:02:20] 770 [17:02:20] what is that 677? [17:02:31] or 2750 [17:02:38] 666 ;p Which is what I'm using. [17:03:24] since my app is running python under cgi, what process would need access? [17:04:18] If you own it, group. If common-interest owns it, owner. [17:05:00] does 2750 make the directory hidden from view? [17:05:18] no. It allows the group to set the group id [17:05:34] redundant somewhat,. [17:05:36] addshore: what do you use? [17:05:38] because using SCP, I can see the peachy directory but not the folder you mentioned [17:05:45] mhh *checks* [17:06:20] greenrosetta, are you in the peachy project folder, or cyberbot? [17:06:31] configs are 0771, my scripts are generally 0644 or 0664 [17:06:32] peachy [17:06:47] greenrosetta, you want cyberbot [17:07:05] I generally dont edit any files in the addshore or addbot projects, I push to git and it pulls automatically [17:07:20] ah, excellent [17:07:22] and then addshore-dev is my lovely mess :) which is actually rather tidy [17:08:03] greenrosetta, notice how you can access all the folders but Config? [17:08:12] yup [17:09:00] do all of the wikimedia databases use the same structure of wiki-en? [17:09:06] you would want to use chmod 750 as common-interests if common-interests owns the file. [17:09:21] other wise, you could simply set it yourself. [17:10:30] greenrosetta: A couple have extra tables because of extensions they have enwiki doesn't, but all the core is the same. [17:10:50] How many of the db's are going to be repliacated? [17:11:00] greenrosetta: all [17:11:06] * Coren recommends leaving sgid on the directories since that keeps group ownership of files consistent. [17:11:14] greenrosetta: All non-private wikis. [17:11:15] is there a list of them somewhere? [17:11:43] greenrosetta: Probably. :-) [17:12:08] While I'm waiting I might as well add an option to let users choose which wiki they want to use [17:12:46] greenrosetta: You're still not talking about that open SQL tool, are you? [17:12:50] no [17:12:56] my common-interests one [17:13:01] Good. >:-} [17:13:03] I'm not going public with the open sql [17:13:34] never planned too. If I do anything with it it is going to be for devs only, but I'd need to get authentication for that [17:47:28] Warning: There is 1 user waiting for shell: Dan Bolser (waiting 0 minutes) [18:09:48] Hm. So it turns out that the initial setup is long and labor intensive. We're not going to have all the databases ready for Amsterdam, but definitely dewiki, enwiki, commonswiki and wikidatawiki. The others will trickle in gradually after that. [18:13:15] Coren: I think that's probably the 80:20 rule in effect here. But why isn't this just a parallel process that's the same for all databases? [18:14:30] The process isn't automated, and still dependent on human oversight to make sure everything goes well. Those important ones were done first, then there's going to be some automation constructed for the smaller ones tested and debugged. [18:14:35] Warning: There is 1 user waiting for shell: CBM (waiting 0 minutes) [18:16:57] It might be nice as a tribute to the hackathon hosts to set up nlwp as well :-). Are the DBs alpha-public somewhere? [18:17:24] Not quite, the routing rules doesn't let them through. Probably tomorrow. [18:17:30] k [18:18:10] Plus, I need to set DNS up too, and some clean way to access them. The different shards sometimes lie on distinct ports; I'll probably set up a DNAT rule for them. [18:18:49] Coren: thanks for doing the shell access so fast for my new labs account [18:18:49] Heyas, CMB. I gave you shell, I expect you want tool labs too? [18:19:33] yeah, now I need a "project" I think - [18:19:37] carl-cbm: Wait for shell is almost always << 24h [18:20:16] You probably don't want a project; that's a whole cluster just for you. Moving tools, you almost always just want access to the tool labs project. [18:20:49] (There are a few exceptions, but that's for OSM-scope things) [18:20:53] that sounds good to me [18:21:14] You've been added to the tools project; if you have an SSH key pushed you're all set. [18:21:17] !toolsdocs [18:21:17] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [18:21:22] ^^ useful pointers. [18:21:26] Coren: Would be nice to replicate Toolserver's enwiki.$SUBDOMAIN.toolserver.org. Do you already have an equivalent of the toolserver.namespace and toolserver.wiki tables, or should I file a bug? [18:21:27] I'm around if you have questions. [18:21:43] thanks [18:22:16] scfc_de: File a bug; that's desirable indeed but I don't know if I'll have the time to set it up before Amserdam (though I might have the opportunity to do so /there/) [18:22:34] carl-cbm: If you can't get an p_enwp10 dump in time, you could always try to access it bit by bit, i. e. "SELECT * FROM table WHERE id >= 0 AND id < 1000;", "... >= 1000 and id < 2000;", etc. [18:22:43] Coren: k [18:24:02] carl-cbm: Or ask DaBPunkt, he's back. [18:26:11] scfc_de: yeah, but at least one of the tables ahs 60 million rows, which makes that more irritating [18:27:08] carl-cbm: Okay, then you should definitely ask DaBPunkt :-). [18:27:45] On the plus side, the replicas live on SSH so they are going to be nicely swift. [18:27:48] SSD* [18:28:40] I would wait with any judgment until hundreds of queries are running on them :-). [18:35:46] Coren: is 'mem' in the resource list the resource comparable to toolservers' virtual_free? [18:36:09] /quit [18:36:42] although it's a bit small for that [18:36:44] valhallasw: Comparable, but not quite the same: it is (a) a hard limit and (b) /total/ virtual memory; it generally needs to be quite a bit bigger as a consequence. (Strictly, it's the h_vmem gridengine complex) [18:37:44] Coren: I have apparently dsone something strange in setting up the tool account for 'veblenbot' [18:37:49] Coren: there's both 'mem' and 'vmem' in the qstat output. I'm basically trying to see what the relation between the two is for real-world processe [18:37:52] s [18:38:03] cbm@tools-login:~$ become veblenbot [18:38:07] sudo: unable to change directory to /data/project/local-veblenbot: No such file or directory [18:40:08] carl-cbm: Gah! That stupid bug still exists? About 1:5 times, it creates the tools with the wrong $HOME value. Andrew is still tracking it down, it seems. [18:40:23] Annoyingly enough, I can change everything /except/ the home in a LDAP user. :-) [18:40:32] I'll have to delete it, it'll just work after you create it. [18:40:38] I think it is because I typed 'local-veblenbot' when I created the tool account? [18:41:00] Oh! That might be it! [18:41:08] * Coren needs to check against that. [18:41:20] Try again? [18:41:26] I was not sure whether to type it or not, but all the other names had it [18:41:28] (WIthout the local-) [18:41:39] Yeay! You found the /why/ :-) [18:42:15] now it works, I was able to become the user [18:42:15] Apparently, it sets the home /before/ it checks for and strips local- :-) [18:42:33] that's what it looked like from here [18:48:33] Hm. Nice. I can select into the replicas. [18:48:38] * Coren is being a tease. [18:51:52] Coren, what is the ETA of total completion and what is the ETA of bare minimum completion and deployement? [18:53:46] That'd depend on how you define both milestones. :-) total completion is probably a week or two after Amsterdam, perhaps a bit more if you include polish and special-case views. If by "bare minimum" you mean "being able to make queries to any database" it's either during AMS or shortly after, with /most/ databases being available for AMS. [18:54:16] When is Amsterdam? [18:54:31] Like I said earlier, what I now can guarantee for AMS is enwiki, dewiki, commonswiki and wikidatawiki. [18:55:00] 24-26 May [18:55:49] Coren, any chance you can squeeze simplewiki in there? [18:56:12] Do you happen to know which shard it lives on off-hand? [18:56:37] Coren, define shard. Cluster server? [18:56:49] Yeah, s1-s7 [18:56:59] I can look it up, but if you know it's faster. :-) [18:57:14] Coren, S3 on toolserver [18:58:20] Then probably not quite in time, though with luck we'll have it around the last day. I know Asher is going to be hard at work adding the extra databases next week. [18:59:17] Coren, while it have a replag table like on toolserver? [19:00:47] Cyberpower678: That fits in the 'polish' category, that will come in a bit later. [19:01:23] Although the replag to the labs DB should be pretty much always the same to the normal cache slaves. [19:03:00] scfc_de: nlwiki is on s2, which means it's almost certainly going to be ready in time BTW. :-) [19:05:23] Coren: Nice :-). Re replag, it would be perfect if we had a proper SECURITY DEFINER function that queried the actual MySQL replag, and not the latest revision in recentchanges. [19:08:28] Coren, when going into a database like enwiki_p there's a table called ts_replag and using SELECT ts_replag FROM ts_replag; will return the replag in seconds. [19:09:14] That's on the 'nice to have' list. Not on my roadmap for the coming week or so at least. :-) [19:09:57] It's not only nice but useful to helping to identify why the edit counter isn't working right. [19:10:09] But ok. [19:11:16] I'd be quite happy with just select on enwiki right now :P [19:12:03] Damianz: AOL. [19:15:32] * greenrosetta agrees with Damianz [19:32:22] Coren: should the '-m b' option to qsub work from labs - to send email? [19:33:34] carl-cbm: It should, though you'll get crappy local mail unless you set a .forward up for yourself or the tool. (The tool, by default, explodes to its maintainers. Maintainers, by default, get local mail) [19:35:15] Hmm. I will look again. On a more minor note, on the login server I see files that have dates in the future, which probably means one of the qsub execution nodes has clock sync issues [19:36:10] or maybe I am seeing things. Let me play with it some more and I will let you know if I still have any problems [19:36:13] They should all be ntp'ed up. If not, it's a bug. [19:37:08] Yeah, all synced up: *linne.wikimedia 75.144.70.35 3 u 489 1024 377 0.486 3.450 0.944 [19:44:00] mysql:dbmanager@not_telling [enwiki_p]> select rev_id, rev_page, rev_user, rev_user_text from revision order by rev_id desc limit 5; [19:44:00] [...] [19:44:00] | 555833230 | 37089210 | 1377980 | Zastavafan76 | [19:44:00] | 555833229 | 963591 | 0 | 77.70.28.120 | [19:44:00] | 555833228 | 39332819 | 16603424 | Misschrisparker | [19:44:00] | 555833227 | 22905469 | 7024006 | George Sharma | [19:44:00] | 555833226 | 39429384 | 12864794 | Lester Foster | [19:44:50] Coren: Now you're just showing off :-). [19:45:35] It's a sneak preview. [19:46:11] * Damianz eats Coren [19:46:13] On Toolserver: "SELECT * FROM revision WHERE rev_id = 555833230;": "Empty set (0.00 sec)". It's torture :-). [19:46:35] toolserver is laggy at best [19:49:00] Coren: BTW, on Toolserver I recently managed to overflow the database server's /tmp/mysql-something with a subquery. Maybe something to guard against if disk space is scarce (which I doubt). [19:49:43] It's not, but the temporary table space is distinct. [19:49:51] k [20:02:36] Coren: I managed to set everything up, and it seems to be running correctly. The only issue remaining is getting emails from qsub. If I explicitly set the email address with -M, I get an email, but if I do not, I don't get one [20:05:19] carl-cbm: Don't you have local mail at tools-login? [20:06:06] no, I looked in /data/project/.system/store/mail [20:07:20] I think the permissions of your ~/.forward need to be tighter. "chmod g-w ~/.forward". [20:12:19] carl-cbm: scfc is probably correct, exim tends to default to paranoid. [20:13:36] I deleted the .forward after he said that, and tried a few more times. NO email. [20:14:12] sorry for the caps there. what would be my local delivery address if I want to try setting -M to that? [20:15:07] Coren: What's the queue length ATM? [20:16:22] AFAICT, all the queues are emtyp. carl-cbm, lemme go check the logs. [20:18:56] scfc_de: Aha. I see what's wrong; qsub is trying to be too smart and not using the fqdn for the default mail. [20:20:12] It's even downright daft: It's trying to send mail to cbm@tools-login -- that's doubly wrong. [20:20:35] I tried that address once, so blame it on me [20:20:38] Although, that means you're using qsub from your own account instead of from the tool's? [20:21:01] carl-cbm: Try this: -M cbm [20:21:03] No domain. [20:21:10] no, from the tool's account. If I use -M and a gmail address, instant feedback. If I use no -M, nothing. I tried 'cbm@tools-login', which should work, but it didn't [20:22:04] No, cbm@tools-login is what it also tries by default, but that can't work. What works is 'cbm' or 'cmb@tools.wmflabs.org' (but I don't recommend the latter, since we might change the domain) [20:22:06] -M cbm does work [20:22:19]