[00:02:09] Ive got a process that I start via grid and twistd but it doesnt appear on the status page [00:04:15] The problem with a network of computery people is that everyone will have their own hackaround something :p [00:04:40] Betacommand: does it show up in qstat from the command line? [00:05:08] anomie|away: nope [00:05:43] did labs get restarted recetnyl? [00:05:50] *recently [00:06:01] Coren|AFK: andrewbogott_afk thoughts on my security group question? [00:06:24] rschen7754: it was a manual start after the NFS failure [00:06:27] nevermind works now [00:06:33] also woohooooo! :D [00:06:40] ok... so stuff should be coming back shortly? [00:07:02] rschen7754: everything should be back up [00:07:19] the failure was over an hour ago [00:07:22] crud, so if it didn't come back i've gotta manually restart [00:07:57] * a930913 hands rschen7754 duct tape, WD-40 and a hammer. [00:11:54] Betacommand: Where is it running then? [00:12:08] anomie|away: I dont know [00:12:41] but I see it on IRC so I know that it is running [00:13:20] anomie|away: thus why Im scratching my head [00:14:02] Betacommand: Hmm. What would be the process name? [00:14:11] Betacommand: You have a bot that's running but that you don't see on the status? [00:14:20] Coren: correct [00:14:29] I started it with the name "logger" [00:14:41] What's the actual executable? [00:15:25] Coren: see OM [00:15:29] *PM [00:16:37] Coren: any ideas? [00:16:44] I'm looking. [00:18:58] Betacommand: I actually see two(!) instances running on the grid, neither under the aegis of the sge_sheperd. Does this thing fork()? [00:19:43] it runs as a daemon [00:20:23] Ah! That's why. Don't daemonize on the grid; that confuses things completely -- sge needs to remain the parent process otherwise it will lose track, [00:20:37] The parent returned, so the job ends. [00:20:41] Coren: Im not sure how to recode it [00:20:57] * Betacommand goes to take a look [00:21:02] It should be simple. Lemme look at the source real quick. [00:21:14] * Coren kills the orphans. [00:21:57] Coren: let me try something [00:22:10] Betacommand: Since you use twistd, just '-n' works. [00:22:25] (as argument to twistd) [00:22:27] ah thanks [00:23:06] restarted, and /me looks at grid [00:23:31] I'm not seeing it at all. [00:24:12] Can you point me at your startup script? I'll take a look if you want. [00:24:45] Coren: I goofed [00:27:36] Coren: thanks looks fixed [00:28:22] gotta love ghost scripts on grid [00:30:52] Betacommand: I need to find some way to prevent that, actually. [00:31:45] Coren: had me scratching my head for a bit [00:32:18] It's a ghost in the shell. :-) [00:32:35] Coren: those bugs scare the shit out of me [00:33:32] Coren: is it possible to get the .err and .out sent in the cron email? [00:34:19] No, because those are asynchronous and normally get generated quite a bit after crone is done. [00:34:41] dam [00:35:00] I loose quite a bit of functionality that way [00:35:32] I used to filter the cron emails based on error messages [00:36:07] non-error emails got deleted and the error emails sent to my inbox with a flag [00:37:45] Coren: have you been able to look at https://bugzilla.wikimedia.org/show_bug.cgi?id=54054 [01:03:02] Coren: are the servers time set to UTC? [01:20:11] Betacommand: It appears they are. [03:03:28] Betacommand: You can set a destination directory with -e and -o [03:03:36] Betacommand: Also yes, they are UTC [03:03:44] Change on 12mediawiki a page OAuth was modified, changed by RobLa-WMF link https://www.mediawiki.org/w/index.php?diff=830908 edit summary: Paring way back to disambiguation page - I almost made this just link to [[Extension:OAuth]] [03:04:12] Coren: is there a way to set that globally? [03:05:45] Hm. Well, I /could/ have jsub & friends parse a dotfile for some defaults. It's a funky support issue though. Perhaps with a "if you break your things you get to fix them" disclaimer of some description. [03:09:33] Coren, I have a rather peculiar question for you. [03:10:00] What's up? [03:10:12] Coren, what happened to jsub? [03:10:43] "happened to"? [03:11:27] Well. What happened is, sometime 22:30 UTC today, according to Ganglia, all of the active jobs on my bot got wiped. [03:12:19] Coren, ^ [03:12:38] Cyberpower678: nfs falure [03:12:50] Oh, poops. I forgot to write a postmortem for labs-l. [03:12:53] * Coren goes do that now. [03:13:07] An NFS failure never did that before. :/ [03:14:40] While Coren does that, I will add all of my continuous tasks to a crontab so they stay running. [03:15:49] Cyberpower678: It wasn't NFS itself that failed, but the underlying filesystem. [03:16:09] Coren, oh. [03:29:33] * Cyberpower678 added 14 new entries to the crontab. All of them fire at midnight. I think Coren should be informed. :p [03:34:56] Cyberpower678: have you thought about offsetting them? or staggering them? [03:35:12] ? [03:35:21] Clarify. [03:35:38] Cyberpower678: midnight is the #1 cron time [03:36:04] It's all running on my queue. So no worries. [03:37:04] Besides it's nothing urgent. It's just resubmitting a continuous task to jsub in the event it somehow terminated. [03:37:47] Switching OS [13:37:10] Coren: remember when i maxed out max_user_connections a few days ago? [13:37:22] local-render@tools-login:~$ sql enwiki_p 'show variables like "max_user_connections"' [13:37:22] Variable_name Value [13:37:22] max_user_connections 512 [13:37:27] local-render@tools-login:~$ sql dewiki_p 'show variables like "max_user_connections"' [13:37:27] Variable_name Value [13:37:27] max_user_connections 10 [13:37:32] local-render@tools-login:~$ sql eswiki_p 'show variables like "max_user_connections"' [13:37:32] Variable_name Value [13:37:32] max_user_connections 10 [13:37:43] these should all be 512. [13:38:03] something must have reset them to the default value, 10, but only for some wikis [13:48:43] AzaToth I need u [13:49:09] AzaToth: https://github.com/huggle/huggle3-qt-lx/tree/debian this thing doesn't want to build because fakeroot wants to execute make install [13:49:21] is it even possible to use fakeroot for that? [13:56:06] petan: looking [13:56:51] fakeroot just fakes root, it doesn't give you root priviledge, nor does it make a chroot [13:57:27] so a make install fails usually unless a PREFIX is set to a dir the _user_ has access to [13:57:31] write access [13:58:37] AzaToth: I know what fakeroot is, I need to know if it's possible to run that deb maker using fakeroot? [13:58:48] aha [13:59:03] another question, I ran it as root, it made a .deb package, but it seems to be defunct [13:59:23] when I do dpkg -i package it install it, but nothing happens, how do I debug what dpkg actually did (all files it created etc) [14:00:03] after you've built it, run "debc" to see it's content [14:00:32] debc huggle_3.0.0.0_amd64.deb [14:00:33] debc: cannot find readable debian/changelog anywhere! [14:00:34] Are you in the source code tree? [14:00:44] how should I run debc [14:01:02] run it on the changes file [14:01:09] eh [14:01:23] so I switch to ./debian directory and run debc changes [14:01:24] ? [14:01:43] usually a plain "debc" in the root directory runs it on the latest changes file built [14:01:56] petanb@petrbena:/tmp/huggle/huggle3-qt-lx-debian/huggle-3.0.0.0/debian$ debc changelog [14:01:57] debc: no appropriate .debs found in the changes file huggle_3.0.0.0_amd64.changes! [14:02:12] in the root dir, not in the debian dir [14:02:14] what is "root directory" [14:02:22] is that a dir where .deb are? [14:02:31] or directory from which I run deb maker [14:02:33] no, the dir above the debian dir [14:02:35] aha ok [14:02:58] Coren: around? [14:03:03] also I highly recommend against placing the source code in a subdir in the git repo like you've done here [14:03:16] AzaToth: HA it installed lot of crap in /usr/share/doc but no binaries :O [14:03:23] hehe [14:03:37] huh, what do you mean, regarding the git repo [14:03:43] how should I do it? [14:04:03] I am following that idiot-guide here https://wiki.debian.org/IntroDebianPackaging it tells nothing about git [14:04:10] matanya: Barely. I'm still pre-first-coffee. [14:04:20] the stuff in "huggle" dir should be in the top dir [14:04:33] i.e. "mv huggle/* ./ [14:04:39] hmmmmm [14:04:46] that will make it very hard to merge master to it [14:04:48] then you can do "git buildpackage" directly [14:05:04] Coren: after you convert coffee to awkeness, please ping me regarding some puppet labs code [14:05:24] petan: I'm just saying ツ [14:05:42] I will probably make a shell script that will do this automagically and call the commands to build packages, because I will never remember them anyway [14:06:10] all you need it "git buildpackage" ツ [14:06:16] ok so why in the world it doesn't include the binaries :(( [14:06:25] making shell scripts to do it is the wrong way [14:06:36] but there is no better way I know of [14:06:42] petan: don't use that for stuff in git [14:06:51] matanya: what should I use then [14:07:12] I don't accept hints like "don't do this" I only accept hints like "do this instead of that" [14:07:41] petan: i type useful comments slower than alerts [14:07:57] well, it does build, so it's probably it doesn't know what to install [14:08:15] it executed make install and installed huggle localy during the build process [14:08:37] yea, but you have to inform dpkg what to actually install [14:08:44] petan: following http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp is a goo idea in generall [14:08:50] hm I think I will need to find some other guide then because this one doesn't mention that I have to do that [14:09:18] Coren, everytime I try to read /data/project on labs, it hangs up. But all other directories load fine. What could be causing that? [14:09:18] petan: do you have debian/rules file? [14:09:23] matanya: are you talking about not using a shell script to move stuff from ./huggle to ./ ? [14:09:26] matanya: yes [14:09:38] yes petan i'm talking about that [14:09:47] matanya: I mean what was " petan: don't use that for stuff in git" regarding to [14:09:52] what are your rules? do you have an install rule? [14:10:05] yes, petan that exactly [14:10:07] matanya: https://github.com/huggle/huggle3-qt-lx/tree/debian [14:10:21] petan: the issue here is that you are using a non-standard configure and it doesn't listen to prefix et al [14:10:27] matanya: ok so how in the world am I supposed to do that if not using shell script? I can't just move the files physically and push it [14:10:55] AzaToth: there is a standard to configure? I never heard about it! [14:11:03] I need to find it then... [14:11:36] petan: git-buildpackage can help you with that [14:11:46] matanya: it can move it? [14:13:04] not sure what you mean by "it" [14:13:20] the folder [14:13:36] folder in incorrect location to correct location [14:13:43] ./huggle/* to ./ [14:13:50] because I can't do that in git [14:14:00] and you say that using a shell script is wrong way [14:14:15] petan: yes, you can specify a build dir: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.building.html#GBP.BUILDING.EXPORT [14:14:22] aha [14:15:03] petan: is that what you were looking for, i hope it helps you [14:15:23] petan: remove configure and Makefile [14:16:12] AzaToth: there is no Makefile, it's created by configure [14:16:18] debhelper has allready procedures for qmake [14:16:24] ok, then remove configure [14:16:35] but that configure script doesn't just execute qmake lol [14:16:44] it does ton of other stuff which needs to happen before you build [14:17:22] I need to execute some kind of configure before starting make [14:21:15] Cyberpower678: Where are you hitting that problem? [14:22:16] Only when I try to view the directory /data/project/ [14:22:30] It takes about 2 minutes to load the directory [14:23:49] Coren: https://bugzilla.wikimedia.org/show_bug.cgi?id=57617 für the thing we talked about yesterday. Is it ok this way? [14:24:11] krd: Yes it is. [14:24:15] Thx. [14:24:29] krd: It's a fairly simple request, shouldn't take long for legal to okay it. [14:30:59] Change on 12mediawiki a page OAuth/en-gb was created, changed by Shirayuki link https://www.mediawiki.org/w/index.php?title=OAuth%2fen-gb edit summary: authorisation ※en-gb [14:31:17] Change on 12mediawiki a page OAuth/en-gb was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=831418 edit summary: authorisation ※en-gb [14:31:35] Change on 12mediawiki a page OAuth/en-gb was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=831420 edit summary: authorisation ※en-gb [14:32:09] Change on 12mediawiki a page OAuth/en-gb was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=831422 edit summary: authorise ※en-gb [14:32:33] Change on 12mediawiki a page OAuth/en-gb was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=831424 edit summary: authorisation ※en-gb [14:38:29] AzaToth: is there a way to mass-build package to multiple architectures? [14:39:01] so that it build like x86, amd64, etc debs? [14:40:06] Cyberpower678: Where exactly? I have no such issue from tools-login, at least (well, a ls -l is slow but that's expected over a large network directory given the number of stat() calls and the sort) [14:43:39] real 0m1.558s [14:43:39] user 0m0.012s [14:43:39] sys 0m0.052s [14:45:00] Coren, so calling a directory with a lot of project folders is slow? [14:46:57] Well, ls has never been very fast given the number of stat() calls it makes. [14:56:08] It /is/ slower than I would hope right now though. I think I'll have to move the filesystem off XFS entirely; that one is probably ill. [14:56:27] But I'm going to plan the outage long in advance. [14:56:45] Coren, glad I could help. [15:10:19] hey there. Any clue how I could get my new user jenkins-slave granted the permission "loginviashell" ? [15:10:47] hashar: Has to be done through wikitech Special:Userrights [15:11:05] hashar: The 'shell' user group has that permission [15:11:06] ahh so that is a normal mediawiki perm, nice :-] [15:11:15] yeah can't access that page :( [15:11:36] hashar: That's because you aren't cool enough. :-) [15:11:37] would you be able to add the permission to the jenkins-slave user I created ? Or do you have any idea where I could request such perm? [15:12:13] hashar: {{done}} [15:12:20] * YuviPanda waves at Coren [15:12:25] think you can do portgrabber today? [15:12:36] * Cyberpower678 sees YuviPanda waving at Coren  [15:12:47] YuviPanda: Unless I have another filesystem explode in my face, yes. [15:12:53] hah! [15:13:00] * YuviPanda douses all the filesystems with water [15:13:29] Coren: you are the most awesome labs operator :-] [15:13:32] Coren: do you know what the timeout is for labsdb queries? [15:13:47] or if there even is one? [15:13:54] no need to investigate if you don't know it off the top of your head [15:14:23] We didn't put one in atm, quite on purpose (some of the reports those DBs are intended for are *long*) [15:14:37] Coren: ah, so long running queries are okay. [15:15:06] I'm working with halfak to build something SQLy to help casual researchers, so just wanted to know [15:15:07] YuviPanda: Yes, though we request they be marked SLOW_OK nonetheless [15:15:18] Coren: right [15:15:35] I'm just going to make them timeout at 10m or so to start with, and then see where we go from there [15:17:06] Slave successfully connected and online [15:17:12] Coren: thank you a ton, huuugee timesaver [15:18:50] !log integration created [[User:jenkins-slave]] for the integration project. Credentials in fenari.wikimedia.org/home/wikipedia/doc/labs-jenkins [15:18:52] Logged the message, Master [15:31:00] wow AzaToth the dependecies are correct - HOW is that even possible O.O [15:31:14] it detected the dependencies from binary or what? [15:31:21] how could it do that [15:32:25] AzaToth: and why it doesn't produce -src package :/ [16:41:33] hey Coren [16:41:48] I'm doing a small project that might involve a bit of load on labsdb, so wanted to run it by you [16:41:54] poke me when you have a few minutes to spare [16:41:55] ty [17:05:39] * YuviPanda gently pokes Coren again [17:06:08] Yeah, I'm around-ish but eating lunch. Talk, I'll keep an eye on. :-0 [17:06:48] Coren: ah, works for me :D [17:07:13] Coren: so, I'm conspiring with halfak to build a tool that lets users query labsdb without having to sign up [17:08:19] Coren: it's meant to be a gateway drug for signing up for labs access [17:08:58] Coren: since for a lot of research oriented folks, managing keys and such doesn't come naturally [17:10:22] Coren: so only question is, will ops kill me if I do that? [17:11:09] YuviPanda: cool stuff! [17:11:21] saper: :D It's insanely cool and I'm super excited [17:12:51] The dewiki_p at labs is stuck since last night? [17:12:58] The idea is sound, but you need to protect it against it being abused. At the very least a strong captcha and ideally some sort of signup; even better would be to restrict it to people with a Wikitech account so they can be blocked. [17:13:00] as far as I know you stage two release will be super cool mobile app for everyone to play with the data :) [17:13:15] Coren: ah, so the idea is to not do that :D [17:13:18] krd: It's being rebuilt now; there were issues during the rebuild but I know Sean is working on it. [17:13:20] saper: what kinda abuse are you thinking of? [17:13:24] err [17:13:25] Coren: [17:13:27] Ok [17:13:29] Coren: queries that run for a week? [17:13:38] YuviPanda: No. 50k queries. [17:13:58] Coren: how about if I rate limit them? [17:14:02] You're suggesting making an easy web interface to DDOS the database. :-) [17:14:29] what about an easy SQL interface to DDoS... [17:14:54] Coren: with the idea being that if there is a DDoS, it takes down my interface machine, and not the database [17:15:01] Unless you rate limit them to some interval longer than your longest timeout, it's still trivial to bring the databases to their knees. Any labs user can do that but then they're a labs user and can be kicked off. [17:15:40] It's still easier than a tools account (no need to ssh, or to manage keys) [17:16:45] So no. "Do SQL queries without some sort of account needing a human to rubberstamb" won't fly. [17:17:02] halfak: ^ [17:17:06] :'( [17:17:09] Although "Easier and requires less technical skill than tools" is cool. [17:17:31] that complicates things a lot [17:18:18] Coren: I think a rate limit could be reasonable. [17:18:36] we could rate limit it aggressively [17:18:36] Sorry, Yuvi, but labsdb will never be as robust as our massively cached and distributed prod stuff, and even /that/ can be given problems by a flood of bad queries (to wit the issues caused by search indexing gone awry) [17:18:41] Are you thinking that you'd limit users to 1 your per few minutes? [17:19:24] *your = query [17:19:27] Coren: I can also rewrite the SQL - for example, no resultsets over 1000 rows (or some other number of rows) [17:19:41] and also rate limit it aggressively [17:19:44] I could still hit the DB pretty hard with a one row query [17:19:46] YuviPanda: you would have rendered wikidata obsolete! ;) [17:19:48] and also iplimit number of concurrent connections [17:19:55] Or hit it very softly with a many-row query. [17:19:56] YuviPanda, halfak: I'd be okay with a publicly accessible interface iff the requests were queued to execute only one at a time /and/ there was both a timeout and a rate limit. [17:20:06] Coren: I can do a timeout *and* a rate limit [17:20:22] YuviPanda: Can you do a queue. I don't think a queue is a bad idea. [17:20:57] YuviPanda: maybe SQL rewrite w/EXPLAIN testing would be ultracool [17:21:16] If you do a queue, execute pending queries one at a time, rate limit them very conservatively, we can talk. [17:21:46] Coren: I'm okay with that [17:22:08] Coren: globally one at a time or one user one at a time? [17:22:18] And I want a BRS [17:22:40] BRS? [17:22:47] https://en.wikipedia.org/wiki/Big_red_button [17:22:58] Coren: ah, a kill switch is built in, yes [17:23:45] Coren: so, let's see. Aggressive Rate limiting, and no concurrent queries from same user - queries from one user are serialized [17:24:54] Coren: is that good enough? [17:25:03] Nonono. You missed my point. Not "no concurent queries from the same user". "No concurrent queries full stop." :-) [17:25:31] Coren: ah, hmm [17:25:50] Coren: how about that number is a small number like '8' [17:25:57] or a tuneable number [17:25:58] Because since you don't have accounts, "same user" is meaningless. [17:26:04] Coren: same IP? [17:26:26] Q.F. the first "D" of DDOS. :-) [17:28:02] Coren: right. [17:28:19] You're still allowing arbitrarily hard queries, even a half dozen of those could be detrimental if we have no control. [17:28:28] Coren: right. [17:29:25] And the result set size is not a proper limiting factor; I can make single row queries that'll make the database cry without having to think hard about it. :-) [17:29:59] Coren: well, we can agressively timeout the. All queries get killed after some-amount-of-time [17:30:52] YuviPanda: I can float the idea on ops-l if you want, but I'm pretty sure everyone will agree with me that you either need wikitech accounts or a limit of a single running query. [17:31:18] YuviPanda: Although some will probably say that it's a horrible idea anyways. :-) [17:32:05] Coren: hmm, wikitech account doesn't sound so bad anymore ;) [17:33:11] Coren: do you know that I was planning on doing this by exposing a limited service that actually talks the mysql protocol? [17:33:12] Also make sure you log the queries (but I'm sure that was planned) [17:33:35] YuviPanda: That seems to be the most... strange way of going about it. [17:33:41] Coren: why so? [17:33:56] Coren: it'll not check auth, and people can just mysql -h something and just start right away [17:34:05] YuviPanda: Wouldn't it have been much easier to just provide a http base API that eats a query and spews out results? [17:34:16] Coren: and if I can put all the DDoS protection in that proxy, then I can build simpler tools elsewhere [17:34:37] Coren: ah, well - the mysql protocol speaking controllable thing is... already written :P [17:34:50] Coren: so I just have to adapt it for use, vs a web thing where I have to fully build it [17:34:54] Heh. Whatever floats your boat. :-) [17:35:02] select repeat ('Hi YuviPanda',1024*1024*1024*1024... [17:35:32] Coren: hmm, so if I implement a form of making sure that only people with wikitech accounts can access it, the mysql query interface is okay too? [17:35:40] Coren: and of course, we can revoke anyone's access at any time [17:36:04] dafuq? [17:36:55] Coren: how does that sound? 1. Limited to people who have wikitech accounts (banning them or revoking a right will deny them access) 2. MySQL protocol [17:37:01] Coren, where did all of these databases come from? [17:37:43] Coren: so it'll have similar access controls as toollabs, just without the sshing [17:38:25] Coren, http://pastebin.com/iy7Jyd1W there should only be 2 at most. [17:39:41] Cyberpower678: people can create databases on the replica databases [17:39:45] YuviPanda: I like the idea of requiring a WikiTech account. It doesn't seem like much to ask people for -- assuming that it isn't too much trouble to require that. [17:39:59] halfak: it complicates things a fair bit, but not impossible [17:40:05] YuviPanda, on MY account? [17:40:20] Cyberpower678: what do you mean 'your account' [17:40:26] this is being accessed from xtools. [17:40:31] tools-db [17:40:49] halfak: we can make them register and be given a token, and asked to pass that token along [17:41:05] Sounds great. :) [17:41:25] halfak: I still want an explicit ok from coren for 'Coren: how does that sound? 1. Limited to people who have wikitech accounts (banning them or revoking a right will deny them access) 2. MySQL protocol ' [17:42:23] halfak: since that assumes that we can relax rate limit / timeout requirements if we require an account [17:42:44] Yes. This would be more optimal. [17:43:31] Though, I still like the idea of query Web UI. It would be fine to just limit that strongly. [17:43:42] And require people to take an account. [17:43:45] *make [17:45:52] halfak: true. with the added requirement that might end up being about the same amount of work anyway [17:46:30] halfak: Coren okay, to rephrase: 1. If we require people have a wikitech account, 2. Is it okay to give them the same level of access that toollabs does? [17:46:40] no matter the protocol (mysql or http) [17:55:35] YuviPanda: Yes. [17:56:01] Coren: sweet! and it doesn't matter if I expose it as mysql or http? [17:56:02] Hm. [17:56:12] Coren: also if we require accounts, no need to aggressively rate limit? [17:56:13] Well, you'd also need to check that they are not blocked. [17:56:19] Coren: oh yeah, of course [17:56:27] Coren: I'll have an auth step that'll query the API [17:56:37] Still rate limit, though not agressively. To guard against errors though. [17:56:47] Coren: sure, sure. [17:56:56] * Coren has a meeting starting. [17:57:03] Coren: alright. Thanks! [18:21:40] andrewbogott: the API separation patch didn't blow anything right? :) [18:30:42] yuvipanda: worked fine as far as I can tell :) [18:31:45] andrewbogott: good enough :) [18:33:05] valhallasw: ping [18:33:14] valhallasw: do you know where the docs for OAuth is? [18:33:22] The ohes I found seem... not too useful [18:33:44] YuviPanda: I sort of hacked it together [18:33:55] valhallasw: no docs on the MW side? [18:34:20] YuviPanda: some, but the semi-official ones actually use some json protocol that's not actually oauth [18:34:36] YuviPanda: flask-mwoauth might be the easiest to read, actuall [18:34:48] https://github.com/valhallasw/flask-mwoauth/blob/master/flask_mwoauth/__init__.py [18:34:55] whoo nice [18:35:18] check the constructor for the URLS [18:35:28] and extra parameters to pass... [18:36:10] if you were building a flask app: you can use MWOAuth.request() to do API requests [18:36:55] valhallasw: I'm in a slightly weird situation. I need to have flask first authorize and get a token, and then show the token to the user [18:37:12] valhallasw: and then they'll use that token in a command line app, which will then pass it to a non-http service running Lua which verifies it [18:37:15] YuviPanda: what use is that? you always need the token + secret [18:37:27] valhallasw: they share the secret? they're two parts of the same tool [18:37:31] ahhh [18:38:44] I don't really get how it's supposed to work, but OK [18:38:52] valhallasw: so my usecase is like this [18:39:03] the user gets redirected to oauth_authorized after authorization [18:39:18] then you can just echo resp['oauth_token'] to the user [18:39:23] yeah [18:39:29] that's the plan [18:39:32] and resp['oauth_token_secret'], probably? [18:39:49] valhallasw: probably, concatenated as a single thing? [18:40:49] valhallasw: my end use case is that you should be able to do 'mysql -h -u -p enwiki_p' [18:40:53] valhallasw: and it'll just let them in :D [18:42:09] valhallasw: so I need to do the verification for username and token from inside an interceptor for the mysql protocol, which will be in Lua [18:42:16] err [18:42:19] wikitech doesn't have oauth [18:42:22] what [18:42:26] why [18:42:42] we can deploy it, I guess? [18:42:47] I suppose [18:42:53] Ryan_Lane: any reason wikitech doesn't have oauth? [18:42:58] andrewbogott: anomie [18:43:01] ^ [18:43:05] not really. we should probably add it [18:43:13] woo! [18:43:20] I noticed the other day that wikitech doesn't have TemplateSandbox either. [18:43:27] DEPLOY ALL THE THINGS! [18:43:43] then we can add our own hacks to wikitech's api ;-) [18:43:57] we need to get wikitech's config into the puppet repo :D [18:44:56] valhallasw: :D [18:46:12] Ryan_Lane: I think I vaguely told you a long time ago about building a service that exposes labsdb to people over the web / other ways that don't need them to work with ssh keys :D doing that now [18:46:37] yeah, I saw your facebook post about that the other day ;) [18:47:50] Ryan_Lane: heh, yeah [18:48:20] Ryan_Lane: I actually had a non-authenticated version of it working already. I took it down now so I won't suffer the wrath of ops :) [18:50:29] hahaha [18:50:34] good call [18:50:52] Ryan_Lane: indeed :D [18:51:17] Ryan_Lane: do you know how long it'll take to deploy OAuth on wikitech? I hope to get this thing finished tomorrow or the day after :D [18:52:12] I have no clue :) [18:52:19] does it need to auth to wikitech? [18:52:24] rather than production? [18:53:32] What changed on tools that broke my perfectally functioning code? -.- [18:53:53] Ryan_Lane: IMO wikitech should auth to production so I can sue my meta account [18:54:15] Damianz: I agree [18:54:38] Ryan_Lane: yeah [18:54:42] once keystone has openid connect it will [18:54:52] Ryan_Lane: since we can block or take away rights from wikitech to block someone trying to DDoS [18:55:00] Ryan_Lane: but we can't do that for prod account :P [18:55:02] YuviPanda: ah. right [18:55:28] Ryan_Lane: whom do I bug to make that happen? [18:55:29] Ryan_Lane: andrewbogott? [18:55:43] An op to get the config in puppet, then do it yourself ;) [18:55:53] andrew or coren, yeah [18:56:01] or maybe anomie, if he's feeling brave :D [18:56:17] yeah, or get an op to put the config in puppet [18:56:29] anomie: can you feel brave? :) [18:57:45] Argh... got to go, will return in a few hours to stab coren with a spoon to figure out what broke php/xml on tools yesterday [19:07:28] YuviPanda: Brave how? To put OAuth on wikitech? Where does wikitech's config live? [19:07:36] andrewbogott: Ryan_Lane ^ [19:08:24] anomie: on virt0 [19:08:52] /srv/org/wikimedia/controller/wikis/config/ [19:09:04] should likely go in Local.php and Private.php [19:09:14] Settings.php is shared with wikitech-static [19:12:13] Ryan_Lane: It seems that I don't have access to virt0. Whew. [19:12:16] * anomie is off the hook [19:12:19] :D [19:12:22] damn [19:13:07] heh I was celebrating too early looks like [20:35:58] petan, if you are around, I'm curious what you remember about labs nagios... [20:36:17] I see two different log files getting status info (one in /var/log/nagios, one in /var/log/icinga) neither making it out to irc [20:38:06] andrewbogott: think you can find some time this week to deploy OAuth on wikitech? [20:38:19] yuvipanda: unlikely, the rest of the week is US holiday [20:38:25] andrewbogott: ah, forgot [20:38:30] andrewbogott: next week? [20:38:34] and also I don't know anything about OAuth at all, so… would be a steep learning curve [20:38:39] <^d> Everything's happening next week :P [20:38:44] andrewbogott: well, I guess anomie can help. [20:38:53] andrewbogott: plus it is just a standard extension install [20:38:55] nothing more [20:39:08] That I can probably manage next week :) [20:39:43] andrewbogott: sweet [20:39:46] andrewbogott: I'll bug you! [20:39:57] andrewbogott: I'll buy you a beer if you do. :) [20:40:08] andrewbogott: i'll too! [20:40:20] well, you probably prefer Dimsum over beer [20:40:29] <^d> I'm gonna buy myself a beer as soon as I get off work. [20:40:35] <^d> In fact, I may even buy TWO beers. [20:40:45] ^d: woah!!1 [21:21:15] Coren, ping [21:21:18] petan, ping [21:22:15] hello? [21:22:39] Anybody from labs home? [21:23:29] Hellooooooo? [21:23:45] anomie, YuviPanda, ping [21:23:51] hmm? [21:24:06] YuviPanda, can you look up an old job id for me? [21:24:21] sorry, don't really know how to do that. [21:24:27] also in the middle of something else [21:24:34] * YuviPanda hasn't done much with the grid [21:24:48] :| [21:24:49] Cyberpower678: qacct -j $JOBID ? [21:25:02] anomie, thnakyouthankyouthankyou [21:25:41] <^d> YuviPanda: Do you accept the...challenge of the grid? [21:25:51] ^d: no :P [21:25:58] <^d> k, jw. [21:26:09] anomie, do you know what exit status 137 is? [21:26:15] ^d: what's 'jw'? [21:26:21] <^d> just wondering. [21:26:23] ah :D [21:26:43] <^d> vwls r nnyng [21:26:52] ^d: the grid isn't webscale enough for my tastes. doesn't even use Redis! [21:27:00] Cyberpower678: kill -9ed? I'm not sure. [21:27:06] ^d: nly smtym! [21:27:11] hmm, is 'y' a vowel? [21:27:20] <^d> "Sometimes" [21:27:25] Cyberpower678: What's up? [21:27:31] ^d: smtyms? [21:27:41] Coren, what does exit status 137 mean? [21:28:15] 128+x = killed by signal x. So, 9. I.e.: sigkill. [21:28:27] <^d> YuviPanda: Appropriate: https://www.youtube.com/watch?v=o8VjltJkdTQ [21:28:39] Coren, what does that mean? [21:28:50] The grid killed your process, hard. [21:28:56] Why isn't jsub restarting the task? [21:29:17] ^d: hahaha! :D [21:29:27] Why has it been killing my tasks that I've been restarting lately. No fatal error, or oom error. [21:29:38] Meaning I've been left clueless. [21:30:01] Coren, ^ [21:30:42] When you restarted the grid, every task but spambot got killed. When I rebooted spambot, it kept dying with in 5 minutes. [21:31:27] Coren, ^ [21:32:57] Coren, any idea what's causing it? [21:35:44] Coren, ? [21:42:30] Hm [21:42:51] Normally, the grid only sigkills if you hit a hard limit. Check the job, see if it has one? [21:45:16] Coren, can you? http://pastebin.com/bLVdUnwp [21:46:17] Coren, this exact task has been in continuous operation since 10/18/13. Ever since I restarted it yesterday, it's being dying within 5 minutes. [21:47:04] Well, that's perfectly consistent with a qdel having been done. [21:47:36] Coren, what I gave you? [21:47:48] If it had been sge killing the grid, it would have that exit_status but also an error state. [21:47:55] s/grid/job/ [21:48:00] Yes, the output you gave me. [21:48:09] So what's causing it? [21:48:17] I'm qdel ing anything. [21:48:29] And it's being sent as continuous. [21:49:36] Well, /something/ is sending it a KILL signal. Maybe you're summoning the OOM killer? [21:50:06] How? Nothing's changed on that script or how it's submitted to jsub. [21:50:42] Well, if you run out of memory then the kenerl will start killing the bigger offenders. [21:50:54] Have you monitored its ram usage? [21:51:10] How do you do that? [21:51:54] Hm. It never got over much over half a gig. Perhaps something else eats your memory? [21:52:02] That's the downside of no grid resource control. [21:52:08] Start it, I'll take a look at it. [21:52:28] Submitting it to the grid in 10 seconds. [21:52:46] 1689124 [21:55:43] Coren, how's it looking so far? [21:55:53] The script is still operational. [21:56:06] It doesn't look like it's growing ridiculously fast. [21:56:41] I'm guessing it'll crash within 20 minutes. [21:57:29] I note, however, that it's hovering close to your hard limit [21:57:59] ??? [21:58:01] The limit is at 512M, and it hovers around 473 [21:58:45] If it hits 512, the grid will kill it. [21:58:57] Wait... [21:59:04] That limit should be 2G [21:59:12] hard resource_list: jobs=1,h_vmem=524288k [21:59:25] How are you momitoring that task? [21:59:32] qstat -j 1689124 [22:00:43] So that's probably what's happening. That task consumes 1G of resources once it has everything loaded. [22:01:06] So you probably want to start it with a more generous allowance then. :-) [22:01:38] aaaand it's gone. [22:01:43] It's dead [22:02:06] So yeah, you might want to start it with moar memory!!1! [22:03:19] Rebooted with 2g limit. Should remain stable. Now why is cratstats crashing? [22:03:30] That thing has more memory than it needs. [22:04:34] Coren, what would slowly cause memory to constantly rise but not steady out? [22:05:02] Well, any sort of leak; that depends mostly on your programming language and what you're doing. [22:05:19] PHP [22:05:30] What causes a memory leak? [22:06:01] Any resource you allocate but never end up releasing. Memory is obvious, but also things like database connections, files, etc. [22:06:37] PHP normally has a hard limit of its own, however, and does garabage collection. [22:07:08] spambot uses 1 gig. Could that be from 2 active database connections? [22:07:15] But you might be running up into issues where you grow some array, or allocate large strings in a way the garbage collector has no way of knowing you aren't using anymore. [22:07:38] So what's happening with cratstats? [22:07:43] Not from the connections themselves, but I expect it's likely that results sets can end up consuming a lot of memory. [22:08:05] And how would I release them? [22:08:15] I wouldn't know. Finding leaks is a complicated process that requires a lot of profiling. [22:08:56] Cause I'm a little stumped right now. I've looped and looped that program and it's always using the same amount of data, but in the end spambot is using 1 gig. [22:09:01] http://stackoverflow.com/questions/849549/diagnosing-memory-leaks-allowed-memory-size-of-bytes-exhausted might help. [22:10:04] I'm afraid that that level of dev support is a bit out of scope, and not my field of expertise. I'm sure you can find people who are very good at it and willing to help on the mailing lists though. [22:10:30] But PHP is renowned for not being very good at memory management since it was never intended for long-running programs. [22:11:56] <^demon|meeting> I remember when we used to have to call ->res() on every freaking database result because they didn't release when going out of scope. [22:12:06] <^demon|meeting> Or release() or w/e we called the damn thing. [22:18:00] Coren, it could be the foreach loop. [22:19:20] <^demon|meeting> Depends. Whatcha doing in the foreach? [22:20:33] An array of course. [22:21:15] It looks like the foreach function is cloning the object it's being fed, and never releases the memory after it was used. [22:22:37] <^demon|meeting> Cyberpower678: Pastebin? I know you'd be using an array in a foreach, I meant what sort of logic are you doing *in* the foreach? [22:22:59] Hold on... [22:25:22] ^demon|meeting, [22:25:24] http://pastebin.com/NaKG7Y9c [22:26:42] <^demon|meeting> So yeah, that regex in a foreach might cause problems if it's a slow regex, so I'd look there. [22:27:07] <^demon|meeting> Also, constructing a new Database each time in the while loop is a waste, you could just construct it outside afaict. [22:27:17] <^demon|meeting> (Probably isn't your problem, but certainly isn't helping :)) [22:28:26] ^demon|meeting, I couldn't agree more, if it didn't go to sleep 5 minutes after being called. That's why I have it re-establish at the end of each iteration. [22:29:09] <^demon|meeting> I'm sure there's a clever way to fix that :) [22:29:47] * YuviPanda whispers "use a language that has proper connection pooling" [22:29:59] I haven't figured out that clever way. Anyway, it's not really leaking. It holds constant at 1 Gig. [22:30:08] YuviPanda, what do you recommend? [22:30:54] Cyberpower678: you can use PHP itself, if that is what your bot is in right now [22:30:55] Cyberpower678: http://www.php.net/manual/en/mysqlnd-ms.pooling.php [22:31:25] YuviPanda, ?? [22:31:48] http://www.php.net/manual/en/mysqli.persistconns.php and http://ir.php.net/function.mysql-pconnect [22:32:02] Cyberpower678: they are called 'persistent connections' in php mysql land apparently [22:32:32] YuviPanda, actually, the problem is the DB goes away server side. [22:32:42] For the local db only. [22:32:51] the *connection* goes away [22:33:19] besides, if you read the links I pasted, you'll notice that they will handle the server closing the connection too [22:33:52] The bot has no trouble maintaining the connection to the enwiki db [22:34:05] well, ok :) [22:34:11] So it's not my bot. [22:34:40] YuviPanda, tl;dr [22:34:42] ;) [22:35:06] Cyberpower678: well, I'll not even bother next time then :) [22:36:54] YuviPanda, can have a look at a script for me and tell me what is causing it to OOM [22:48:01] Seriously, Cyberpower678? YuviPanda takes the time to research things for you and give you good pointers to information you need to debug your problem and you respond with "tl;dr"? [22:48:34] No. I'm not being serious. [22:48:58] I have those links open now. [23:22:51] Users may be interested in http://tools.wmflabs.org/sigma/wikimedia-labs-button.png [23:24:11] SigmaWP: ah, nice, I already did something else: http://1.1.1.5/bmi/tools.wmflabs.org/static/img/powered-by-tool-labs.png [23:24:34] eehhh [23:24:48] it loads not [23:24:50] http://tools.wmflabs.org/static/img/powered-by-tool-labs.png [23:25:00] Hm. [23:25:34] Meh [23:26:28] I’m hosting some common scripts as Bootstrap or jQuery on /static/, so you don’t have to copy them in every tool. if you need something else, I can add it to that project [23:26:52] You should add my icon to that :D [23:27:26] SigmaWP: you even can do this yourself, all code is on GitHub ;) [23:28:55] ireas:SigmaWP: Is your artwork already on commons? [23:29:01] hedonil: no [23:29:03] * SigmaWP goes to upload [23:32:16] hedonil: https://commons.wikimedia.org/wiki/File:Powered_by_labs_button.png [23:33:13] ... so what now [23:33:19] * hedonil confesses he has borrowed the logo some days ago from some other tool, maybe copyrighted by ireas :-D [23:33:59] SigmaWP: I’ll add yours to /static/, one moment [23:34:01] ;-; [23:34:04] ireas: Thanks [23:35:08] hedonil: well, I’m German, and in Germany, simple icons like these are not protected by copyright. So feel free to use it as you want, as long as the Foundation does not complain ;) [23:35:20] SigmaWP: does your logo has any improvements? new features? more layers? better alpha? round corners? [23:35:31] hedonil: just being curious, where did you find the logo? [23:35:46] * hedonil thinking [23:35:50] hedonil: i just took https://toolserver.org/images/wikimedia-toolserver-button.png and changed it up a bit [23:37:59] ireas:can't remember exactly, what are your favourite tools? [23:38:04] Σ [23:38:15] Interesting, the irc logs have terrible unicode support [23:40:07] ireas: yeah, it was stimmberechtigung. [23:41:00] SigmaWP, expect to hear from lawyers. [23:41:16] They want to give you a barnstar for the buttons. [23:41:35] :D [23:41:43] !cookie SigmaWP [23:41:43] no cookie for you [23:41:59] Well...that's mean. [23:42:09] hedonil: ah, okay :) [23:42:27] Cyberpower678: found your memory leakage? [23:42:35] No [23:43:04] But it's not severe so, I'm not stressing over it. [23:43:47] Spoken like a true PHP programmer [23:44:43] Cyberpower678: like 50cent says - die tryin' to find it :-D [23:45:10] SigmaWP, is that a good or bad thing? [23:45:41] SigmaWP: http://tools.wmflabs.org/static/ [23:46:18] ireas: Thanks :D [23:47:15] Cyberpower678: �I'm not a real programmer. I throw together things until it works then I move on. The real programmers will say "Yeah it works but you're leaking memory everywhere. Perhaps we should fix that." I�ll just restart Apache every 10 requests�, from https://en.wikiquote.org/wiki/Rasmus_Lerdorf . And that's not up to me to decide. [23:48:25] I at least make an effort to make it run efficiently. [23:51:25] I never said you didn't [23:55:16] ireas: how about putting your logo on commons too? [23:56:43] hedonil: nothing bad about that, I think. one moment…