[00:32:20] andrewbogott or anyone, what can labs "Project Members" do? Can they ssh to project instances and sudo on them? https://wikitech.wikimedia.org/wiki/Help:Access#Project_Members (new section) doesn't say [00:33:21] VolkerE and I need to be able to ssh to https://wikitech.wikimedia.org/wiki/Nova_Resource:Living-style-guide.design.eqiad.wmflabs and fiddle with labs-vagrant, do we need to be admins or just members of Nova_Resource:Design ? [00:39:23] spagewmf: yes, ssh access. And possibly sudo, depending on the project-specific sudo rules. [00:39:37] I think that by default members have complete sudo rights [00:41:33] Really? [00:41:40] I thought it was only admins by default [00:45:00] andrewbogott, oh, you're right [00:45:43] sudo policy name = default, users = all project members, commands = all, options = !authenticate [01:05:39] andrewbogott: thanks, I'll update. For labs-vagrant, the key right is sudo su vagrant or sudo -u vagrant , do project members have that [01:06:41] they should, yes, unless intentionally excluded. [01:16:56] 6Labs, 10LabsDB-Auditor, 5Patch-For-Review: Remove unused views from labsdb - https://phabricator.wikimedia.org/T85867#1542039 (10Krenair) hitcounter killed in https://gerrit.wikimedia.org/r/#/c/187119/ [01:19:31] 6Labs, 10LabsDB-Auditor, 5Patch-For-Review: Remove unused views from labsdb - https://phabricator.wikimedia.org/T85867#1542046 (10Krenair) That could probably be dropped from production actually... [03:05:32] 6Labs: Labs proxy 502 should indicate where to ask for support - https://phabricator.wikimedia.org/T109078#1542117 (10scfc) Causes //more// issues? I would find a clearer error message on the proxy very useful. When I had set up a MediaWiki or whatever and I couldn't access it at the URL I thought was the corr... [03:58:37] labs down for anyone else? [03:58:37] MusikAnimal: think so [03:58:38] Yeah [03:58:48] it had been slow for the past few hours for me [03:59:10] Got stuff coming back online npw [03:59:28] same [04:00:00] doesn't seem as slow now either [04:01:57] ok, now on to my actual question: still trying to get my Ruby tool running. I've gone by the [[Help:Tool Labs/Web#Other web servers]] and ended up with a httpserver.sh that looks like: http://pastebin.com/U357uE8Y [04:03:24] someone mentioned earlier the portgrabber was flaky when you need to set export stuff to PATH, so I made it call a function [04:03:54] anyway the service is running, e.g. I can go to http://tools.wmflabs.org/musikanimal/ and get a 404, but the Ruby app was not booted [04:04:05] so nothing is being served :( [04:06:50] MusikAnimal, Betacommand, Earwig — I didn’t get any alerts about labs downtime. Can you tell me what symptoms you saw? [04:07:01] just really slow [04:07:09] andrewbogott: seemed like an NFS issue, if I had to guess [04:07:20] andrewbogott: SSH connections stalled, and I saw by bots go down for a short time [04:07:21] slowness when doing directory-related things but not regular ssh [04:08:06] yeah, I find that I can't tab anything out, as with `cd ` [04:08:11] Can you give me an estimate for when it started and when it got better? [04:08:37] I first noticed it about two hours ago [04:08:55] maybe less [04:09:00] hm, ok [04:09:07] yeah and around 20 minutes ago I couldn't browse to anything on labs, even https://tools.wmflabs.org [04:09:20] but a minute or two later it came back [04:09:35] I ran a big archive script on NFS earlier in the day, but it finished 5 hours ago. So probably unrelated... [04:10:05] it still seems a little slow [04:10:11] a random `ls` took several seconds [04:10:58] andrewbogott: see mailing list [04:11:11] hm, yep, I see evidence of poor performance in the monitoring graphs [04:11:16] ok, catching up on email :) [04:13:05] yeah, someone is reading/writing like mad to /data/scratch and ruining the party for everyone [04:13:14] who to blame, who to blame... [04:14:22] Coren: if you’re around, help me figure out who’s abusing NFS? [04:16:16] sorry, sounds like we've got bigger problems, but back to my tool: anyone know how the portgrabber stuff works? I've done everything the docs said to do [04:16:54] I feel like there must be better logging somewhere, httpserver.err just has some nonsensical python output (my tool is Ruby) [04:40:37] anyone able to provide advice about "qalter" reading the man page is a nightmare to decipher among all the commands [04:44:04] tools.wikisource-bot@tools-bastion-01:~$ qmod -rj 1798276 [04:44:04] The job 1798276 is running in queue task@tools-exec-1219.eqiad.wmflabs where jobs are not rerunable. [04:44:10] huh? [04:47:11] not an issue [04:55:47] gifti: are you around? [05:08:38] !log tools suspending tools-exec-gift, just for a moment... [05:10:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, dummy [05:14:44] !log tools resumed tools-exec-gift, seems not to have been the culprit [05:14:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, dummy [05:36:10] Earwig, Betacommand, better now? [05:36:51] seems normal [05:37:05] guess you figured it out? [05:37:14] …maybe? I killed some things :) [05:37:39] great :P thanks for your work [05:38:02] I’m going to go to sleep before the graphs spike again and demonstarte that correlation != causation [05:38:12] sounds like a plan [05:38:35] sorry for the bumpy ride [08:02:45] andrewbogott: i'm now [09:02:33] i switched that to local storage now, that should be safe, i think [09:29:12] sDrewthedoff: why would you need to qalter anything? [09:29:27] sDrewthedoff: and you can only reschedule continuous jobs [10:30:45] 6Labs, 10Labs-Infrastructure: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#1542310 (1080686) I would be happy if, instead of discussing this for months and locking me out of tools and Git, someone could take a few minutes and fix this, so I have a valid login again. If you rename it -... [10:40:59] 6Labs, 10Labs-Infrastructure: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#1542322 (10valhallasw) >>! In T63967#1542310, @80686 wrote: > I would be happy if, instead of discussing this for months and locking me out of tools and Git, someone could take a few minutes and fix this, so I ha... [10:42:54] 6Labs, 10Labs-Infrastructure: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#1542326 (1080686) I also had a toolserver account with that name and would rather want to keep my history in SVN / Git etc. Deleting it or renaming it shouldn't matter. [11:10:06] 6Labs, 10Labs-Infrastructure: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#1542352 (10valhallasw) Accounts from the Toolserver have not been migrated to Tool Labs, so that account doesn't exist anymore. The source control history is not linked to your actual user account, so that should... [12:26:22] 6Labs: Labs proxy 502 should indicate where to ask for support - https://phabricator.wikimedia.org/T109078#1542411 (10valhallasw) Let me clarify what I mean. - If the /proxy/ throws a 503 because of a relay failure, we should return more information. - If the /application/ returns a 404, we shouldn't touch the... [12:26:42] 6Labs, 10Tool-Labs, 5Patch-For-Review: Let error responses pass through the proxy if they contain contents - https://phabricator.wikimedia.org/T66393#700815 (10valhallasw) [12:27:07] gifti: yep, switching to local storage sounds good — nfs performance is still looking fine. [12:27:32] Although I just got an alert about tools-exec-giftbot’s disk filling up [12:27:38] … and then an ‘all clear' [12:27:45] So you must be on top of that, one way or another [12:28:01] yeah i know, i'm looking into it [12:28:08] great [12:28:22] Hope I didn’t mess you up by killing your job :) [12:28:35] somehow it uses absurd amounts of storage, but i don't know why yet [12:28:56] no, if it blocks labs you're right to do that [12:30:41] 6Labs, 10Labs-Infrastructure: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#1542424 (10Andrew) @80686: part of the context here is that renaming users is a tremendous pain in the neck :) I've tried it several times, never with complete success. [12:31:55] gifti: I was sort of charmed to see you running a tool in tcl. I was a full-time tcl developer a decade or so ago. [12:32:06] wow :) [12:32:17] My first thought was, Oh, tcl, I can debug this! But then I remembered the pain and thought better of it [12:32:26] ^^ [12:32:50] iirc every single bug ever was a quoting bug. [12:33:20] you have to get used to the quoting, but then it is easy [12:33:33] hm, maybe I never got used to it :( [12:52:55] 6Labs, 10Labs-Infrastructure: Labstore primary needs more frequent cleanup of snapshots - https://phabricator.wikimedia.org/T109176#1542466 (10coren) 3NEW a:3coren [13:54:47] 6Labs, 10CirrusSearch, 10Datasets-Archiving, 6Discovery, 10Labs-Infrastructure: Make available an XL labs instance with ~350GB available disk space. - https://phabricator.wikimedia.org/T108767#1542512 (10Hydriz) Will be great to have this for the Dumps project as well [15:01:28] ok, i'm clueless, the programm at a point just freaks out and writes lots of question marks, then device is filled up, the end [17:18:53] (03PS1) 10Sitic: Add support for seperated watchlists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/231789 [17:53:52] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GWicke was created, changed by GWicke link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/GWicke edit summary: Created page with "{{Tools Access Request |Justification=Some ad-hoc database queries, like summing up the size of old revisions. |Completed=false |User Name=GWicke }}" [17:59:06] (03PS2) 10Sitic: Add support for seperated watchlists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/231789 [18:02:41] (03PS3) 10Sitic: Add support for subdivided watchlists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/231789 (https://phabricator.wikimedia.org/T109188) [18:04:33] Gah! [18:04:47] Who's doing this crazy grep with tools.cluebot right now? [18:14:20] check the sudo log? :-) [18:15:27] Aug 15 18:01:32 tools-bastion-01 sudo: pam_unix(sudo:session): session opened for user tools.cluebot by richs(uid=0), probably [18:15:49] = [[User:Rich Smith]] [18:15:56] * valhallasw`cloud looks expectedly at wm-bot [18:16:14] Coren: ^ [18:16:56] valhallasw`cloud: That's allright - I have already savagely murdered the process and left a message on the pty [18:16:57] :-) [18:17:06] * Coren got pinged by an alert, so that works. [18:18:32] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GWicke was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=174027 edit summary: [18:32:10] Coren: did you have a chance to think about the spf record? [18:33:01] Yes, it's kinda cruddy because we have no reasonable mechanism otherwise, so I don't think we have a choice for now. There are issues with the LDAP schema though which I'll need to consult Andrew on. [19:01:58] Coren: how does the ldap schema come into play? [19:03:18] valhallasw`cloud: I don't think it does anymore, but we didn't use to support TXT elements without a lot of trickery in LDAP. With the move to designate, I'm not sure if the fact that LDAP is kept in parallel will affect this. Hence, check with Andrew. [19:03:25] Otherwise yeah, we need the TXT RR [19:04:08] Ah, right. I thought it would be easy given that there already is a record at the moment :-) [19:22:29] 6Labs, 10Tool-Labs: webservice stuff encoding isn't utf8 - https://phabricator.wikimedia.org/T108283#1542778 (10valhallasw) 5Open>3Invalid a:3valhallasw sys.stdout.encoding is "ANSI_X3.4-1968" because the job is running under SGE and thus under the 'C' locale. The 'C' locale specifies 'ANSI_X3.4-1968' (=... [19:24:31] 6Labs, 10Tool-Labs: Please install hugin-tools and pillow - https://phabricator.wikimedia.org/T108210#1542791 (10valhallasw) [19:24:32] 6Labs, 10Tool-Labs, 7Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#1542790 (10valhallasw) [22:49:11] Hi gentlemen, They could help me with something? The script XTools.js stored in the tool labs is It is not working [22:49:44] I don't know if the xtools maintainers are here... [22:52:48] Hunm, Hedonil I know, are inactive for a long time ... [22:53:00] Le0n_: Is the issue that the XTools.js is unreachable or that there is some bug in it? [22:53:17] Le0n_: I might be able to help the former, but not the latter. [22:59:45] https://meta.wikimedia.org/w/index.php?title=User:Hedonil/XTools/XTools.js&action=raw&ctype=text/javascript [23:01:09] You can check it? [23:03:33] Le0n_: I've restarted the xtools web service, maybe this will help? [23:07:15] the channel does not allow to use the term scri.p.t [23:12:25] well, that was it, byeeeeee [23:17:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Diego Grez-Cañete was created, changed by Diego Grez-Cañete link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Diego_Grez-Ca%c3%b1ete edit summary: Created page with "{{Tools Access Request |Justification=I am an admin on MediaWiki and the Scots Wikipedia. In the past, I was a Toolserver user, with primarily some IRC bots and loggers for Wi..." [23:26:05] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Diego Grez-Cañete was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=174038 edit summary: