[00:01:34] yeah, I'm doing it again [00:02:28] I am in the same group as the tool [00:03:25] but the file's group is wikidev [00:06:37] * bd808 goes looking for the take source code [00:07:51] isn't the problem that the file has the wrong group by default? [00:09:02] there's no "right" group by default, but I understand what you mean [00:09:27] and yes, looking at the source that appears to be a requirement [00:10:50] by "wrong group", I meant a group that doesn't let me take the file [00:10:56] so the correct magic seems to be: cp; chgrp -R tools. ; become ; take [00:11:50] and yes this is horrible. I think we have a feature request/bug report about all of this being too hard already... let me find it [00:12:35] this wasn't even documented on the wiki until someone here helped me through it and updated it a couple days ago [00:14:06] I have this funny feeling that at some point in the past we had the group sticky bit set on all tool $HOME directories (which would force the group) and then that changed at some point [00:15:50] ls -ld /data/project/admin -- drwxrwsr-x 16 tools.admin tools.admin 4096 Jun 6 22:35 /data/project/admin/ [00:16:02] the group sticky bit is set there like I expected [00:16:09] ksft: what's your tool name? [00:16:13] rfastats [00:16:52] hmmm... it has the group sticky bit set as well... maybe I'm not remembering correctly what that does [00:18:45] "A directory has the sgid mode enabled means that any files or directories created within it will inherit the group id of the directory. This can be very useful for directory that is used by a group of people working on say the same project..." -- http://www.miniaturelinux.com/Linux-Access-Modes-and-Sticky-Bits-Examples.php#Directories-and-sgid [00:18:56] that's what I thought it should do... [00:19:12] I uploaded the file to my home directory, and then I moved it to /data/project/rfastats [00:19:17] oh... but not resursively [00:20:00] ksft: did you already chmod the file(s)? [00:20:06] err... chgrp [00:20:27] yeah [00:20:41] I can do it again if you want [00:21:03] sure. I'd like to at least have a guess at what is going wrong [00:22:16] okay, I just uploaded an empty file to /home/ksft/foo [00:22:39] and I moved it to /data/project/rfastats [00:22:56] and I can't take it as rfastats [00:23:12] *nod* I see it and I see that the group is not forced to tools.rfastats [00:23:50] * bd808 grumbles under his breath [00:24:03] okay, this time I ran `scp bar ksft@login.tools.wmflabs.org:/data/project/rfastats' [00:24:13] uploaded a new file directly to the tool's home directory [00:24:23] it has the right group [00:24:30] tools.rfastats, that is [00:24:38] interesting [00:25:57] unrelated to this, where can I put static files for it to serve? [00:26:13] apparently sgid works for cp but not for mv -- https://ubuntuforums.org/showthread.php?t=1128418 [00:26:23] TIL [00:33:50] ksft: static files for the flask app? It looks like we don't set that up by default, but you can do it by adding a $HOME/www/python/uwsgi.ini file with a "[uwsgi]" section having "check-static = /data/project//" in it. $HOME/www/static would be an ok choice for that and then you will also be able to get those same static files from https://tools-static.wmflabs.org/tools-static.wmflabs.org/ [00:34:31] See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Static_file_server for the "magic" tools-static.wmflabs.org docs [00:35:13] $HOME/www/python/static would also be a reasonable choice for check-static [00:35:30] see also https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html [00:37:17] that's interestingly complicated [00:37:29] much like life ;) [00:37:50] but yes, many options [00:38:44] yeah [00:40:17] We could probably pick either $HOME/www/static or $HOME/www/python/static and make that the default in the config that the `webservice` command generates for uwsgi [00:40:46] does it put that config in a file somewhere? [00:42:28] not really, no. It is actually done using the CLI flags when starting the uwsgi service. See https://github.com/wikimedia/operations-software-tools-webservice/blob/master/toollabs/webservice/services/uwsgiwebservice.py#L19-L27 [00:43:24] I created that file and added what you suggested, but it doesn't seem to be working [00:43:52] you will need to `webservice restart` for it to be read [00:44:20] I did [00:44:31] https://github.com/wikimedia/operations-software-tools-webservice/blob/master/toollabs/webservice/services/pythonwebservice.py#L19-L36 -- that's actually the default config for the --backend=kubernetes python* uwsgi container [00:44:39] do I need to pass it all those options again? [00:45:01] what's `webservice shell'? [00:45:15] and is this all documented somewhere so I don't have to keep asking you? [00:45:51] ksft: no need to duplicate the default config [00:46:38] what's that? [00:47:55] the last default arg is `--ini $HOME/www/python/uwsgi.ini`. This makes anything in the ini file override the defaults that came before it [00:48:39] Having trouble adjusting to the timezone, bd808? 😉 [00:49:15] bstorm_: apparently. :) I was doing manager junk and then found a user to help [00:49:27] oh, no I meant the options "--backend=kubernetes python3.5 shell" [00:50:39] ksft: webservice --backend=kubernetes python3.5 shell -- starts a Kubernetes pod (a Docker container) using the python3.5 Docker image and then attaches your ssh session's input and output to it [00:50:54] Its kind of like an ssh session, sort of [00:51:18] and it stops it when I exit the shell? [00:51:23] yeah [00:51:48] now it's returning a 502 [00:52:12] We have docs about using `shell` in places like https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python_(Python3_+_Kubernetes), but I don't see a real description there of what the command technically does. :/ [00:52:30] ksft: anything being written to $HOME/uwsgi.log? [00:53:05] ImportError: No module named 'flask' [00:53:21] yep [00:53:39] that didn't fix the 502 error, though [00:53:51] https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool#Add_Flask_to_the_virtual_environment [00:54:17] which I'm getting for both the static file and the path Flask should be handling [00:54:50] ksft: did you restart the webservice again after adding Flask to the venv? [00:55:12] no [00:55:21] should that affect its ability to serve static files? [00:55:33] I hope an ImportError wouldn't cause that to break [00:55:56] restarted, same thing [00:56:18] NameError: name 'flask' is not defined [00:56:25] oh, hah [00:56:32] typo in the code [00:56:42] but I still don't think that should affect the static file [00:57:22] yeah, I'm not sure. This may be a sign that the static mount thing isn't working as hoped [00:58:09] you at least keep getting different errors: ImportError: No module named 'mwapi' [00:58:48] yeah, I just fixed that one too [00:58:48] "mounting /data/project/rfastats/www/python/src/app.py on /rfastats" looks like a good sign [00:58:58] but I'm still getting a 502 [00:59:10] wait, not anymore [00:59:13] https://tools.wmflabs.org/rfastats/sharededitors.html [00:59:16] yeah :) [00:59:19] oh, that's working [00:59:29] yay, now I just have to fix my code [00:59:32] took it a bit to get all connected up to the front end proxy [00:59:45] yeah [00:59:58] I didn't manually restart it that last time [01:00:38] I need to try and sleep :) Good luck with your continued experiments! [01:00:57] of course [01:01:01] thank you for all the help [01:01:56] sure! thank you for asking questions and struggling through the frustration [02:00:00] I maybe found a neat trick... is `SELECT 1 FROM actor_revision WHERE actor_name = 'Foo'` a safe way to tell if Foo has made any edits? [02:00:53] it's faster than `SELECT 1 FROM revision_userindex JOIN actor_revision ON rev_actor = actor_id AND actor_name = 'Foo'` [02:04:25] *`WHERE actor_name = 'Foo'` [07:52:40] musikanimal: yeah, that should be functionally the same and pretty fast [13:54:53] !log wikidata-dev wikidata-new-wbterm updated core to e516669a9e and Wikibase to 4a32bd435d [13:54:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [14:01:47] !log wikidata-dev wikidata-new-wbterm set tmpItemTermsTwoMillionMigrationStage to MIGRATION_WRITE_BOTH [14:01:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [15:02:59] !log libraryupgrader -04 has a full disk, and is broked. oops. [15:03:00] legoktm: Unknown project "libraryupgrader" [15:03:00] legoktm: Did you mean to say "tools.libraryupgrader" instead? [15:03:04] no [15:03:11] !log library-upgrader -04 has a full disk, and is broked. oops. [15:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [15:03:15] but thanks stashbot [15:13:05] its not even letting me delete the stopped docker containers ugh [15:13:11] I'll just blow it up [15:13:14] and start over [17:16:03] !log ores staging ores-wmflabs-deploy:989c436 [17:16:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [17:19:25] !log ores deploying ores-wmflabs-deploy:989c436 [17:19:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:01:50] Who can help me with Quarry. Downloads are painfully slow. My large dataset is coming in at 140 KB/s. I've got 1.3 million rows in the resultset. [18:03:47] !log tools deleted anomalous kubernetes node tools-worker-1019.eqiad.wmflabs [18:03:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:48:36] !log ores staging ores-wmflabs-deploy:079f18c [18:48:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:50:57] !log ores deploying ores-wmflabs-deploy:079f18c [18:50:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [19:26:14] ;;;; [21:05:57] replag has steadily been increasing for a long while, now up 25+ hours on s8, 10+ hours on s1, etc. Is there any cause for concern? https://tools.wmflabs.org/replag/ [21:07:32] musikanimal: Not overly, DB maintenance still going on AFAIK [22:43:44] !log tools.pbbot Deploy eebdc13 (link to SAL) [22:43:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL [23:26:14] There is some seriously wonky caching happening with the webservice [23:27:13] Because I've been trying to fix an inexplicable bug that works perfectly on my instance, so for shits and giggles, I deleted a dependency from the tool and ran it, and it still runs even though it should be erroring out now. [23:27:30] Cyberpower678: "caching" where? [23:27:39] bd808 iabot tool. [23:28:11] Nothing in the https://tools.wmflabs.org/ front proxy caches pages [23:28:25] I'm running a quick and dirty test script and it's been giving me headaches. When I deleted the supposedly faulty file, it still continues to run even the file isn't there anymore to require_once it [23:29:27] The file I deleted is needed to even do the job it's supposed to be doing, so the fact that it's still able to load this non-existent file tells me there is a problem with the NFS cache somewhere. [23:30:34] did you try restarting the webservice to verify that changes the behavior? [23:30:40] iabot should be completely broken now, including the web-interface, but if you go there, you see it loads just fine. [23:31:18] I haven't yet. Assuming that every file change shouldn't prompt a restart, I'd figured I'd leave it for you to look at before I do. [23:31:56] what file are you expecting to not exist (full path please) [23:32:30] Even odder is that it takes on file changes of other files like the init.php file, but files like parse.php, now deleted, don't register [23:33:30] [23:31] < bd808> what file are you expecting to not exist (full path please) [23:33:38] bd808: /mnt/nfs/labstore-secondary-tools-project/iabot/IABot/parse.php is deleted, but the webservice is still using some cached version of it and loading it. [23:34:23] parse.php is a critical core file of IABot, and nothing works without it. https://tools.wmflabs.org/iabot should be completely broken. [23:35:02] This would also explain why IABot has been misbehaving so often, and I couldn't even figure out what was causing it. This has been causing headaches for 2 weeks now. [23:35:03] Cyberpower678: -rwxr-xr-x 1 cyberpower678 tools.iabot 210K Apr 6 12:47 /data/project/iabot/public_html/parse.php [23:35:18] I think you are deleting a file that is in a different path than your webroot [23:36:23] Your path is completely different to mine. When I login, I'm taken to the path I mention above. I've been writing to that path for over a year now. [23:37:03] I don't see any custom lighttpd config, so your --backend=gridengine lighttpd webservice will be using /data/project/iabot/public_html/ as the webroot [23:38:11] the /data/project/iabot and /mnt/nfs/labstore-secondary-tools-project/iabot base paths are the same in reality. /data/project is a symlink to /mnt/nfs/labstore-secondary-tools-project [23:38:21] bd808 /data/project/iabot/public_html redirects me to /mnt/nfs/labstore-secondary-tools-project/iabot/public_html [23:38:47] right, symlink. Now do an `ls -lh cache.php` in that dir [23:38:48] Which is what I'm writing to. parse.php does not exist for me. [23:40:17] Okay something weird is going on. [23:40:30] It just broke. [23:44:31] bd808 you help me in ways you don't even realize. When you echoed back the parse.php file, since IABot has multiple include paths, it was including an unintentionally cloned and out date file. XD [23:44:44] Deleting it forces inclusion of the correct file. [23:45:07] bbl