[16:56:07] !log depooling tools-sgeexec-0935, 36, 38, 40 and moving to Ceph [16:56:08] andrewbogott: Unknown project "depooling" [16:56:14] !log tools depooling tools-sgeexec-0935, 36, 38, 40 and moving to Ceph [16:56:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:59:02] Hello, nobody answered about my problem with the venv from the cloud mailing-list (python packages are found or not found depending on the activation of the interactive pod). Anybody has a clue or could explain a bit? [16:59:48] let me see [17:00:21] I copy my email [17:00:23] Hello,I am still a bit stuck with my worker. It worked briefly last week, but Isuspect that my environment was not consistent. I suppose the source of myproblems is my bad understanding of the virtual environment (see also myemail last week about python 3.5 and 3.7), and of the "root" path. Today, Icleaned everything and started from scratch (for [17:00:23] the 10th time :/). I madesome observations that are not clear to me. As a recall, my website usesdjango and has a celery/redis worker.*Try 1: following*https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/PythonSo I install the venv using "webservice --backend=kubernetes python3.7shell", which means using the interactive pod. When I am in the [17:00:24] interactivepod, with venv activated, kubectl is not found (bash: kubectl: command notfound). The packages are however found, for instance "celery" in the shellreturns no error. It is also possible to migrate the app. There is a foldervenv/lib/python3.7... There is a celery and a django folder inside. Then ifI "exit out of webservice shell" and [17:00:24] re-activate the venv. Kubectl isfound, but celery sends me:Traceback (most recent call last): File "/data/project/cycling-init-bot/www/python/venv/bin/celery", line 5,in from celery.__main__ import mainImportError: No module named 'celery'If I go to www/python/src (or don't actually) and try to migrate, I get:Traceback (most recent [17:00:25] call last): File "manage.py", line 10, in main from django.core.management import execute_from_command_lineImportError: No module named 'django'The "webservice --backend=kubernetes python3.7 start", works.So I don't quite understand what are the discrepancies between interactiveand normal pod. We install things in one context but then they are [17:00:25] notfound in the other one... Shall I do something with the PYTHONPATH maybe?*Try 2: without interactive*Now I don't use interactive at all. When typing "python3 -m venv$HOME/www/python/venv". It creates a python3.5 venv (lib/python3.5/). Inthis case, celery command and migrations works. However by starting thewebservice "webservice [17:00:26] --backend=kubernetes python3.5 start", I get inuwsgi.logs*** Operational MODE: preforking ***mounting /data/project/cycling-init-bot/www/python/src/app.py on/cycling-init-botTraceback (most recent call last): File "/data/project/cycling-init-bot/www/python/src/app.py", line 5, in from django.core.wsgi import [17:00:26] get_wsgi_applicationModuleNotFoundError: No module named 'django'For try 1 and try 2, the worker didn't start (python image version wasadapted/variated, it changed this time nothing). As kubectl logs is emptyand the describe deployment gives no information, it is quite difficult todescribe what happens. I noticed [17:00:27] thathttps://github.com/Wikidata/editgroups usesa separate venv at the root of the project for celery.Psemdel [17:00:55] I suppose that I miss something foundamental [17:01:00] fundamental :) [17:01:10] please don't paste lots of text here, linking your previous email would've been fine [17:01:47] but in any case a venv created in the `webservice shell` is only expected to work inside that shell or when doing `webservice start` [17:01:58] ok [17:02:18] so you can do your migrations in the webservice shell [17:02:27] as celery has to be started from a separated deployment (without webservice) it means it cannot work [17:02:57] if you need to run celery on the grid, then you can either create a separate venv, OR use python3.4 everywhere [17:03:48] but why is kubectl not found in the shell? [17:04:07] you can also run celery as a k8s continuous job with 3.7, see e.g. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_continuous_jobs [17:04:24] why do you want to use kubectl in the shell? [17:04:58] for nothing, but I am surprise to do action inside and other outside [17:05:02] it is a "python" shell? [17:05:19] it's an interactive k8s pod [17:05:25] ok [17:05:55] and if I create venv without shell (try 2) then in the command, it finds django [17:05:58] so I can migrate [17:06:07] but why uwsgi does not find it? [17:06:52] if you did webservice --backend=kubernetes python3.5 shell it would work I believe [17:06:58] er, webservice --backend=kubernetes python3.7 start [17:07:08] one more time: webservice --backend=kubernetes python3.5 start ** [17:07:24] I think I did that, but nevermind [17:07:26] the Python version you use to create the venv has to match what you run it with [17:07:34] it sounds rational [17:07:53] thank you for the explanation, I will give it a new try [17:08:28] for my tools where I have a worker process and a web process, I usually just create a separate venv [17:08:55] it seems to be the best solution [17:09:04] you put it at the root of the tool [17:09:11] or at www/python/ ? [17:10:39] I have my worker venv at ~/venv and web at ~/www/python/venv [17:10:52] feel free to look around /data/project/phpcs, it's pretty simple [17:12:07] editgroups, which I copied used also this config. It is just, that I didn't understand what I was doing, leading to low chance of success. I will give it a try. [17:12:37] and I know that it is pretty simple, no idea, why it keeps doing "CountainerError" [17:12:43] and LoopCall... [17:15:39] what is creating that error? [17:18:53] that is the problem, I don't know, as the logs is empty and the describe deployment explains nothing [17:19:06] but if the venv is not found or celery is not found, it cannot work [17:22:50] I mean, what command are you running? [17:22:54] What's the full error message? [17:24:59] kubectl create --validate=true -f $HOME/www/python/src/script/deployment.yml [17:25:52] then it happens nothing [17:25:59] when i make kubectl get pods [17:26:05] then I see LoopCall... [17:26:39] what's your tool name? [17:26:52] cycling-init-bot [17:27:00] but no idea what is the venv right now [17:27:13] I tried last week and sent the email [17:27:27] let me try with a separate venv [17:27:41] maybe I will have more success then [17:27:51] > /data/project/cycling-init-bot/www/python/venv/bin/celery -A CyclingInitBotSite worker -l info [17:27:57] I think you're still using the web venv? [17:28:23] yes I had only one [17:28:39] in the command, I can type celery -A CyclingInitBotSite worker -l info [17:28:41] and it starts [17:29:20] ok, but we just talked about creating a separate venv for celery... [17:29:25] but automatically with the deployment, there was an error. So that's why I supposed that the "state" was different [17:29:37] I said I will do it [17:29:50] I eat in 5 min, I do it at once afterwards [17:29:59] those thing can last some time for me... [17:30:27] but I think, with your explanation it will be better. [17:30:52] ok :) [19:04:55] unfortunately still not working :/ [19:06:12] !log tools repooling tools-sgeexec-0935, 36, 38, 40 and moving to Ceph [19:06:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:07:18] !log tools depooling tools-sgeexec-0932, 33, 34 and moving to Ceph [19:07:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:21:51] it (seems) to work! I pulled my git repo last time, but forgot to make chmod again on the script... [19:45:48] hello all [19:46:14] I am logging into toolforge paas after long time [19:46:18] webserver seems down [19:46:38] webservice --backend=kubernetes web start [19:46:52] this command is not starting the web server [19:47:37] !log tools repooling tools-sgeexec-0932, 33, 34 and moving to Ceph [19:47:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:48:03] shrini: what error message are you seeing? [19:48:44] it asks to give a type [19:48:57] and lists all the programming language available [19:49:04] I started with [19:49:05] webservice --backend=kubernetes start [19:49:12] not it started [19:49:28] but still i get 502 bad gateway [19:49:31] https://shrinitools.toolforge.org/indic_ws_stats/ [19:49:34] how to solve this? [19:51:01] bd808: check above [19:52:22] Usually a 502 error means that the proxy server for Toolforge cannot connect to your tool's webservice [19:52:33] * bd808 takes a look at the tool [19:53:17] !log tools depooling tools-sgewebgrid-lighttpd-0911, 12, 13, 16 and moving to Ceph [19:53:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:54:16] ok [20:00:00] shrini: I'm still looking. I have tried a stop/start cycle twice with no change. Not seeing an obvious problem yet [20:00:11] ok [20:00:25] thanks for the updates bd808 [20:01:31] I have file as /data/project/shrinitools/public_html/indic_ws_stats/index.html [20:01:41] should be server normally [20:02:13] I have a hunch about the problem... trying something now [20:03:39] ok [20:03:43] !log tools.shrinitools Cleared stale proxy registration by starting and stopping with --backend=gridengine [20:03:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.shrinitools/SAL [20:03:57] shrini: ^^ It seems to be working now [20:04:53] Your tool had some hidden state in the front proxy from prior attempts to run the webservice on the legacy grid engine platform [20:05:40] awesome bd808 [20:05:54] thanks [20:06:07] The "fix" I used to clear the proxy's state was to do `webservice --backend=gridengine start; webservice --backend=gridengine start`. [20:06:48] this should not happen in normal situations, but it can happen if a webservice is running on the grid engine and that service dies unexpectly [20:08:02] great [20:08:06] thanks [20:13:23] bd808: how to check the crontab entries for my tool? [20:13:40] crontab -l [20:14:13] nothing found [20:14:26] but I remember I added a crontab entry [20:14:33] to scrap a report daily [20:31:49] !log tools Deployed toollabs-webservice v0.74 [20:31:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:35:53] !log tools repooling tools-sgewebgrid-lighttpd-0911, 12, 13, 16 [20:35:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:37:45] !log tools depooling tools-sgewebgrid-generic-0901 and tools-sgewebgrid-lighttpd-0915 [20:37:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:46:02] thanks bd808 [20:46:13] added the cron entries [20:46:22] thanks for the quick help [21:00:48] !log tools repooling tools-sgewebgrid-generic-0901 and tools-sgewebgrid-lighttpd-0915 [21:00:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:00:05] bd808: I am pleased with your work on fleshing out the technical specification for Toolhub. I think it could set a useful precedent for building specialized apps over trying to shoehorn novel experiences into MediaWiki [22:00:57] Thanks hare :) Time will tell if I'm doing the right thing, but it feels pretty good so far [22:01:22] hare: you left me with a pretty good set of data and decisions to start from too :) [22:02:04] There’s a sort of bravery to it because you are intentionally breaking with years of precedent with how the Foundation does things but I think it could be a way forward. Not everything needs to be a MediaWiki extension. [22:07:46] I think we have been moving away from that "everything must be a wiki" mindset for a while on the development side, but I'm not sure that the on-wiki communities are aligned with that yet. [22:09:04] I wok in a strange space between things though, so I may have a skewed perspective. Even before I moved over to work on Cloud things I did not do a lot of MediaWiki development. Mostly I have built things that sit near wikis to help with some workflow.