[12:15:16] <lucaswerkmeister>	 !help my tool (quickcategories) appears to have died yesterday for no apparent reason
[12:15:16] <wm-bot>	 lucaswerkmeister: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[12:15:26] <lucaswerkmeister>	 uwsgi.log says “SIGINT/SIGQUIT received...killing workers...”
[12:15:36] <lucaswerkmeister>	 last modified yesterday 19:31 UTC
[12:16:08] <lucaswerkmeister>	 the same thing seems to have happened to another tool, ordia (I can’t read its uwsgi.log but it was also last modified 19:31)
[12:16:13] <chicocvenancio>	 lucaswerkmeister: restarting it works? 
[12:16:18] <lucaswerkmeister>	 haven’t tried it yet
[12:16:22] <lucaswerkmeister>	 probably
[12:16:24] <lucaswerkmeister>	 but I’m curious why it died :)
[12:16:27] <lucaswerkmeister>	 I couldn’t find any SAL entries
[12:16:35] <chicocvenancio>	 Grid engine? 
[12:16:39] <chicocvenancio>	 Or k8s? 
[12:16:42] <lucaswerkmeister>	 k8s
[12:16:55] * chicocvenancio is curious as well
[12:17:01] <lucaswerkmeister>	 `webservice start` says “your job is already running”
[12:17:24] <lucaswerkmeister>	 `kubectl get pods` shows the pod as “0/1” in the “ready” column, “pending” in “status”
[12:17:32] <chicocvenancio>	 Ohh
[12:17:40] <chicocvenancio>	 Can you describe it? 
[12:17:50] <lucaswerkmeister>	 describe what?
[12:17:58] <chicocvenancio>	 `kubectl describe pod PODNAME`
[12:18:20] <lucaswerkmeister>	 oh, okay
[12:18:24] <lucaswerkmeister>	 https://paste.gnome.org/patdtqecq
[12:18:32] <chicocvenancio>	 That is a question because I'm not sure Toolforge k8s permissions allows that
[12:18:33] * lucaswerkmeister is still mostly a k8s noob, sorry
[12:19:11] <lucaswerkmeister>	 I can try a `webservice restart`
[12:19:15] <chicocvenancio>	 lucaswerkmeister: is that the full output from describing it? 
[12:19:26] <lucaswerkmeister>	 yes
[12:19:38] <lucaswerkmeister>	 sorry, except for a “no events.” line at the bottom
[12:19:43] <chicocvenancio>	 Toolforge is on an old k8s version, so I am a bit unfamiliar with it
[12:19:52] <lucaswerkmeister>	 (which doesn’t have a newline after it, so it bleeds into my prompt, that’s why I didn’t copy it)
[12:20:04] <chicocvenancio>	 Yeah, that was the line I wanted, but populated with the events
[12:20:25] <chicocvenancio>	 Your user probably doesn't have access to the events
[12:20:31] <lucaswerkmeister>	 ah okay
[12:20:42] <chicocvenancio>	 There is definitely an event if it is in pending mode
[12:20:49] <chicocvenancio>	 Pending status
[12:21:07] <chicocvenancio>	 Or at least there should be one
[12:21:27] <chicocvenancio>	 Webservice restart is your best bet to get it up
[12:21:33] <lucaswerkmeister>	 trying that
[12:21:40] <lucaswerkmeister>	 “your job is not running, starting…”
[12:22:32] <chicocvenancio>	 You can watch it with `kubectl get pods --watch`
[12:22:46] <chicocvenancio>	 To see if it is properly scheduled and starts
[12:23:11] <lucaswerkmeister>	 ah, nice
[12:23:20] <lucaswerkmeister>	 `kubectl get pods` still shows the same pod, though (-xqip5)
[12:23:29] <lucaswerkmeister>	 same age, too
[12:23:43] <chicocvenancio>	 That's not a good sign
[12:23:50] <chicocvenancio>	 You can probably delete it
[12:23:52] <lucaswerkmeister>	 !log tools.quickcategories kubectl delete pod quickcategories-654583560-xqip5
[12:23:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[12:23:58] <chicocvenancio>	 Yeah
[12:23:59] <chicocvenancio>	 That
[12:24:06] <lucaswerkmeister>	 might as well start with the per-tool SALs, haven’t played with those before :)
[12:25:06] <lucaswerkmeister>	 okay, that started a new pod, which is also pending now
[12:25:13] <lucaswerkmeister>	 no output from `kubectl logs`
[12:25:58] <chicocvenancio>	 If it stays pending then there might be something wrong with k8s in Toolforge 
[12:26:20] <lucaswerkmeister>	 seems so
[12:27:06] <lucaswerkmeister>	 I tried `kubectl exec -it … bash` but it says the pod does not have a host assigned
[12:27:16] <lucaswerkmeister>	 (I guess that’s what “pending” means? the container isn’t running yet?)
[12:27:27] <chicocvenancio>	  Yes
[12:27:50] <chicocvenancio>	 The event would tell you why it doesn't have a node yet
[12:28:13] <chicocvenancio>	 But I think users don't have that permission, root might
[12:28:44] <lucaswerkmeister>	 okay
[12:29:47] <lucaswerkmeister>	 none of the recent Phabricator tasks sound related, I guess I’ll create one
[12:30:22] <chicocvenancio>	 It is a Sunday, and very early on the timezone of the oncall person for WMCS. So yeah, phab task and waiting seems sensible 
[12:42:08] <lucaswerkmeister>	 https://phabricator.wikimedia.org/T220912
[12:42:12] <lucaswerkmeister>	 anyways, thanks for your help so far :)
[12:42:21] * lucaswerkmeister starts making lunch
[16:07:09] <bstorm_>	 Hrm. Sounds like too many k8s workers are down
[16:20:25] <fnielsen>	 I am wondering whether https://phabricator.wikimedia.org/T220912 is a Python3.4/Python3.5 problem?
[16:23:11] <andrewbogott>	 !log tools moved all tools-worker nodes off of cloudvirt1015 and uncordoned them
[16:23:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:27:05] <fnielsen>	 I have installed python3.5 in a Python virtualenv on Toolforge, but Kubernetes seems to use python3.4. It is unclear to me how I can construct a Python3.4 virtualenv on the tool's Toolforge account. 
[16:28:14] <chicocvenancio>	 fnielsen: you need to use a k8s shell to create the veenv
[16:28:55] <chicocvenancio>	 From the tool account `webservice backend=kubernetes python shell`
[16:31:32] <fnielsen>	 ok
[16:38:34] <fnielsen>	 I have gotten into a shell with "webservice --backend=kubernetes python shell", but where do I find virtualenv or similar?
[16:40:52] <fnielsen>	 "python3 /usr/lib/python3/dist-packages/virtualenv.py --python=python3.4 venv3.4" seems to create a 3.4 virtualenv
[17:04:33] <chicocvenancio>	 fnielsen: https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool#Create_a_Python_virtual_environment_for_the_application's_external_library_dependencies
[17:04:49] <chicocvenancio>	 There are several ways to do it. 
[17:05:33] <chicocvenancio>	 Toolforge k8s expects to be in the $HOME/www/python/venv folder