[00:05:29] Hi i get this error when trying to use role::labs::mediawiki_vagrant, anyone have an idea why? [00:05:32] https://www.irccloud.com/pastebin/qY4hOqIr/ [03:00:22] fixed with a reboot [04:11:22] !log library-upgrader added SSH key to ssh-agent [04:11:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [05:01:46] !log library-upgrader sudo systemctl enable libup-push [05:01:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [09:18:23] !log admin draining cloudvirt1033 for T275753 [09:18:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:01:40] !log tools.zppixbot clean a bunch of junk up [10:01:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [10:03:27] !log tools.zppixbot-test cleanup used stuff [10:03:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [10:04:48] dcaro: doubt it made much of a dent but I cleaned 2 tools I maintain up of stuff not in use [10:05:39] RhinosF1: thanks! Much appreciated :) [10:05:46] Np :) [10:25:24] !log admin rebooting cloudvirt1033 for T275753 [10:25:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:26:29] !log admin draining cloudvirt1034 for T275753 [10:26:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:47:47] !log admin rebooting cloudvirt1034 for T275753 [10:47:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:49:18] !log admin rebooting cloudvirt1035 for T275753 [10:49:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:01:41] !log tools.stewardbots restart both bots as they had disconnected from freenode [11:01:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [12:22:27] !log tools stop tools-redis-1004 to ease draining of cloudvirt1035 [12:22:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:25:07] !log admin rebooting cloudvirt1035 for T275753 [12:25:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:30:37] !log tools started tools-redis-1004 again [12:30:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:30:59] !log admin draining cloudvirt1036 for T275753 [12:31:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:22:52] !log traffic briefly rebooting traffic-cache-atsupload-buster because reboot of the hypervisor cloudvirt1036 [16:22:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Traffic/SAL [16:23:22] !log admin rebooting cloudvirt1036 for T275753 [16:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:10:14] is there a formal process to get dns zones delegated to projects? I'd like to have beta.wmcloud.org available for deployment-prep and possibly an svc zone as well [21:11:15] !log tools.wikibugs restarted wb2-phab process [21:11:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [21:11:36] Hello andrewbogott, harshinee and I have been working to deploy a ML model on toolforge via a flask API. The API works on our local systems but doesn't work on toolforge. the name of the tool is image-content-filtration. Any pointers you could give us? [21:13:40] can you expand on what's not working? [21:13:46] chtnnh: not staff but have you checked logs [21:15:39] okay basically when we start the webservice and curl -X POST -F @image=filename "https://image-content-filtration.toolforge.org/predict" there is no response for a solid 5 mins [21:15:58] Then times out? [21:16:00] and then we get a 502 bad gateway (sometimes) [21:16:15] Is there an error.log file in the tool homedir [21:16:21] just checking [21:17:21] chtnnh: I don't know a lot about making an api on toolforge but are you launching it with k8s or the grid? [21:17:46] chtnnh: I think you created your virtualenv on the bastion and not in a k8s pod? see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#Creating_a_virtual_environment [21:18:34] also if you look in ~/uwsgi.log you can see its complaining of a syntax error now [21:19:12] andrewbogott, im sorry i dont follow [21:19:17] the earlier ModuleNotFoundError errors seem to be related to what I mentioned earlier about the venv creation [21:19:23] andrewbogott: it's running on k8s [21:19:28] legoktm, we created venv in both the tool and the user [21:20:08] did you follow the steps using the link I provided? [21:20:23] just checking that [21:21:02] !log admin replacing 'observer' role with 'reader' role in eqiad1 [21:21:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:22:37] Majavah: we don't have anything formal for DNS delegations, but that sounds like a reasonable one. Make a phag task and tag it with #wmcs-kanban and we can probably get it setup [21:23:09] legoktm, we did follow those instructions [21:23:43] bd808: sure, thanks [21:23:57] ok [21:24:03] you still have a syntax error in app.py [21:24:09] File "/mnt/nfs/labstore-secondary-tools-project/image-content-filtration/www/python/src/app.py", line 34 [21:24:09] label = "Unsafe" if np.argmax(preds) == 0 else "Safe" [21:24:14] TabError: inconsistent use of tabs and spaces in indentation [21:24:53] shell editing is unsafe :/ [21:24:55] chtnnh: legoktm is giving you the right advice. your ~/www/python/venv was built with python3.5 and you are trying to us it from a python3.7 Kubernetes container [21:25:06] ohhhhhh [21:25:39] so if we make a new python venv with 3.7 it should work fine? [21:25:43] You need to build the venv from inside a `webservice --backend=kubernetes python3.7 shell` session [21:25:56] alright [21:25:59] trying that [21:26:50] chtnnh: you should be able to use autopep8 to fix TabErrors but having CI and tracking changes in something like git is a very good recommendation [21:28:17] thank you for the suggestions RhinosF1 will keep them in mind [21:29:09] legoktm: (gentle nerd snipe attempt) can you think of simple way we could add a check in `webservice` for mis-matched venv and runtime versions? [21:29:31] omg that would be so cool bd808 :D [21:29:46] This happens far too often in large part due to the grid/k8s split [21:30:33] invoke `venv/bin/python --version` and it doesn't fail then raise an error? :p probably provides a better error condition than checking if it doesn't fail in the container [21:30:36] the "real" fix will be the next-generation webservice tooling that folks are working on, but that won't be general release for quite a while [21:30:45] * legoktm checks to see if that will actually work [21:31:41] hm, no [21:32:48] ok, ./www/python/venv/bin/pip should fail [21:34:02] a more reliable way might be to look in ./www/python/venv/bin/ and see if pip3.X exists, for whatever image you're trying to use [21:34:03] bd808: https://phabricator.wikimedia.org/T276624 [21:34:41] * legoktm writes up a task [21:35:59] thanks Majavah. is this blocking you right now? Mostly asking so I know if I should fix this right now or if I can let it sit for a day or two without causing you troubles [21:37:15] * bd808 is having brain based preemptive multitasking issues :) [21:37:16] bd808: no, not blocking at all [21:38:08] Cool. I'll try to get back into my planned Friday coding then. :) [21:39:57] T276626 [21:39:57] T276626: Toolforge: Automatically identify when people are trying to launch a Python webservice with a venv created using the wrong Python version - https://phabricator.wikimedia.org/T276626 [21:40:49] !log admin replacing 'observer' role with 'reader' role in eqiad1 T276018 [21:40:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:40:53] T276018: Investigate new roles and policies in openstack train - https://phabricator.wikimedia.org/T276018 [21:41:02] thanks legoktm! [21:42:31] successfully sniped :) though I'll leave the implementation for someone else [21:48:42] bd808, made the venv as you suggested [21:48:49] now trying to fix the indent errors [21:48:52] and should hopefully run [22:05:19] well we still seem to be facing some issues [22:05:38] but its getting late here now, and my mind is fried [22:05:46] we will be bothering you on monday [22:06:16] thank you bd808 legoktm andrewbogott RhinosF1 [22:06:30] o/ [22:06:57] np :) [22:24:29] !log tools.lexeme-forms deployed 109f22a415 (Czech verbs update) [22:24:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [23:13:10] Curious, using the mediawiki vagrant role, do I just edit ls.php and cs.php using like vim or is there something special i have to do [23:41:23] Zppix: look at the README in the settings.d directory that MediaWiki-Vagrant provisions. MediaWiki-Vagrant has a very elaborate settings management setup to handle the wiki farm [23:45:07] !log codesearch restarting for new shouthow profile (T253597) [23:45:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [23:45:11] T253597: Add ShoutHow to Codesearch - https://phabricator.wikimedia.org/T253597 [23:50:07] bd808: ty