[05:43:21] Square root principal [05:43:44] Identify matrix [11:24:45] hello! I have my bot source hosted on Bitbucket. I want it to automatically update and build on labs whenever I push my changes. I tried to make a cron job for that, but I cannot make it work: 1) there's no `hg` on grid machines; 2) so I tried to use `jlocal` and I don't get any output from that. I tried running jlocal manually and got '-bash: jlocal: command not found'. what is the proper way? [11:26:53] !help sorry forgot to use !help [11:26:54] kf8: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [11:27:03] kf8: I think hg should /probably/ be installed, can you create a ticket for that? [11:27:38] jlocal is just on the cron machines, but you shouldn't really use it for this [11:28:18] valhallasw`cloud: actually, you are right! git is installed on the grid, so it makes sense [11:28:54] but if you want it to pull immediately after pushing, it's probably better to either have a script on your computer (which pushes to bitbucket, logs in to tool labs, then pulls there) [11:29:23] or to use a webhook, where bitbucket talks to tool labs directly. We use that for wikibugs' auto-update -- let me see if I can find the code [11:31:35] kf8: see /data/project/wikibugs/public_html/pull.php. [11:34:20] valhallasw`cloud: thanks [11:44:27] harej: for T197997, are you sure the error log is not from a prior run? (i.e. if the log starts empty will the error message still generate?) [11:44:28] T197997: Script dependency appears to not exist when script is run in job grid even though dependency is otherwise installed - https://phabricator.wikimedia.org/T197997 [11:45:09] I never encountered this error before. venv always worked [11:45:57] valhallasw`cloud: looks like I would still need hg to be installed though? or is it on the web grid already? I don't know how to check [11:46:10] Krenair: yes, would need to be installed [11:46:48] eh, kf8 [11:47:33] kf8: could you create a subtask of https://phabricator.wikimedia.org/T140110 with the request to install hg + what you're trying to do? [11:47:52] doesn't seem to be installed on grid [11:48:02] (I mean hg) [11:49:27] valhallasw`cloud: I will, thanks! [11:50:08] valhallasw`cloud: should't it be https://phabricator.wikimedia.org/T55704 ? [11:50:32] argh, yes! kf8: please use the link zhuyifei1999_ posted :-) [11:51:56] ok, thanks! [11:53:39] sorry, how do I create a subtask?.. [11:53:57] nvm, found it [11:57:54] https://github.com/wikimedia/puppet/blob/3adba829315e01ea4ece3b803c73ab45b8b649d3/modules/toollabs/manifests/dev_environ.pp#L83 [11:58:03] ^ why it's installed on bastions [12:02:20] the bot no longer posts new phab tasks? [12:02:23] https://phabricator.wikimedia.org/T198008 [12:04:41] zhuyifei1999_: reasonably sure; I deleted the log files beforehand. But I can always try again (when I have my laptop out) [12:05:01] I reaaaaally wish those logs had time stamps [12:07:00] kf8: it's in -feed [12:07:14] see the channel 'topic' [12:07:30] harej: can I test it out myself? [12:08:22] zhuyifei1999_: I see, thanks [12:08:35] zhuyifei1999_: worth a shot, I think. [12:08:41] k [12:09:22] Does the fact that it’s part of a Kubernetes backed web service make a difference? [12:09:42] hmm [12:10:12] if it works on bastion it should work on grid [12:13:18] harej: yeah I can reproduce it [12:13:39] (after become mediaviews-api and executing the same commands [12:13:50] Woohoo! I’m not crazy. And have definitely had this problem before. [13:46:46] harej: did you do something "esoteric" to that venv? [13:48:19] chicocvenancio: should just be a run of the mill venv [13:49:40] I think https://phabricator.wikimedia.org/T197997#4309273 is the cause [13:49:41] ahhh [13:49:48] zhuyifei1999_ ftw [13:50:40] harej: could you 'un-symlink' and see if it makes a difference? [13:51:07] > Grid engine 'normalizes' [13:51:08] ughhh [13:51:24] Dang computers, I specified an absolute path for a reason! [13:51:36] harej: I wonder if calling `python` would work [13:51:57] (maybe two symlinks is too much for grid engine to follow, lol [13:52:14] does python3 symlink to python? [13:52:18] harej: I hate grid engine for lots of reasons. this is yet another one [13:52:23] `/data/project/mediaviews-api/www/python/venv/bin/python` that is [13:52:34] I'm used to calling it python3 out of habit since python is usually python2 [13:52:45] zhuyifei1999_: I have lots of ideas for replacing it :) [13:53:12] * harej has hated Grid Engine for literally years [13:53:44] I think bd.808 already has a replacement in mind? [13:54:51] he and I have talked a lot about redhat openshift [13:55:04] * chicocvenancio likes a lot were this is going [13:55:41] one of my goals for this quarter is to map out what people use cloud services for so that we can come up with a suite of options depending on your goal [13:55:53] sounds fun [13:57:17] Unfortunately none of this will happen quickly but hopefully it will be done "right" for some definition of right [13:59:54] the two symlinks theory doesn't work, calling python or python3 will have the same arg0 in grid engine [14:00:34] I'm wondering why this isn't a more common occurence [14:01:01] Yeah, presumably absolute paths aren't normalized every time or there'd be riots [14:01:31] it's symlinks being followed before passed to the final script run [14:02:55] harej: could you have created that veenv inside a kubernetes pod? [14:03:26] I think I did [14:03:31] that is the issue [14:03:45] The interaction of kubernetes and venv are all mysterious to me [14:03:52] venvs created in k8s don't work with the grid [14:03:57] and vice-versa [14:04:26] to use one in the grid you need to create it on the bastion (or on grid itself, but that is just masochism) [14:06:07] I'm pretty sure it's just those venvs created on grid uses an old method of having copies of python executable instead of symlinks [14:06:54] well, indeeed [14:08:02] there might also be a problem with the discrepancy between python stdlib versions [14:08:26] basically, a python 3.5 venv on 3.6 will get confused because of changes in the stdlib [14:10:57] Okay so I got into this mess by reciting shell commands the way a priest mumbles Latin expressions. How do I get out of it? [14:11:14] valhallasw`cloud: huh, got python 3.4.3 from grid venv and 3.4.2 from k8s [14:11:22] not sure that is enough to cause troubles [14:11:48] harej: I like to use `mkvirtualenv` [14:12:06] chicocvenancio: I'm not entirely sure where the threshold is; I think patch level might indeed be OK [14:12:07] that's a thing?? [14:12:38] yes, it automates the creation a bit [14:13:33] `mkvirtualenv -p /usr/bin/pynthon3 venv` will leave you with a grid-compatible venv on ~/.virtualenvs/venv [14:14:31] you can pass it a requirements files and it will install it for you as well, use `-r requirements` for that [14:15:03] further docs http://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html#mkvirtualenv [14:19:34] harej: so, reciting this should get you there: https://www.irccloud.com/pastebin/PLt6ADdn/ [14:21:35] ^from the bastion, to be clar [14:21:38] *clear [14:37:37] -m 1G to add more memory, right? [14:37:58] is it needed for this? [14:38:17] I just had an EOM, so probably [14:38:28] but it would be `-mem` [14:38:55] ah, thank you [14:55:47] zhuyifei1999_: If you get some time, could you please help me resolve this - https://pastebin.com/prMYRmLB [14:56:06] what's that? [14:56:35] I am doing `pip install kubernetes` [15:00:09] uh [15:00:59] could you try version 0.3.0? [15:01:00] Neha16: this is in toolsbeta? [15:01:10] chicocvenancio: Yes [15:01:31] zhuyifei1999_: Which package? [15:01:33] you can install the package exempting the reinstall of urllib [15:03:38] chicocvenancio: Doesn't this require upgrade of urllib3? And there is ecdsa too which needs to be upgraded [15:04:35] if it does, you need to remove urlib3 manually before the pip command see https://github.com/pypa/pip/issues/5247 [15:05:31] chicocvenancio: if she do that, the webservice command will get uninstalled [15:05:42] Neha16: kubernetes [15:05:49] it should break temporarily [15:06:03] but manually uninstalling urlib3 doesn't mean pip [15:06:19] yes, apt [15:06:24] in fact, pip cannot uninstall urllib3 in that state [15:06:37] the webservice command has a dependency on urllib3 [15:06:47] in apt? [15:09:34] seems like a task is needed to think this through, what is happening there is that something (apt?) is using distutils to install urllib3, and distutils doesn't play nice and leave the metadata of the install [15:10:36] urllib3 is installed as a deb package [15:10:52] like all other global packages [15:11:04] and debs get installed into dist-packages [15:11:13] (distutils) [15:12:07] yes, that dependency is an apt/deb dependency [15:16:56] for development, it shouldn't be a problem to manually (`rm /usr/lib/python2.7/dist-packages/urllib3`) but there that won't fly in prod [15:17:18] in the end it probably means us building a newer deb package for urllib3 [15:18:33] also, removing things like that will leave things in a inconsistent state, do record what you did [15:19:27] Shouldn't this be breaking webservice ? [15:19:34] (I'm assuming that we actually need the `kubernetes` 0.6.0 package, yife suggestion to use version 0.3.0 should be used if possible) [15:20:21] Neha16: it might, if there is any removed or altered behavior in urllib between the current version and the one installed [15:20:44] you could use a veenv to test things out before going nuclear on dist-packages [15:20:49] *venv [15:23:25] (yeah, if we backport the package from stretch it's 0.3.0) [15:24:06] it requires `urllib3 >=1.19` [15:24:30] I think last time I downgraded pip's version to 8 that handled distutils. [15:24:47] that is equivalent to what I'm suggesting [15:25:11] manually removing things and leaving things in an inconsistent state [15:25:23] (only pip will do that for you) [15:27:44] Neha16: which, again, I think is fine for toolsbeta during development [15:28:08] manual removal will still be inconsistent. apt will still insist and different version [15:28:27] *and =>a [15:28:32] Probably `pip install kuberenetes == 3.0.0` will be fine for development? [15:28:40] 0.3.0 [15:28:43] not 3.0.0 [15:28:56] that doesn't exist in pip [15:29:00] huh [15:32:54] hmm python-k8sclient might be a wrong package [15:32:57] might be https://github.com/openstack/python-k8sclient [15:33:54] kubernetes is https://github.com/kubernetes-client/python [15:34:20] ohh, you mean apt [15:34:33] yes [15:34:43] https://phabricator.wikimedia.org/T197930 [15:35:02] yes, got a phab notification here and connected the dots [15:35:04] apt search kubernetes isn't giving many results [15:38:55] we can use `kubernetes==1.0.0`actually, but that doesn't solve the problem, still needs `urllib3>=1.19.1` [15:39:26] so yeah, T197930 is the solution, but perhaps use a venv to develop in the meantime [15:39:26] T197930: Build or backport .deb for kubernetes python client for Jessie and Trusty - https://phabricator.wikimedia.org/T197930 [15:40:04] Neha16: venv is probably the best solution if we're already working in making a deb package [15:46:57] chicocvenancio: Well then I'll go nuclear in venv. Thanks for your help! [15:59:13] zhuyifei1999_: are we pinning the urllib3 version? [16:00:57] no [16:01:16] but trusty won't provide latest versions [16:01:38] ahhh, trusty [16:01:51] * chicocvenancio forgets about that [16:01:58] 1.7.1 [16:02:10] that's the one it provides [16:03:32] so I was right on my first comment and wrong to strike that on the task [16:07:27] seems jessie won't give as a new enough version either