[00:00:26] something like https://gerrit.wikimedia.org/r/#/c/431285/ [00:00:57] cool. I can make a patch like that and get someone to merge it for us [00:01:50] ok [00:02:51] it should still be modules/role/manifests/toollabs/k8s/master.pp [00:05:01] * bd808 hopes an.drew is still around [00:08:51] !log tools running /usr/local/bin/git-sync-upstream on tools-puppetmaster-01 to speed puppet changes up [00:08:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:09:32] (why is it taking so long) [00:10:50] zhuyifei1999_: we think tools-puppetmaster-01 is a bit overloaded right now but we don't have a clear reason why other than lots of new clients attached to it [00:11:20] yeah [00:11:54] load is like 13 with 2 cpus. so that's 7x load [00:12:42] !log tools forcing puppet run on tools-k8s-master-01 [00:12:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:12:54] we are basically running a double sized project for tools right now with the parallel Trusty and Stretch grids [00:13:27] sounds like a time to buff the puppet master :P [00:14:15] or shutdown Trusty grid nodes ;) [00:26:27] I gtg now. didn't expect puppet to need so much time [00:26:49] zhuyifei1999_: what parts are left? [00:26:49] is there a way to live-change the config? or somehow speed it up [00:26:59] Info: Applying configuration version '1550708043' [00:27:19] and no output after that [00:27:23] yeah, the puppet run is going to take a while [00:27:45] I meant what else is needed to roll out the new webservice build once the k8s master is happy? [00:27:57] I don't think anything else [00:28:06] just test if everything works [00:28:10] I thinl [00:28:13] *think [00:29:14] cool. I might do the mean thing of restarting the puppetmaster process. If I'm fast enough I might be able to get the one we want right now to finish before the horde of vms attack again [00:30:45] ugh. puppet is running on the puppetmaster itself right now. I don't want to interrupt that [01:39:46] bd808: thanks for taking care of it [07:36:12] !log admin deleted wmcs-nfs project [07:36:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:42:17] !log admin created project cloudstore [07:42:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:09:52] !log admin restarted uwsgi-labspuppetbackend.service on labpuppetmaster1001 [09:09:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:22:20] gtirloni: I think its the right moment to create a WMCS infra naming conventions document :-) [09:22:41] !log paws upgraded and rebooted paws-proxy-02 [09:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [09:22:50] !log quarry updated and rebooted all servers (debian 9.8) [09:22:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [09:24:15] arturo: agreed. I've heard we don't like to have "wmcs" on things because the team name could change, or the service could be migrated to another team, etc.. so that's why I went with "cloudstore". My preference would be for a everything to be in a single project though... any problems with that? it might be easier with the security groups and everything (but it also might be less secure) [09:24:45] arturo: when I created "wmcs-nfs" that was the immediate feedback I got "wmcs- not good but okay" [09:26:42] I would use a common string for all projects relates to the infra itself [09:26:54] be it testing, BD, or storage [09:27:24] I dont mind unsing wmcs- or svc. or svc- or whatever-infra [09:27:32] got it [09:27:39] (we already have cloudinfra BTW) [09:27:44] we have clouddb-services (dunno what the -services is for) [09:27:47] yeah [09:28:12] cloudinfra, clouddb-services, cloudstore.. it's clouds all the way :) [09:28:15] we need that document I think, I would welcome a bit less naming caos [09:28:20] +1 [09:28:22] :-) [09:29:28] !log quarry applied CSP change T214637 [09:29:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [09:29:32] T214637: Setup CSP http header - https://phabricator.wikimedia.org/T214637 [09:59:12] !log tools upgraded all packages in all stretch nodes [09:59:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:29:37] !log ores staging ores-wmflabs-deploy:ea153f6 [15:29:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:29:04] Hi everybody, what is the way to update a package on toolforge? [18:29:19] what kind o package? [18:30:08] PyMySQL [18:30:11] on Stretch [18:30:21] on jsub [18:30:32] (gridengine probably?) [18:30:42] venv [18:30:51] I see [18:31:16] Dvorapa: the system packages are system packages, but as chicocvenancio mentions you can create a virutalenv for your tool [18:31:57] btw why is the system package outdated? (or what is its source?) [18:32:16] a better explanation of "system packages are system packages" is that we do not install thise using pip. They are installed via deb packages maintained upstream by the Debian project [18:32:39] So it is from Stretch repositories, am I right? [18:32:45] correct [18:33:40] https://packages.debian.org/stretch/python-pymysql and https://packages.debian.org/stretch/python3-pymysql [18:33:59] And just to be sure there isn't any system package installed from pip on forge, am I right? [18:35:39] I see, both 2 years old (typical for stable distros like Deb/Ubu) [18:44:49] Sooo close! The issue I'm facing was fixed in v0.7.11 and Stretch has v0.7.10 (today, 2 years later pip suggests 0.9.3 btw) [18:45:26] there is no system package installed from pip, correct [18:48:08] bd808: is it a possibility that we backport the package from buster to stretch? [18:48:56] technically it is possible, but practically it is difficult [18:49:27] it doesn't seem to have any version dependency conflicts [18:49:30] I would really rather that Python tools use venv and manage their own dependencies [18:50:03] every thing we do that is custom for Toolforge locks us into a long term support arrangement [18:50:34] sigh [18:50:35] so if we backport a deb, we need to keep tracking the upstream for security fixes, etc until we drop our local backport [18:51:38] or is it possible to convince upstream to include a patch? [18:52:28] I've never worked with debian packagers before, so... [18:54:53] is there a ticket for whatever the problem folks are seeing here? I don't think I have the context to understand if this is "pymysql is busted for all" or "pymysql has a bug that hurts N tools" [18:54:59] I reported a bug on launchpad, hopefully they'll notice it. [18:55:13] https://phabricator.wikimedia.org/T216741 [18:55:47] connection.close can crash if the remote disconnects first [18:56:37] and 'MySQL server has gone away' is really common here [18:57:10] Dvorapa: launchpad.. you mean the ubuntu one? [18:58:38] yeah [18:59:20] well, we use debian, not ubuntu [18:59:48] let me see if I can use $ reportbug [19:00:52] But used Ubuntu recently? What's the Debian issue tracker? [19:02:13] https://www.irccloud.com/pastebin/jNcjDUT4/ [19:02:28] yes, trusty is ubuntu [19:02:35] https://www.debian.org/Bugs/ [19:03:07] zhuyifei1999_: getting the buster package into stretch-backports upstream might be the sane resolution. I don't know the request process for that myself, but I do have a Debian Developer on my team we could ask [19:03:18] ok [19:09:32] btw is there a way for pywikibot to provide predefined, preinstalled and up-to-date env for forge users (using pip)? I know pywikibot has some entries in https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/toollabs/manifests/exec_environ.pp but they can be outdated as well [19:09:57] Like some shared config file, which loaded into newly created wvenv will install all the prerequisites [19:10:40] I could try that, yes, for the shared install [19:10:42] Or even predefined,preinstalled venv users can just "open" [19:11:09] predefined,preinstalled venv => will be chaos [19:11:14] I see [19:11:56] So just the config file maybe. Well only if it does not require to update as pywikibot prerequisites update (quite often) [19:12:26] that's functionally the same problem as custom system packages. Multiple people sharing the same libraries *always* ends up with someone being made that they are updated too often and someone else being made that they are updated too slowly [19:12:45] s/made/mad/ [19:13:50] Dvorapa: pip install pywikibot, basically. [19:14:17] that installs pywikibot, and all required dependencies [19:14:36] but doesn't install pwb.py et al, so for that a local checkout is still required [19:15:36] valhallasw`cloud hi, i have a change to add wip support in wikibugs :) [19:15:40] this time it should work [19:15:57] https://gerrit.wikimedia.org/r/#/c/labs/tools/wikibugs2/+/491868/ [19:16:38] actually, pywikibot even bundles a requirements.txt [19:18:12] so this should work: virtualenv -m python3 pywikibot-venv && source pywikibot-venv/bin/activate && pip install -r /shared/pywikibot/core/requirements.txt && python /shared/pywikibot/core/pwb.py version [19:19:06] paladox: could you add some tests? [19:19:12] hmm [19:19:35] see https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/wikibugs2/+/b2cd7acfeb0f5ae1b29815282e98fe7697057d1c/tests/data/stream_events/ / https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/wikibugs2/+/b2cd7acfeb0f5ae1b29815282e98fe7697057d1c/tests/test_json_events.py for examples [19:21:02] Dvorapa: https://github.com/pywikibot/Pywikibot-nightly-creator/blob/master/nightly#L25 myparserfromhell is included this way [19:21:28] I could send a patch to add pymysql this way as well [19:21:32] cc legoktm [19:21:45] @seen legoktm [19:21:45] zhuyifei1999_: legoktm is in here, right now [19:22:06] (oops, I didn't see the name) [19:22:26] zhuyifei1999_: iirc mwparserfromhell is included because it's the only non-optional dependency [19:22:33] but I wrote that code a _long_ time ago [19:23:33] paladox: it would also be good to have end-to-end tests for gerrit changes, i.e., from event stream json to actual irc message [19:23:43] oh [19:23:49] * paladox has no idea how to do that :) [19:23:50] a lot of people uses pymysql/mysqldb/oursql [19:24:57] valhalla: pip install pywikibot is not recommended practise by toolforge I though? [19:25:59] Dvorapa: it depends on what you want to do. If you want to use it as library, it's a good option, if you want to use pwb.py, not so much. Given that there's a requirements.txt, using that is a much better option. [19:27:30] paladox: the stream-event tests are the most important; for those you can take a look at the other tests. [19:27:40] ok [19:27:42] paladox: has the json format changed? in that case it would be good to add tests for both the old and the new format [19:27:43] valhalla: I see. mwparser seems not to be a required dependency anymore [19:28:15] valhallasw`cloud not much, i doin't think it uses "change" and not patchSet. [19:28:45] paladox: 'almost the same but not quite' is a good way to crash software ;-) [19:29:25] valhalla: but still it is insluded with pywikibot there. And required requests are not included with pywikibot. Quite a mess [19:30:07] valhallasw`cloud https://gerrit-review.googlesource.com/c/gerrit/+/191490 [19:30:15] I see, requests are installed from Stretch repositories and mwparser is missing there [19:30:52] heh i was wrong, it uses patchSet's, looks like the doc is wrong :( [19:31:10] requests is one of the libraries that pywikibot cannot run without. hell is only need for s few scripts and some testlib stuffs [19:31:14] *textlib [19:31:45] (same case for pymysql) [19:32:10] though still uses "change" [19:33:19] Dvorapa: https://bpaste.net/show/f891eec5ee48 would be my suggested method [19:33:58] that installs PyMySQL-0.9.3 [19:34:42] valhallasw`cloud https://gerrit-review.googlesource.com/c/gerrit/+/215132 [19:35:52] paladox: just go through the WIP workflow on the current gerrit install and the latest gerrit and log the result of gerrit stream-events? [19:36:34] ok [19:40:13] valhallasw`cloud: Seems cool to me. What's in the source if I can know? (just of my interest) [19:41:50] Dvorapa: the `source` command executes all the commands in the parameter, so `source pywikibot-venv/bin/activate` is equivalent to running all the commands in `pywikibot-venv/bin/activate` [19:42:10] Dvorapa: the `activate` script fiddles with the path, to make sure that `python` actually runs the python in the virtualenv [19:43:02] valhallasw`cloud https://phabricator.wikimedia.org/P8117 and https://phabricator.wikimedia.org/P8118 and https://phabricator.wikimedia.org/P8119 [19:43:37] Dvorapa: instead of the `activate` stuff, you could also directly run `pywikibot-venv/bin/pip` and `pywikibot-venv/bin/python` [19:44:12] paladox: (y) [19:44:20] (y)? [19:45:47] 👍 [19:47:03] I see [19:47:16] Thank you [19:47:40] Dvorapa: the whole virtualenv business is a bit overwhelming at first, but it is quite a nice way to handle dependencies [19:48:38] (am I the only one to use `~/.local` as venvs?) [19:49:11] (venv for a tool) [19:49:41] For me until now all stuff went into ~/.profile [19:50:18] zhuyifei1999_: ah, that's where pip install --local goes? [19:50:18] and I still have no idea, how to run venv in cron [19:51:14] valhallasw`cloud: I think so, but I have .local for everything, from venvs to compiled stuffs [19:51:39] Dvorapa: easiest way is to refer to the python in the virtualenv. So if you had e.g. /usr/bin/jsub python /path/to/script; you can change it to /usr/bin/jsub /path/to/virtualenv/bin/python /path/to/script [19:51:43] like updated ffmpeg... for one tool that needs it [19:53:16] I see [19:57:08] valhallasw`cloud the test for test_post_merge_build dosen't look right [19:57:17] "repo" is not defined in the json [19:58:48] paladox: the json is the input to process_events, the test verifies the output of process_events [19:58:56] ah ok [19:59:07] that output is then used to build an IRC message [20:00:07] valhallasw`cloud do i do assert messages[0] == {} when it contains wip: true in the json? [20:00:25] since it should not be outputting anything to irc unless it is not a wip. [20:01:13] paladox: the messages array should then be empty (messages[0] == {} implies there is a message, but without fields defined) [20:01:21] so somethinl ike assert len(messages) == 0 [20:01:44] ah ok, so i doin't need to do assert messages[0] == {} too [20:02:11] no, if len(messages) == 0, trying to access the first element in that (empty) list will throw an Exception [20:04:19] valhallasw`cloud https://gerrit.wikimedia.org/r/#/c/labs/tools/wikibugs2/+/491868/6/tests/test_json_events.py [20:06:57] paladox: yep, something like that [20:07:57] bd808: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot is the best Toolforge manual from all Toolforge manuals. I just misses some stuff about how to create a tool and maybe a little more detail about how to run webservice (but it is not in the scope of the page) [20:13:18] paladox: I'll try to review it tomorrow. [20:13:19] am I using the wrong bastion to connect to cloud instances (something.eqiad.wmflabs)? I've got bastion-eqiad.wmflabs.org and I was trying bastion-eqiad1-01.wmflabs.org [20:25:50] valhallasw`cloud ok, thanks! [21:27:02] milimetric: 'bastion.wmflabs.org' is easy to remember [21:28:48] thanks andrewbogott, I tried this ProxyJump thing in the meantime and found out that my openssh is ancient [23:56:37] so apparently running webservice status from the trusty or stretch bastion is the exact same thing [23:56:56] that was confusing, would be worth mentioning in the migration guide [23:57:11] so how do I check if a webservice needs to be migrated still?