[00:16:22] bd808: 'dupdet' php tool dies on some queries and doesn't automatically restart [00:16:31] bd808: I'm ignorant about why this is the case, please help [00:16:57] bd808: (I am one of its maintainers, and would be glad to do anything that is possible to prevent it from dying) [12:43:46] !log tools T223148 depool tools-sgewebgrid-generic-0904 [12:43:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:43:49] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [12:49:33] !log tools T223148 reallocating tools-sgewebgrid-generic-0904 to cloudvirt1001 [12:49:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:49:37] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [12:50:36] !log tools T223148 depool tools-sgeexec-0942 [12:50:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:52:34] !log tools T223148 reallocating tools-sgeexec-0942 to cloudvirt1001 [12:52:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:56:15] !log tools T223148 depool tools-worker-1023 [12:56:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:56:18] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [12:58:43] !log tools T223148 reallocating tools-worker-1023 to cloudvirt1001 [12:58:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:03:20] !log tools T223148 repool tools-sgewebgrid-generic-0904 [13:03:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:03:23] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [13:16:43] !log tools T223148 repool tools-sgeexec-0942 [13:16:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:16:46] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [13:57:12] helloo all, i just pushed an update which will have caused keyholder to restart. if you are using keyholder you will need to re-arm it with `sudo keyholder rearm` sorry for the interruption [14:26:59] !log tools.wdvd deployed 7f0953e395 (wb_terms → wbformatentities) [14:27:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdvd/SAL [15:13:11] !log tools T223148 repool tools-worker-1023 [15:13:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:13:15] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [15:23:03] !log tools T223148 depool tools-worker-1009 [15:23:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:23:06] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [15:24:02] !log tools T223148 last SAL entry is bogus, please ignore (depool tools-worker-1009) [15:24:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:24:24] !log tools T223148 depool tools-sgeexec-0909 and reallocate to cloudvirt1002 [15:24:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:26:39] !log tools.stashbot Restarted bot to join new channel (T222755) [15:26:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [15:26:42] T222755: #wikimedia-sre is missing stashbot - https://phabricator.wikimedia.org/T222755 [15:52:35] !log tools T223148 repool tools-sgeexec-0909 [15:52:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:52:38] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [15:56:49] !log tools T223148 depool tools-sgeexec-0911 and reallocate to cloudvirt1003 [15:56:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:36:47] !log tools T223148 repool tools-sgeexec-0911 [16:36:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:36:51] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [16:37:29] !log tools T223148 depool tools-sgeexec-0920 and reallocate to cloudvirt1003 [16:37:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:12:16] !log tools T223148 repool tools-sgeexec-0920 [17:12:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:12:19] T223148: Cloud Services: reallocate workload from rack B5-eqiad - https://phabricator.wikimedia.org/T223148 [17:34:32] !log tools.admin Update to 07a15d8 "Block crawlers from wsexport and its staging" (T222684) [17:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [17:34:35] T222684: Disallow robots from scraping ebooks - https://phabricator.wikimedia.org/T222684 [17:57:10] Hey folks. I'm maintaining a toolforge tool that is running a kubernetes-backed webservice. Recently the tool went down. I believe this is because the python version was changed out from under us and that broke our virtualenv. Does that sound right? Either way, rebuilding our virtualenv on top of the current version of python worked to bring it back online. How do we avoid these kinds of issues in the future? [17:57:25] This is ores-support-checklist FWIW [17:58:19] halfak: I don't think we have rolled out any major version change for the python Kubernetes instances yet... [17:58:38] Some more details here: https://phabricator.wikimedia.org/T222270 [17:58:54] Gotcha bd808. It looks like kubernetes is using python 3.4 [17:59:07] But the bastion we built the venv from is using python 3.5 [17:59:18] yeah, the "python" image is still based on jessie [17:59:44] I wonder if this never really worked in k8s at all. I'm kind of inheriting this project as a maintainer :) [17:59:55] venvs for kubernetes should always be built in from inside a kubernetes shell [17:59:59] But it does work now. So maybe I shouldn't worry about it. [18:00:03] Right. That was my fix :) [18:00:19] its a confusing mess for people honestly [18:00:39] Heh. It is a bit confusing. But honestly I found these docs very helpful: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Using_virtualenv_with_webservice_shell [18:01:01] someday™ we will make more progress on tooling :) [18:01:38] I'm hoping to roll out a python3.5 kuberntes container at the hackathon [18:01:58] Cool! I'll stay tuned. [18:03:44] halfak: if that was an "adopted" tool one thing that could have happened may have been that it used to run on the Trusty grid and then got switched to Kubernetes without having the venv rebuilt. [18:04:03] that bit quite a few people in the Trusty->Stretch grid conversion [18:04:14] Aha! It's quite possible. [18:06:32] * bd808 wonders if it would be easy to detect that kind of mismatch in the `webservice` code [22:35:44] After creating a wikitech account and uploading ssh keys what else needs to happen to have a user be a user of bastion? I signed up and uploaded my ssh but don't see the new user name on https://tools.wmflabs.org/openstack-browser/project/bastion (after 5 minutes). [22:36:04] User name in question is "epenn" [22:36:55] I'm not sure anything should be necessary... [22:37:26] krenair@tools-sgebastion-07:~$ ldapsearch -x member=uid=epenn,ou=people,dc=wikimedia,dc=org dn -LLL [22:37:27] krenair@tools-sgebastion-07:~$ ldapsearch -x member=uid=krenair,ou=people,dc=wikimedia,dc=org dn -LLL | grep bastion [22:37:27] dn: cn=project-bastion,ou=groups,dc=wikimedia,dc=org [22:37:27] krenair@tools-sgebastion-07:~$ [22:38:14] guess I can just add you... [22:38:49] notconfusing, oh, IIRC it only happens when you get added to another project? [22:44:48] notconfusing, which project are you expecting to join? [22:45:26] I'm trying to add my colleague to the wikidumpparse project right now [22:46:41] user:epenn to wikidumpparse, but they cannot ssh through bastion at the moment [22:47:10] okay well you cannot add members to a project you are not an admin of [22:47:18] and you can't be an admin without being a member [22:48:30] Wait are you saying you're *not* Epenn? [22:49:35] i am attempting to assist my colleague, i elided that fact just for brevity before, sorry if you found that misleading [22:50:19] I am an admin of the wikidumpparse project, trying to get my colleague set up on that. [22:51:44] Okay, I think you need to add them as a member of the project before they'll be able to SSH anywhere [22:51:56] do you know how to do that? [22:52:51] In Horizon right? [22:53:08] yes [22:53:48] you log in, select your project at the top, go Project -> Access -> Project Members. Then Add Member [22:54:09] you can choose to enter either CN or UID in there, and optionally tick the admin box [22:54:34] and... hopefully they get bastion access from that. If not I can probably sort that out [22:55:44] Wünderbar! Thank you so much. [22:57:37] looks like it worked: [22:57:37] krenair@bastion-eqiad1-01:~$ ldapsearch -x member=uid=epenn,ou=people,dc=wikimedia,dc=org dn -LLL [22:57:38] dn: cn=project-bastion,ou=groups,dc=wikimedia,dc=org [22:57:38] dn: cn=project-wikidumpparse,ou=groups,dc=wikimedia,dc=org