[07:37:58] <Neha16>	 andrewbogott: Hi, around ?
[07:40:07] <andrewbogott>	 Neha16: I am!  What's up?
[07:40:46] <Neha16>	 andrewbogott: I just ssh'ed into toolsbeta and found that the webservice files have been archived?
[07:41:09] <andrewbogott>	 let me look
[07:44:07] <andrewbogott>	 Neha16: which files in particular?
[07:44:47] <Neha16>	 andrewbogott: In /usr/local/bin/ webservice and webservice2
[07:46:14] <andrewbogott>	 huh, I see what you mean
[07:46:56] <andrewbogott>	 actually, when you say 'archived' do you mean 'deleted'?  Or do you see them moved someplace?
[07:47:52] <Neha16>	 I mean it has been deleted and a new file webservice.save has the code now
[07:48:18] <Neha16>	 I wanted to reproduce a bug and I got the error of `webservice command not found`
[07:48:31] <andrewbogott>	 oh, ok, I see webservice.save now
[07:49:01] <andrewbogott>	 let's see… zhuyifei1999_ merged some changes there on the 15th, but surely this happened more recently than that
[07:49:53] <Neha16>	 Yes, everything was working fine a couple of days ago.
[07:55:29] <andrewbogott>	 someone else was clearly experimenting here and then forgot to clean up after themselves.  So far I can't tell who.  I think you should go ahead and restore that file to whatever version you want to test and leave a note in the service log about what you did (even though whoever broke it did not leave a note :( )
[07:55:35] <andrewbogott>	 Do you know how to use the service log?
[07:55:54] <Neha16>	 andrewbogott: No, I don't know how to use the service log
[07:56:06] <andrewbogott>	 Neha16: like this:
[07:56:31] <andrewbogott>	 !log toolsbeta someone removed /usr/bin/webservice
[07:56:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[07:56:54] <andrewbogott>	 Neha16: then you can see all the notes that people made by clicking that link.  Make sense?
[07:57:32] <Neha16>	 andrewbogott: Yup, got it. While we are at it, do you know where is the `kubernetesbackend.py` file located?
[07:58:14] <andrewbogott>	 looking
[07:59:37] <andrewbogott>	 (so far it's not where I expected)
[08:01:48] <andrewbogott>	 argh, it's present on the production tools-bastion but not on toolsbeta
[08:02:24] <andrewbogott>	 on the regular bastion:
[08:02:33] <andrewbogott>	 https://www.irccloud.com/pastebin/p9KSsoM2/
[08:03:05] <andrewbogott>	 installed a spart of the ' toollabs-webservice' package.  That package isn't present on toolsbeta-bastion-01
[08:03:14] <andrewbogott>	 sorry, 'installed as part of'
[08:04:13] <Neha16>	 andrewbogott: Should I install toollabs-webservice on toolsbeta?
[08:05:37] <andrewbogott>	 probably!  that file has been present historically hasn't it?
[08:06:26] <Neha16>	 andrewbogott: Actually, no. i couldn't find it even when the webservice files were intact.
[08:06:38] <andrewbogott>	 I see someone installing it and then removing it on the 17th.  Lemme see if I can tell who that was...
[08:08:45] <Neha16>	 andrewbogott: Even I have upgraded a few packages in toolsbeta for installing kuberenetes. Is that the issue?
[08:09:07] <andrewbogott>	 Neha16: ok, it looks like it was zhuyifei1999_ who was messing with this, so he's the one to ask about what's going on here.
[08:09:36] <andrewbogott>	 I think it's fine for you to go ahead and install what you need there.  Hopefully you don't get stuck in a dependency mess...
[08:10:29] <Neha16>	 andrewbogott: Ok, will do. I have emailed you a bunch of questions. Please look at it if you get some time.
[08:10:53] <andrewbogott>	 Neha16: is that the email "re: help"?
[08:11:16] <Neha16>	 andrewbogott: Yup. And I am again in stuck in dependency mess 
[08:11:39] <andrewbogott>	 I read the email and didn't know the answers so am hoping that bryan will respond when he's awake :)
[08:12:45] <Neha16>	 andrewbogott: https://pastebin.com/LJg2QuYm
[08:12:48] <zhuyifei1999_>	 yes?
[08:13:14] <zhuyifei1999_>	 I would check that with aptitude
[08:13:17] <andrewbogott>	 zhuyifei1999_: is it not 3AM where you are?  Sorry for making your phone buzz or whatever :/
[08:13:25] <andrewbogott>	 zhuyifei1999_: you were installing/removing this package recently?
[08:13:27] * zhuyifei1999_ is 4PM
[08:13:33] <zhuyifei1999_>	 I don't think so
[08:13:38] <zhuyifei1999_>	 maybe some upgrades
[08:14:04] * zhuyifei1999_ is UTC+8 right now
[08:14:36] <andrewbogott>	 oh!  Forgot you were elsewhere for the summer :)
[08:15:37] <zhuyifei1999_>	 I installed aptiitude. will check what's going on with toollabs-webservice not being installed
[08:15:56] <andrewbogott>	 thx
[08:16:07] <Neha16>	 zhuyifei1999_: Thanks
[08:16:47] <zhuyifei1999_>	 The following packages have unmet dependencies:
[08:16:48] <zhuyifei1999_>	  libpython2.7-stdlib : Breaks: python-urllib3 (< 1.9.1-3) but 1.7.1-1ubuntu4 is to be installed.
[08:17:41] * zhuyifei1999_ runs `sudo apt upgrade` and see if that fixes anything
[08:18:39] <zhuyifei1999_>	 (no)
[08:20:01] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/uVh38iGX/
[08:20:22] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/yreFSDTx/
[08:21:15] <andrewbogott>	 wow, messy
[08:21:16] <zhuyifei1999_>	 ^ don't seem like a good idea
[08:21:58] <Neha16>	 zhuyifei1999_: I have upgraded the python version and installed a few packages required for kubernetes. Could this be the cause?
[08:22:14] <zhuyifei1999_>	 could be possible
[08:22:17] <andrewbogott>	 https://www.irccloud.com/pastebin/0olGyBOd/
[08:22:41] <andrewbogott>	 so, still running 2.7.5 on regular tools
[08:22:43] <zhuyifei1999_>	 Neha16: how did you upgrade python version?
[08:23:26] <Neha16>	 zhuyifei1999_: I think I used pip
[08:23:38] <zhuyifei1999_>	 pip can't upgrade python
[08:24:18] <zhuyifei1999_>	 zhuyifei1999@toolsbeta-bastion-01:~$ cat /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list 
[08:24:18] <zhuyifei1999_>	 deb http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main
[08:24:18] <zhuyifei1999_>	 # deb-src http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main
[08:24:28] * zhuyifei1999_ kills that ppa
[08:25:46] <zhuyifei1999_>	 performing downgrade
[08:25:48] <Neha16>	 zhuyifei1999_: Yes, I used that ppa
[08:26:10] <zhuyifei1999_>	 done. now installed. ppa is removed
[08:27:16] <zhuyifei1999_>	 for any python packages you need to install, search apt first
[08:27:28] <zhuyifei1999_>	 like $ apt search pykube
[08:28:53] <zhuyifei1999_>	 using a ppa have a ton of issues and can break stuffs if not done carefully. I don't think we ever used a ppa on toolforge
[08:30:04] <Neha16>	 zhuyifei1999_: Can I use pip to install packages ?
[08:31:11] <zhuyifei1999_>	 !log toolsbeta on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
[08:31:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[08:31:29] <zhuyifei1999_>	 you can, but only for experimentation
[08:31:53] <zhuyifei1999_>	 we won't use pip to install packages globally on toolforge
[08:32:42] <Neha16>	 zhuyifei1999_: I wanted to install kubernetes package for python https://github.com/kubernetes-client/python#installation.
[08:33:30] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/Fwnm4rdz/
[08:33:42] <zhuyifei1999_>	 the package isn't available in apt
[08:34:05] <Neha16>	 zhuyifei1999_: Yes, so how do I install that?
[08:34:24] <zhuyifei1999_>	 andrewbogott: what do you think about building a deb?
[08:35:42] <zhuyifei1999_>	 this package doesn't exist on jessie either
[08:36:30] <andrewbogott>	 I'm not sure.  I'm surprised the package isn't available on trusty!  We might be able to backport it from a newer ubuntu version.
[08:37:50] <zhuyifei1999_>	 I don't find it in xenial either
[08:38:02] <zhuyifei1999_>	 even pykube is missing
[08:38:06] <andrewbogott>	 zhuyifei1999_: happen to know if it's available for Stretch?  It's so new it might not have made it into pre-2017 distros...
[08:38:16] <zhuyifei1999_>	 (I run xenial locally)
[08:39:09] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/CTCk2cYi/
[08:39:37] <zhuyifei1999_>	 I think yuvi packaged pykube
[08:39:51] <zhuyifei1999_>	 let me check stretch
[08:40:11] <andrewbogott>	 I think the thing to do to get neha unstuck is install that package from source on toolsbeta
[08:40:24] <andrewbogott>	 and also create a ticket about properly packaging it for Trusty and Jessie for future purposes
[08:40:46] <andrewbogott>	 Of course the 'install from source' step only works if it doesn't have a million dependencies that also aren't present
[08:41:31] <Neha16>	 andrewbogott: https://pastebin.com/kdXAQAwg
[08:41:46] <Neha16>	 I am still stuck
[08:41:56] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/oDRGbUGA/
[08:42:09] <zhuyifei1999_>	 yeah, exists on stretch
[08:42:17] <andrewbogott>	 that's encouragning
[08:42:42] <zhuyifei1999_>	 do you know how to backport stuffs?
[08:43:09] <zhuyifei1999_>	 Neha16: locale is missing? how broken is python o.O
[08:43:11] <andrewbogott>	 I do, although that only works if we don't have to backport incompatible dependencies
[08:43:58] <zhuyifei1999_>	 it'll be needed on both trusty and jessie
[08:44:30] <zhuyifei1999_>	 Neha16: what happened right before locale gets missing?
[08:45:17] <Neha16>	 zhuyifei1999_: This is the first error I got after the downgrade.
[08:45:39] <zhuyifei1999_>	 was there any errors before the upgrade?
[08:45:54] <Neha16>	 zhuyifei1999_: No
[08:48:07] <zhuyifei1999_>	 the module should be located at /usr/lib/python2.7/locale.py
[08:49:07] <andrewbogott>	 zhuyifei1999_: you tried just installing the package already?
[08:49:30] <Neha16>	 zhuyifei1999_: Nope, it's not there
[08:49:42] <zhuyifei1999_>	 on xenial it's provided by:
[08:49:42] <zhuyifei1999_>	 $ dpkg-query -S /usr/lib/python2.7/locale.py
[08:49:42] <zhuyifei1999_>	 libpython2.7-minimal:amd64: /usr/lib/python2.7/locale.py
[08:50:45] * zhuyifei1999_ reinstalls this
[08:51:07] * andrewbogott tried that already :(
[08:51:12] <zhuyifei1999_>	 hmm
[08:51:15] <zhuyifei1999_>	 06:40:03 0 ✓ zhuyifei1999@tools-bastion-02: ~$ dpkg-query -S /usr/lib/python2.7/locale.py
[08:51:15] <zhuyifei1999_>	 libpython2.7-stdlib:amd64: /usr/lib/python2.7/locale.py
[08:51:39] <zhuyifei1999_>	 (it's different on trusty, apparently)
[08:52:58] <zhuyifei1999_>	 ran `sudo apt install --reinstall libpython2.7-minimal libpython2.7-stdlib`, I think it's fixed
[08:53:33] <andrewbogott>	 ok, I think I've got it
[08:53:34] <andrewbogott>	 via apt-get install --reinstall libpython2.7
[08:53:53] <andrewbogott>	 Neha16: is 'webservice' better behaved now?
[08:55:18] <Neha16>	 andrewbogott: Now, I get different sorts of error. It is related to urllib 3 and pykube. The libraries that I messed with the other day :(
[08:56:18] <andrewbogott>	 Neha16: do you have changes that you value for any of those?  We can just reinstall those packages too, if not
[08:56:45] <Neha16>	 andrewbogott: No, I do not have meaningful changes
[08:56:48] * zhuyifei1999_ is thinking about nuking and rebuilding that bastion
[08:57:25] <zhuyifei1999_>	 too bad that I know know of a way to recursively reinstall packages :(
[08:57:43] <andrewbogott>	 Neha16: any better now?
[08:58:45] <Neha16>	 andrewbogott: https://pastebin.com/W9tHJM6S
[08:58:50] <Neha16>	 SSL warnings
[08:59:23] <Neha16>	 andrewbogott: I had turned that off in kubernetes
[09:00:02] <zhuyifei1999_>	 Neha16: let's just rebuild that vm?
[09:00:14] <Neha16>	 zhuyifei1999_: that would be great
[09:00:18] <andrewbogott>	 at this point that might be the right choice
[09:00:31] * zhuyifei1999_ goes to get my 2fa keys
[09:01:25] <zhuyifei1999_>	 Neha16: oh, did you do any code modifications?
[09:01:38] <Neha16>	 zhuyifei1999_: I have code snippets saved.
[09:01:43] <zhuyifei1999_>	 better save them before the vm is killed
[09:01:43] <zhuyifei1999_>	 ok
[09:03:06] <andrewbogott>	 Neha16: unfortunately I need to go in a few minutes (and will be traveling/flying for much of the weekend).  I'll try to catch up here or via email when I have the chance.
[09:03:23] <zhuyifei1999_>	 !log toolsbeta killing and rebuilding toolsbeta-bastion-01
[09:03:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[09:03:25] <andrewbogott>	 with luck, a fresh bastion will get you mostly unstuck
[09:03:36] <Neha16>	 andrewbogott: That's ok. I just need webservices working for now :)
[09:03:52] <andrewbogott>	 zhuyifei1999_: thank you for working so hard on this
[09:04:16] <zhuyifei1999_>	 np :)
[09:04:23] <Neha16>	 zhuyifei1999_: Thanks a lot! and sorry for all the trouble because of me.
[09:05:48] <zhuyifei1999_>	 Neha16: afterwards, you can install the k8s module on that bastion for testing, and submit a pach, but it will not be deployed till someone gets the deb package backported 
[09:06:46] <Neha16>	 zhuyifei1999_: Did you mean installing it from source. That is the only options left I guess
[09:06:57] <zhuyifei1999_>	 installing from pip
[09:07:11] <zhuyifei1999_>	 only for testing
[09:07:11] <Neha16>	 zhuyifei1999_: Ok. Thanks a lot
[09:09:59] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/c2bxqA23/
[09:10:44] <zhuyifei1999_>	 of the four python packages that it depends on, only python-pbr is not installed on a default bastion build
[09:12:13] <Neha16>	 Apart from that there were other unmet dependencies such as tornado, ecdsa and a couple more
[09:14:30] <zhuyifei1999_>	 ?
[09:14:44] <zhuyifei1999_>	 what do you mean?
[09:14:55] <zhuyifei1999_>	 fyi, https://phabricator.wikimedia.org/T197930
[09:15:27] <Neha16>	 while installing k8s I got many unmet dependency - tornado and ecdsa being one of them
[09:15:39] <Neha16>	 Can I ssh in toolsbeta now?
[09:19:54] <zhuyifei1999_>	 puppet is still running
[09:19:59] <zhuyifei1999_>	 so, maybe
[09:20:24] <zhuyifei1999_>	 installing with pip?
[09:20:54] <Neha16>	 zhuyifei1999_: Host key verification failed. Does this require updating the ssh keys?
[09:20:59] <zhuyifei1999_>	 yes
[09:21:01] <Neha16>	 Yes, installing with pip
[09:21:12] <zhuyifei1999_>	 it's a new vm, so new keys
[09:29:26] <zhuyifei1999_>	 Neha16: it should be built now
[09:30:06] <zhuyifei1999_>	 uh, puppet is broken
[09:33:38] <zhuyifei1999_>	 !log toolsbeta puppet is broken on toolsbeta-bastion-01, investigating
[09:33:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[09:39:01] <zhuyifei1999_>	 !log toolsbeta fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
[09:39:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[09:41:51] <zhuyifei1999_>	 Neha16: should be completely functional now
[09:45:27] <Neha16>	 zhuyifei1999_: It's functional now but there is another module missing `python_2_unicode_compatible`.
[09:45:41] <zhuyifei1999_>	 oh I remember that
[09:45:51] <zhuyifei1999_>	 run $ sudo apt upgrade
[09:47:16] <Neha16>	 Unmet dependecies. It suggests `apt-get -f install`. Should I go with it?
[09:48:02] <zhuyifei1999_>	 huh
[09:48:05] <zhuyifei1999_>	 checking
[09:48:28] <zhuyifei1999_>	 yes
[09:48:32] <zhuyifei1999_>	 just add sudo
[09:49:19] <zhuyifei1999_>	 Neha16: ^
[09:49:32] <Neha16>	 zhuyifei1999_: Nothing changes. Same errors
[09:49:37] <zhuyifei1999_>	 hmm
[09:49:41] * zhuyifei1999_ does this
[09:50:25] * zhuyifei1999_ gets aptitude
[09:51:14] <zhuyifei1999_>	 aptitude doesn't complain at all
[09:51:16] <zhuyifei1999_>	 weird
[09:53:08] <Neha16>	 zhuyifei1999_: `python-websocket : Breaks: python-websocket-client but 0.23.0-1 is to be installed`
[09:53:19] <Neha16>	 This is the unmet dependency
[09:53:39] * zhuyifei1999_ did the upgrade with aptitude
[10:01:38] <zhuyifei1999_>	 Neha16: everything working now?
[10:04:29] <Neha16>	 zhuyifei1999_: Works like a charm. Thank you so much for your help
[10:04:52] <Neha16>	 And sorry for the trouble.
[10:08:43] <zhuyifei1999_>	 np :)
[19:18:25] <halfak>	 Hey folks.  If you want help running a WDQS query, where would you go? 
[19:18:31] <halfak>	 Is there a query service page or something?
[19:18:58] <halfak>	 E.g. https://www.mediawiki.org/wiki/Talk:Quarry works great for running SQL on quarry. 
[19:20:58] <halfak>	 Seems like https://www.mediawiki.org/wiki/Talk:Wikidata_Query_Service isn't used in that way
[19:29:44] <quiddity>	 halfak, the dropdown menu in WDQS itself, points to this other page, which might be what you want? https://www.wikidata.org/wiki/Wikidata:Request_a_query
[20:06:21] <framawiki>	 !log tools.totoazero create a "sql-dump.sh" script with corresponding daily cron and 7d old cleaner to mysqldump tool db
[20:06:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.totoazero/SAL
[20:56:07] <framawiki>	 !log tools.totoazero deploy drp_warn.py 4aebda8 https://github.com/Toto-Azero/Wikipedia/pull/11
[20:56:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.totoazero/SAL
[22:25:08] <framawiki>	 !log tools.totoazero deploy sysops_archives.py sysops_archives_new.py sysops_archives_renew.py 4a599b6 https://github.com/Toto-Azero/Wikipedia/pull/12
[22:25:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.totoazero/SAL
[22:26:14] <chicocvenancio>	 !log toolsbeta reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
[22:26:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL