[02:05:14] <zhuyifei1999_>	 !log quarry Deployed 6069904
[02:05:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[02:07:40] <chicocvenancio>	 something broke the culler in PAWS 
[02:12:31] <chicocvenancio>	 :(
[02:46:25] <zhuyifei1999_>	 uh oh
[02:47:20] <zhuyifei1999_>	 the admin panel looks mostly good to me
[02:47:51] <chicocvenancio>	 I manually killed all pods with more than one hour innactivity
[02:47:57] <chicocvenancio>	 ahh, btw
[02:48:32] <chicocvenancio>	 !log paws killed 25 pods with more than one hour inactivity through admin interface
[02:48:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[02:49:02] <chicocvenancio>	 I'm at a loss on how this could happen
[02:49:44] * zhuyifei1999_ looks to the logs...
[02:57:17] <zhuyifei1999_>	  /paws/hub/api/users isn't being scalable :(
[02:57:56] <chicocvenancio>	 we're only at 708
[02:58:04] <chicocvenancio>	 something is wrong there
[03:01:44] <zhuyifei1999_>	 on which port inside the pod does the hub listen?
[03:04:10] <chicocvenancio>	 8081/TCP
[03:04:24] <zhuyifei1999_>	 ok
[03:04:58] * zhuyifei1999_ just did the hard way of reading /proc/net/tcp
[03:05:39] <chicocvenancio>	 `kubectl -n prod get service` is the easy way
[03:06:55] <zhuyifei1999_>	 ok
[03:10:20] <zhuyifei1999_>	 ok so I can reproduce that request with `python3 -c 'print(__import__("requests").get("http://localhost:8081/paws/hub/api/users", headers={"Authorization":"token c[REDACTED, see /proc/23/environ]8"}).text)'`, running `time` on it gives 0m23.629s
[03:10:51] <chicocvenancio>	 can you restart the hub?
[03:11:02] <chicocvenancio>	 delete the pod that is running will make it restart
[03:11:24] <zhuyifei1999_>	 uh, can't you do it?
[03:11:35] * chicocvenancio is no longer root
[03:13:16] <zhuyifei1999_>	 kubectl delete pod hub-7d68647bf4-pqm74?
[03:14:10] <zhuyifei1999_>	 chicocvenancio: ^
[03:15:34] * zhuyifei1999_ guess yes, hopefully I won't screw up
[03:15:49] <chicocvenancio>	 -n prod
[03:15:56] <zhuyifei1999_>	 oh right
[03:16:38] <zhuyifei1999_>	 !log tools.paws restarted PAWS hub pod by `kubectl delete pod hub-7d68647bf4-pqm74 -n prod`
[03:16:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws/SAL
[03:17:34] <zhuyifei1999_>	 https://paws.wmflabs.org/ 'Service Unavailable' not good...
[03:17:54] <chicocvenancio>	 saw that as well
[03:17:58] <chicocvenancio>	 is it starting?
[03:18:17] <chicocvenancio>	 `kubectl get pods -n prod`
[03:18:31] <zhuyifei1999_>	 I see one named hub-7d68647bf4-9kj22
[03:19:05] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/PEe7lqjn/
[03:19:08] <zhuyifei1999_>	 ^ logs
[03:19:24] <chicocvenancio>	 then that looks fine
[03:19:48] <chicocvenancio>	 should be up in a minute or two
[03:19:59] <chicocvenancio>	 though, that is slow
[03:20:07] <chicocvenancio>	 it is up now
[03:20:12] <zhuyifei1999_>	 yeah I see [I 2018-05-04 03:19:27.059 JupyterHub app:1528] Hub API listening on http://0.0.0.0:8081/paws/hub/
[03:20:42] <zhuyifei1999_>	 culler just failed again
[03:21:30] <zhuyifei1999_>	 wait, maybe not
[03:21:43] <chicocvenancio>	 time the api now
[03:23:25] <zhuyifei1999_>	 0m32.440s
[03:23:38] <chicocvenancio>	 that is not an improvement
[03:24:02] <zhuyifei1999_>	 the logs are spewing all sorts of weird errors
[03:24:29] <chicocvenancio>	 if the api is that slow, I imagine few things won't break
[03:25:43] <chicocvenancio>	 that last one is know issued I fixed upstream 
[03:26:28] <zhuyifei1999_>	 how do you update?
[03:26:41] <chicocvenancio>	 T189534
[03:26:42] <stashbot>	 T189534: PAWS doesn't kill server for users with non-ascii characters - https://phabricator.wikimedia.org/T189534
[03:27:25] <zhuyifei1999_>	 k
[03:28:15] <zhuyifei1999_>	 just tell me when it's ready for us to upgrade. (and please tell me how when you do :) )
[03:28:34] <chicocvenancio>	 in theory we can push this locally, but they are going to be releasing soon upstream https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/617
[03:28:41] <zhuyifei1999_>	 k
[03:28:45] <chicocvenancio>	 it should be just pushing a code change
[03:28:55] <chicocvenancio>	 and watching to make sure nothing catches fire
[03:29:50] <zhuyifei1999_>	 thanks for fixing btw
[03:32:39] <chicocvenancio>	 you're welcome
[03:36:16] <chicocvenancio>	 I just remembered this is a sqlite db
[03:36:20] <chicocvenancio>	 on NFS
[03:36:53] <zhuyifei1999_>	 oh, that makes sense
[03:38:19] <zhuyifei1999_>	 I don’t know if it’s intentional or not, but I still see your root key at https://wikitech.wikimedia.org/wiki/Hiera:Tools
[03:39:08] <chicocvenancio>	 ohh, that is not intentional
[03:39:12] <chicocvenancio>	 bd808: ^
[03:52:27] <bd808>	 doh. thanks chicocvenancio and zhuyifei1999_ 
[04:05:16] <zhuyifei1999_>	 (np)
[16:17:25] <paladox>	 andrewbogott https://gerrit.wikimedia.org/r/#/c/430937/ thanks :)
[16:17:38] <paladox>	 that was annoying me too.
[16:17:54] <andrewbogott>	 paladox: I'll have to get many many people to sign off on that before it's merged...
[16:18:00] <andrewbogott>	 also it doesn't work for some reason?  Still poking.
[16:18:00] <paladox>	 oh
[16:18:16] <paladox>	 andrewbogott :set mouse=r
[16:19:12] <paladox>	 andrewbogott unless it's this https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864074
[16:19:21] <andrewbogott>	 https://unix.stackexchange.com/questions/44513/disabling-mouse-support-in-vim-in-a-gnome-terminal-environment?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
[16:19:40] <andrewbogott>	 paladox: yep, my paste says the same thing as yours
[16:25:15] <paladox>	 andrewbogott i think this may be the work around https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864074#40
[16:25:45] <andrewbogott>	 yep, trying...