[02:02:29] <zhuyifei1999_>	 Betacommand: do you have a few lines of code that I can reliably reproduce the error?
[02:04:15] <Betacommand>	 zhuyifei1999_: see PM
[02:33:44] <paladox>	 bd808 i've noticed that using the ldapauth puppet class dosen't really hook it into mw so mw dosen't use the auth.
[02:33:50] <paladox>	 is that intended?
[02:34:13] <bd808>	 it provisions a special wiki 
[02:34:20] <bd808>	 rather than taking over the main wiki
[02:34:37] <paladox>	 oh
[02:34:52] <bd808>	 I think its name is "ldapauth" :)
[02:35:33] <paladox>	 heh.
[02:35:45] <paladox>	 bd808 would you know the url and how i can expose it please?
[02:37:35] <bd808>	 ldapauth.local.wmftest.net:8080 on a laptop. On a Cloud VPS it would be ldapauth-<base name> where <base name> defaults to the Cloud VPS vm name I think
[02:38:16] <bd808>	 paladox: see https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS#Run_a_wikifarm
[02:38:21] <paladox>	 thanks
[06:53:55] <Amir1>	 !log wikilabels 27a4fe9 is going to staging
[06:53:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[06:56:55] <Amir1>	 Everything looks fine, going to prod
[06:58:33] <Amir1>	 !log wikilabels b15a87c is going to staging
[06:58:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[07:00:47] <Amir1>	 !log wikilabels deploying b15a87c to prod
[07:00:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[08:32:12] <wikibugs>	 (03PS1) 10Legoktm: Blacklist MediaWikiFarm in dashboard too [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/398793
[08:32:24] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Blacklist MediaWikiFarm in dashboard too [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/398793 (owner: 10Legoktm)
[08:32:40] <wikibugs>	 (03Merged) 10jenkins-bot: Blacklist MediaWikiFarm in dashboard too [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/398793 (owner: 10Legoktm)
[11:50:09] <arturo>	 !log tools restart pod hub-deployment in paws to try to fix the 502
[11:50:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:51:49] <arturo>	 !log tools the restart was with: kubectl get pod -o yaml hub-deployment-1381799904-b5g5j -n prod | kubectl replace --force -f -
[11:51:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:58:16] <arturo>	 !log tools The restart didn't work. I could see a lot of log lines in the hub-deployment pod with something like: 2017-12-17 04:08:17,574 WARNING Connection pool is full, discarding connection: 10.96.0.1
[11:58:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:04:23] <arturo>	 !log tools it seems jupyterhub tries to use a database which doesn't exists: [E 2017-12-18 11:59:49.896 JupyterHub app:904] Failed to connect to db: sqlite:///jupyterhub.sqlite
[12:04:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:50:13] <Sagan>	 !log rcm Xenon: Doing update of Phabricator
[12:50:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[15:24:07] <leszek_wmde>	 howdy all! I am having some problems with npm on a jessie instance. Is there someone available so I could present some rather unspecific problem?
[15:28:37] <andrewbogott>	 leszek_wmde: I'm here, but I've never used npm.  Is the problem within npm or does it have to do with getting set up in the first place?
[15:31:13] <leszek_wmde>	 andrewbogott: hi! so the thing is actually like this: on the instance there used to be nodejs/npm set up. honestly speaking I didn't do it myself, so I cannot tell for sure how was it set up. But what we've just noticed that there is no /usr/bin/npm on this instance any more
[15:31:31] <leszek_wmde>	 andrewbogott: while nodejs -v still says "i'm here"
[15:31:58] <andrewbogott>	 This is on a new VM or one that y'all have been working on for a long time?
[15:32:11] <andrewbogott>	 (It's not obvious to me that installing nodejs would necessarily also install npm)
[15:32:13] <leszek_wmde>	 andrewbogott: around 2 years on the same VM
[15:33:20] <paladox>	 try apt-get install npm
[15:33:24] <chasemp>	 leszek_wmde: I would do an 'aptitude search npm | grep ^i' to see what's insalled and then 'dpkg -L $package' to see what files are included in what's installed
[15:33:33] <paladox>	 will only work on jessie.
[15:33:41] <chasemp>	 to see if you should in theory have an npm interpreter
[15:33:56] <chasemp>	 that's investigative, alternatively, yeah just try to install and see what happens
[15:33:58] <andrewbogott>	 yeah, I would say 'just install it' but of course I can't guess if/why it was there before and where it went
[15:34:41] <Steinsplitter>	 paws seems down
[15:35:12] <chasemp>	 Steinsplitter: known but unknown cause, we are waiting to ask yuvi when it's his waking hours in a bit
[15:35:32] <leszek_wmde>	 so aptitude search npm returns nothing
[15:35:39] <leszek_wmde>	 did apt-get install npm though
[15:35:49] <andrewbogott>	 Steinsplitter: https://phabricator.wikimedia.org/T183142
[15:35:55] <Steinsplitter>	 thanks :)
[15:36:00] <leszek_wmde>	 which seems to restore it, so that helps, thanks!
[15:36:12] <leszek_wmde>	 still makes me wonder why did it go away in the first place :)
[15:36:25] <chasemp>	 leszek_wmde: that's weird, are you on a different instance than you thought or something?
[15:36:27] <chasemp>	 but yeah, odd
[15:37:42] <leszek_wmde>	 haha, that would be a nice fail. but i am on the right instance
[15:41:39] <leszek_wmde>	 thanks everyone for help. "just install it" did a trick. If the issue comes back, I will be back here :)
[17:16:15] <Krinkle>	 bd808: Regarding guc db server use, see https://tools.wmflabs.org/guc/?src=hr&user=KrinkleBot&debug=true (debug output at the bottom)
[17:17:45] <bd808>	 Krinkle: *nod* so it is opening 8 connections per request. I bet we can ask for a higher concurrent connection limit. The default is 10 right now
[17:18:27] <bd808>	 We'd have to look at the code, but it can be hard to tell when PHP actually releases a connection
[17:19:25] <bd808>	 in reality it could do all the queries over a single connection today, but changing to assume that would probably break things at some point in the future
[18:38:28] <zhuyifei1999_>	 bd808: could you check the logs of bigbrother to see if there is anything wrong with it? It failed to automatically restart the job 'ircbot' of tool 'media-dubiety' today.
[18:52:10] <marmick>	 hi guys
[18:52:21] <marmick>	 I haven't been using the labs for a while and i'd like to get back on track to dev n research, but it seems that things are not as i remember (or perhaps i remember them incorrectly). anyone can help me with a couple of doubts?
[18:52:51] <zhuyifei1999_>	 marmick: yes?
[18:52:53] <bd808>	 marmick: let's hear what you are confused about. Hopefully we can point you in the right direction
[18:53:08] <marmick>	 :) thanks
[18:53:09] <marmick>	 I am trying to connect to any database and i can't (i remember just using mysql cawiki was enough to connect to that version of the wikipedia)
[18:53:19] <marmick>	 It gives me the typical error saying that I am using my username (marcmiquel is the one I use to connect with ssh) password: NO. If I try passing the parameters (my real username is the id u3532) and cawiki as a database (or cawiki_p) it does not work either...
[18:53:26] <bd808>	 Try `sql cawiki`
[18:53:34] <marmick>	 I have the replica.my.cnf with my user, so I thought it would work automatically as it was, but perhaps I am missing or missunderstanding something
[18:53:47] <marmick>	 ok, here i go
[18:54:02] <bd808>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_to_the_database_replicas
[18:54:08] <marmick>	 ok, first one solved fast, great thanks bd808
[18:54:21] <marmick>	 second...
[18:54:28] <marmick>	 is it possible to create new databases as I used to?
[18:54:36] <marmick>	 i am going to be building a tool
[18:54:57] <marmick>	 and i need to create a database with a list of articles and some attributes
[18:55:00] <bd808>	 yes, but only on the tools.db.svc.eqiad.wmflabs database server
[18:55:03] <marmick>	 (name, id, etc.)
[18:55:17] <bd808>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases
[18:55:29] <marmick>	 then I need to change the code to create and insert in the new database. will it work with my user and password?
[18:55:59] <bd808>	 yes, the credentials from your replica.my.cnf will work there
[18:56:08] <marmick>	 great, thanks :)
[18:56:26] <marmick>	 third and last question
[18:56:37] <marmick>	 I would like to retrieve data from Wikidata. I am checking out some alternatives and I've seen some people use a python sparql wrapper and some other people use pywikibot. Considering that when I connect to Wikipedia langauge editions databases in tool labs they are a copy, is there a copy of Wikidata that I could connect to? Which of the two previous ways I mentioned you think it is best to retrieve/check? Or maybe, it
[18:56:37] <marmick>	  is better to use WikiData Toolkit in Java?
[18:56:39] <bd808>	 the change about where user created dbs can be hosted is related to https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown
[18:57:15] <bd808>	 there is a wikidatawiki database in the Wiki Replicas
[18:57:45] <bd808>	 so yes you can dig there for statements if you want to
[18:58:13] <marmick>	 is that the best in terms of performance?
[18:59:06] <bd808>	 marmick: I think that's hard to say generally. The folks in the #wikidata channel may be able to help you understand what is best for a particular use case
[18:59:16] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/395934 (owner: 10L10n-bot)
[18:59:25] <marmick>	 ok! :) thanks bd808 for all the answers, u've been very helpful
[18:59:43] <bd808>	 :) thanks. It's my job, but it's nice when people notice
[18:59:56] <marmick>	 :)
[19:00:00] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/397258 (owner: 10L10n-bot)
[19:00:58] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/398220 (owner: 10L10n-bot)
[19:01:34] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/398873 (owner: 10L10n-bot)
[19:13:08] <Zppix>	 Hey cloud is it possible to ssh into vps instances from bastion.wmflabs.org?
[19:13:57] <zhuyifei1999_>	 Zppix: yes, but not recommended
[19:14:24] <Zppix>	 How does one do this?
[19:14:34] <zhuyifei1999_>	 you can use agent forwarding for that, but it allows any root to use your key :P
[19:14:59] <Zppix>	 How?
[19:15:17] <zhuyifei1999_>	 openssh: ssh -A
[19:15:29] <zhuyifei1999_>	 when you ssh into that bastion
[19:16:00] * zhuyifei1999_ dunno about pytty
[19:16:03] * zhuyifei1999_ gtg
[19:16:08] <chasemp>	 yeah, please do access the bastion directly via ssh
[19:16:13] <chasemp>	 that only works because we haven't locked it down yet
[19:16:18] <chasemp>	 do not even :)
[19:16:31] <Zppix>	 I have no other choice
[19:16:53] <chasemp>	 why is that?
[19:18:11] <Zppix>	 On mobile and wont have access to a pc for awhile
[19:32:11] <Zppix>	 chasemp: ^
[19:32:56] <chasemp>	 Zppix: I saw, circle back when you have time it's not an emergency, but the model of landing on the VPS bastion to use key forwarding is not guaranteed fyi
[19:36:20] <Zppix>	 :/
[19:36:31] <Adotchar>	 hi
[19:40:12] <mutante>	 https://www.chimuadventures.com/blog/2017/02/cloud-people-peru/
[19:40:52] <Zppix>	 mutante: wut?
[19:42:05] <mutante>	 heh, sorry, it was about the term "cloud people" that i was about to use
[19:42:34] <mutante>	 what i really wanted is ask if if can turn off the ganglia stats for labsdb
[19:42:43] <mutante>	 it's really the last thing that is it now
[19:42:49] <mutante>	 2 hosts out of $ALL
[19:43:24] <chasemp>	 mutante: I think only the dbas could answer that well, I'm not sure if they still look at ganglia but if they say it's cool then +1
[19:44:54] <mutante>	 chasemp: cool! *nod* and thanks
[19:51:30] <mutante>	 chasemp: ...and i see that the specific pgsql views are existing but .. empty and no data in them, i guess that question answered itself :)
[19:52:29] <chasemp>	 mutante: postgres does exist on labsdb1006-7 iiuc and the dbas and/or alex may know if it's related
[19:53:29] <mutante>	 yes, it's those hosts. ack. alex is working on replacement in prometheus already
[19:53:46] <mutante>	 i think just one merge pending
[19:54:09] <chasemp>	 cool
[19:54:30] <Zppix>	 I got a question, what is the db for users on toolforge running? Same as prod?
[19:57:02] <mutante>	 this is the ticket, fwiw https://phabricator.wikimedia.org/T179306
[19:58:27] <mutante>	 and the change for it https://gerrit.wikimedia.org/r/#/c/392441/  (just linked)
[20:10:47] <marmick>	 i'm trying to install this https://pypi.python.org/pypi/Wikidata/0.6.1 using pip install but it tells me "error in Wikidata setup command: Invalid environment marker: python_version<"3.5""
[20:11:05] <marmick>	 then i checked and the latest python is 3.4
[20:11:24] <marmick>	 could i update it in my virtual environment somehow?
[20:11:42] <Zppix>	 You have to recreate it
[20:11:47] <Zppix>	 If on toolforge
[20:12:06] <marmick>	 recreate?
[20:13:35] <chasemp>	 marmick: odds are you are not using virtualenv? https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python_.28Python3_.2B_Kubernetes.29
[20:14:12] <marmick>	 yes, i'm using virtualenv
[20:14:25] <marmick>	 an old one i used one year and half ago
[20:15:13] <chasemp>	 ah yes I see, I missed the last line
[20:15:51] <chasemp>	 I'm not sure beyond googling around honestly
[20:16:10] <marmick>	 ok, thanks anyway
[20:16:24] <marmick>	 i'm going to have some dinner
[20:16:25] <Zppix>	 chasemp: i asked a similar quesion about going to py3 from py2 and was told you have to recreate to change py version
[20:16:44] <marmick>	 Zppix: does recreate mean install inside venv?
[20:16:53] <Zppix>	 No i mean recreate venv
[20:17:07] <marmick>	 ah, start venv from zero
[20:17:20] <marmick>	 ok :)
[20:17:24] <marmick>	 thx
[20:18:05] <Zppix>	 Np
[20:19:14] <bd808>	 we also only have python3.5 available on the Debian Jessie Kubernetes images I believe.
[20:20:07] <Zppix>	 bd808: you do i was just playing around with them
[20:20:43] <bd808>	 the "Invalid environment marker" error may be from an old version of pip inside the venv. `pip install -U pip` might fix that
[20:20:49] <Zppix>	 K8s shell is a bit buggy when you first start it (the press enter to prompt part)
[20:21:07] <bd808>	 its not buggy so much as weird. :)
[20:21:12] <Krinkle>	 bd808: does that mean https://phabricator.wikimedia.org/T176886 is now resolved?
[20:21:16] <Zppix>	 Yeah maybe its just weird
[20:21:45] <bd808>	 Krinkle: they are still pointing to *.labsdb. I want to fix that
[20:21:52] <Krinkle>	 Right, good point.
[20:21:57] <bd808>	 but *.labsdb is now an alias for *.analytics.db.svc.eqiad.wmflabs
[20:22:14] <Krinkle>	 Hm.. right, so what is the desired outcome?
[20:22:27] <Krinkle>	 It'd be super nice if somehow it would expand to the one currently used to connect.
[20:22:37] <Krinkle>	 Or, alternatively, if it were only the prefix.
[20:22:58] <Krinkle>	 But the former seems an implementation nightmare for you, and the latter would be a breaking change :/
[20:23:16] <bd808>	 *nod* I think that ticket is the best we can do. data update plus schema change to add the raw slice label
[20:23:41] <Krinkle>	 maybe switch the current field to analytics as default, and add a new column that we encourage users to use. 
[20:23:48] <Krinkle>	 Yeah
[20:23:50] <bd808>	 yeah
[20:24:09] <bd808>	 and know that nobody with old code will really change ;)
[20:24:21] <bd808>	 s/nobody/few/
[20:26:21] <Krinkle>	 Yeah.
[20:30:54] <Zppix>	 Dont you love it when your doing something on ssh and your internet decides your done for the day?
[20:45:43] <maxlath[m]>	 hello! I'm new with the toolforge, I just started a web service with `webservice --backend=kubernetes nodejs start` and I'm looking for a way to access the service logs, but no success so far: any tip? :)
[20:47:57] <Zppix>	 maxlath[m]: usually its in your home directory, if not i think if you do kubectl get pods and then kubectl logs <podname> it should display them
[20:48:01] <chasemp>	 maxlath[m]: (slightly distracted but..): [as your tool] kubectl get pods; kubectl logs <mypod> or kubectl -f -f <mypod>
[20:48:07] <chasemp>	 those logs are ephemeral fyi
[20:48:13] <chasemp>	 but accessible during runtime
[20:48:53] <Zppix>	 I wonder how i ended up with logs in my home dir then... im using k8s
[20:50:40] <chasemp>	 ah I imagine lighttpd writes them directly
[20:51:07] <chasemp>	 a slight turn is stdout for SGE is directed on disk but I believe remains ephemeral in k8s land
[20:51:12] <chasemp>	 so it depends on what you are up to :)
[20:59:41] <maxlath[m]>	 Zppix: chasemp thx! I found the documentation mentioning `kubectl get pods` but the `kubectl` wasn't accessible: the reason seems to be that before starting the service, I had to run `webservice --backend=kubernetes nodejs shell` (to access a recent version of nodejs). After `exit`ing that shell, I now can find `kubectl` :)
[21:19:28] <doctaxon>	 mysqlconnect/db server: User 'u4802' has exceeded the 'max_user_connections' resource (curren
[21:19:31] <doctaxon>	 t value: 10)
[21:19:47] <doctaxon>	 10 is not enough for my bot jobs - what can I do?
[21:20:28] <doctaxon>	 chasemp: ^
[21:38:04] <Zppix>	 doctaxon: what all are you doingn?
[21:39:45] <doctaxon>	 Zppix: building and actualizing of hundreds of worklists in a time window in the early morning. 
[21:40:04] <doctaxon>	 it's quality management in dewiki
[21:41:30] <doctaxon>	 Zppix: and archiving of discussion and meta pages
[21:42:14] <Zppix>	 Could you maybe merge some of the queries into one?
[21:42:21] <doctaxon>	 i already did
[21:42:32] <doctaxon>	 it's not enough
[21:42:40] <doctaxon>	 10 are much to less
[21:43:46] <doctaxon>	 Zppix: The 10 max is new for some weeks, right?
[21:43:51] <Zppix>	 Maybe space the times the jobs run farther apart?
[21:43:56] <Zppix>	 I dont know
[21:44:43] <doctaxon>	 Maybe space the times the jobs run farther apart? <- i have done this too, it are too much tasks
[21:47:00] <doctaxon>	 Zppix: I wanted to talk chesemp <- maybe labs can do anything to help me
[21:47:13] <doctaxon>	 * chasemp
[21:51:15] <zhuyifei1999_>	 doctaxon: semaphore :)
[21:52:37] <Zppix>	 doctaxon: okay just wanted to see if i could help sorry
[21:53:29] <doctaxon>	 Zppix: I thank you nevertheless
[21:55:46] <zhuyifei1999_>	 this case is exactly what semaphores are designed for, in case you can’t get a quota increase
[21:56:14] <doctaxon>	 zhuyifei1999_: I already tried semaphore functions, possibly it's not optimized enough
[21:57:29] <doctaxon>	 but I have to shovel the queries within a time window, what makes semaphores a difficult way
[21:58:16] <zhuyifei1999_>	 hmm
[22:01:03] <doctaxon>	 okay, I'm trying something further, but thanks for the suggestions of you both
[22:06:11] <Amir1>	 !log wikilabels e7e7680 on staging (T183196)
[22:06:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[22:06:15] <stashbot>	 T183196: Deploy wikilabels mid-December - https://phabricator.wikimedia.org/T183196
[22:08:05] <maxlath[m]>	 Is there a trick to execute a command remotely identified as a tool? I would like to do something like `ssh mytool@login.tools.wmflabs.org "kubectl get pods"`
[22:09:29] <mutante>	 maxlath[m]: what's missing? the "become" line?
[22:11:29] <maxlath[m]>	 mutante: isn't the `become` command opening a sub shell?
[22:14:35] <bd808>	 doctaxon: you can open a phabricator task to ask for a higher limit for your user account. I'm in a meeting right now to point you to an example, but basically write a task explaining how many more connections you think you need, why, and tag it with cloud-services and dba.
[22:14:48] <bd808>	 we need to document a process for this on wikitech
[22:17:52] <Amir1>	 !log wikilabels e7e7680 on prod (T183196)
[22:17:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[22:17:55] <stashbot>	 T183196: Deploy wikilabels mid-December - https://phabricator.wikimedia.org/T183196
[22:18:41] <bd808>	 maxlath[m]: it is not documented anywhere, but it should work to do `ssh mytool@login.tools.wmflabs.org "become $MYTOOLNAME -- kubectl get pods"`
[22:19:08] <bd808>	 become is a sudo wrapper and passes extra args to the sudo exec
[22:19:40] <bd808>	 oh.. not mytool@ you@
[22:21:00] <maxlath[m]>	 bd808: it works!!! thanks :D 
[22:25:48] * maxlath[m] sent a long message: maxlath[m]_2017-12-18_22:25:47.txt <https://matrix.org/_matrix/media/v1/download/matrix.org/JFBXMjcPAlYAeBVELPUiPZeq>
[22:27:11] <bd808>	 maxlath[m]: hmmm... I haven't seen that before
[22:28:35] <bd808>	 that error seems to say that something died from a segmentation fault (errno=11)
[22:29:15] <Platonides>	 probably a consequence of the fist error
[22:29:23] <Platonides>	 it went into an untested branch
[22:31:38] * maxlath[m] sent a long message: maxlath[m]_2017-12-18_22:31:36.txt <https://matrix.org/_matrix/media/v1/download/matrix.org/LDoOZbFifkQFjKtXuzoswrsy>
[22:34:46] <bd808>	 maxlath[m]: is that all just from running kubectl get pods?
[22:36:10] <maxlath[m]>	 bd808: that's the output I got after entering the command `ssh login.tools.wmflabs.org "become hub -- kubectl get pods"` (where hub is the name of my project)
[22:37:51] <bd808>	 interesting. I'm able to run `ssh login.tools.wmflabs.org "become stashbot -- kubectl get pods"` repeatedly.
[22:38:09] <bd808>	 let me see if I can spot anything weird going on on tools-login
[22:38:10] <maxlath[m]>	 but for a moment, simply loging with `ssh login.tools.wmflabs.org` and the trying to `become hub` was failing with the same `/etc/profile.d/Z99-cloud-locale-test.sh` errors
[22:39:02] <maxlath[m]>	 now I can do it repeatedly too
[22:40:41] <bd808>	 maxlath[m]: what it your local LC_ALL?
[22:41:52] <maxlath[m]>	 echo "$LC_ALL" outputs an empty line
[22:42:47] <bd808>	 hmmm... how about `ssh login.tools.wmflabs.org 'echo $LC_ALL'`?
[22:43:16] <maxlath[m]>	 empty too
[22:44:22] <bd808>	 that file that is the failing to fork reduces down to `locale 2>&1 | locale_warn`
[22:44:51] <bd808>	 Apparently if you touch ~/.cloud-locale-test.skip then it won't run anything at all
[22:45:08] <bd808>	 but that may just move the problem around somehow
[22:46:00] <bd808>	 shells into the bastions have some restrictions on resources applied to try and keep people from running away with processes on those shared hosts
[22:46:33] <bd808>	 but they should not be dialed down so tight as to make the startup scripts for the shells fail
[22:48:14] <bd808>	 !log tools.drini Killed several instances of ortog.sh running on tools-login. Please run scripts on the [[Help:Toolforge/Grid|Job Grid]].
[22:48:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.drini/SAL
[22:48:36] * bd808 feeds stashbot a cookie
[22:53:06] <maxlath[m]>	 not sure if it has anything to do with the issue above, but I have an container named `interactive` coming out of I don't know where, with for only logs `Unrecognized input header: 116`
[22:53:18] <maxlath[m]>	 Is there something like `kubectl rm interactive` to stop it?
[23:01:02] <bd808>	 the "interactive" container would be from `webservice --backend=kubernetes ... shell`
[23:03:03] <bd808>	 maxlath[m]: you should be able to kill it with `kubectl delete pod/interactive` I think.
[23:04:24] <maxlath[m]>	 bd808: thx, it worked
[23:15:10] <Platonides>	 /78/77