[00:03:24] <xinbenlv>	 Great, I manage to get the kubernates deployed
[00:08:55] <chicocvenancio>	 xinbenlv: kubernetes in toolforge is not an easy way to setup cron jobs
[00:09:24] <chicocvenancio>	 the recommended way is to edit your tool crontab and use jsub to send the job to the grid engine
[00:18:40] <xinbenlv>	 didn't get the jsub up and running because I rely on node10 and environment variables
[00:31:10] <xinbenlv>	 <chicocvenancio> do I understand it correctly that the latest official release of kubernetes is currently at v1.13, while WMF's Toolforge's kubernetes is at version of v1.4? 
[00:31:24] <chicocvenancio>	 yes
[00:32:02] <chicocvenancio>	 xinbenlv: k8s image doesn't have node10 out of the box either
[00:32:27] <chicocvenancio>	 so, install nvm in your tool directory to get node10
[00:33:07] <chicocvenancio>	 create a script that sets your env variables, calls nvm to use node v10 and does your thing and send that script to the grid engine via jsub
[00:33:51] <xinbenlv>	 I managed to get my job working in node10 through this official docker  image: docker-registry.tools.wmflabs.org/toollabs-node10-base:latest
[00:34:04] <xinbenlv>	 But I was having a hardtime using kubernetes to run CronJob
[01:40:49] <chicocvenancio>	 it was in a really alpha stage in k8s 1.4
[01:57:20] <xinbenlv>	 is it anytime soon the wmf toolforge going to upgrade to latest k8s?
[01:58:17] <chicocvenancio>	 hopefully, but there are customizations that make it difficult and the team doesn't have many resources
[02:03:26] <chicocvenancio>	 https://phabricator.wikimedia.org/T214513 is the task for that
[02:09:50] <bd808>	 !log tools.itwiki Force killed job 1045009 (itwiki-orphanizerbot) which was stuck in deleting state
[02:09:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.itwiki/SAL
[02:37:39] <xinbenlv>	 what's the timezone of the toolforge crontab based at?
[02:37:47] <bd808>	 UTC
[02:38:40] <bd808>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Specifying_time_zones
[02:38:49] <bd808>	 xinbenlv: ^
[03:05:33] <xinbenlv>	 ok
[03:15:21] <xinbenlv>	 crontab not working 
[03:15:35] <xinbenlv>	 it looks like this 
[03:15:47] <xinbenlv>	 crontab -e https://www.irccloud.com/pastebin/28EgUaky/mycrontab
[03:18:21] <xinbenlv>	 not seems to load it
[03:27:11] <xinbenlv>	 `jsub: error: argument program: Program '/usr/local/bin/kubectl' not found.`
[03:27:15] <xinbenlv>	 says this 
[08:30:23] <xinbenlv>	 I end up getting it to run on Google Cloud
[10:47:17] <danmichaelo>	 Hi. I'm in the process of migrating my tools away from Trusty. My plan was to use Kubernetes when possible, but it's a bit off-putting to get an older environment (jessie with Python 3.4, which is already end of life) than when using the gridengine (stretch with Python 3.5). What is the status of adding updated Docker images?
[10:47:39] <danmichaelo>	 (on Toolforge)
[11:09:37] <gtirloni>	 danmichaelo: very good point, just created T219091
[11:09:38] <stashbot>	 T219091: toolforge: Update python image to Stretch - https://phabricator.wikimedia.org/T219091
[11:10:08] <gtirloni>	 danmichaelo: you can also use the python35 image
[11:11:20] <danmichaelo>	 gtirloni: Thanks! Yes, for scripts, I guess, but webservice seems to only support "python" (3.4) and "python2"
[11:42:50] <gtirloni>	 yeah, I think someone is working on it and it's not public yet.. I just see python35 in our repo, CC'ed the relevant people
[12:07:00] <chicocvenancio>	 gtirloni: yeah bd.808 is working on that. 
[12:08:04] <gtirloni>	 cool
[12:29:24] <chicocvenancio>	 http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-cloud/20190322.txt [18:01:56] <bd.808>	 pin.toch: we have a python3.5 Kuberentes container built, but it is not exposed for use yet. I expect to make it available in the first week of April
[13:05:19] <danmichaelo>	 chicocvenancio: Thanks for linking to the discussion, good to see there is activity on this
[13:06:11] <chicocvenancio_>	 Yeah, saving some big problems it will be done in a few weeks
[13:12:25] <thedj>	 hmm. weird.. my / filled up, yet i couldn't find where.. reboot. 12GB free.
[13:13:01] <thedj>	 wasn't /tmp, nor /var... 
[13:13:19] <chicocvenancio>	 thedj: perhaps a file handle by an open process? 
[13:15:00] <chicocvenancio>	 Cyber.power678 had an issue with improperly redirected logs to /dev/null that did just that
[13:16:49] <thedj>	 whats the best way to find something like that ?
[13:17:51] <chicocvenancio>	 lsof should give you open file handles and their sizes
[13:18:21] <chicocvenancio>	 Then you have to investigate why that process is creating/keeping that
[13:41:23] <Zoranzoki21>	 !help Hi, what happens here: https://i.snag.gy/3lvdCF.jpg
[13:41:23] <wm-bot>	 Zoranzoki21: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[13:42:04] <thedj>	 Hi, who are you and why should i click your link ;)
[13:43:20] <Zoranzoki21>	 thedj: It is question for cloud services team :)
[13:43:25] <thedj>	 serious question. I don't just click links (virusses are everywhere)
[13:43:45] <Zoranzoki21>	 thedj: It is picture of problem which I found at tools.wmflabs.org/replag
[13:44:19] <Zoranzoki21>	 I understand you in this case
[13:46:32] <thedj>	 looks like a number conversion issue
[13:46:42] <thedj>	 that's close to a 64bit number
[13:47:05] <Zoranzoki21>	 thedj: Looks so, at console everything is 0
[13:47:24] <Zoranzoki21>	 I think to restart of tool maybe can help
[13:48:46] <thedj>	 ping bd808 as owner of the tool
[13:50:55] <Zoranzoki21>	 I cloned repository of tool at my tool account and started it.. Same behaviour: https://tools.wmflabs.org/ircclient/
[13:52:14] <Zoranzoki21>	 I will open task on Phabricator
[16:50:10] <bstorm_>	 Please make a ticket for that? Zoranzoki21
[16:50:30] <bstorm_>	 Oh I see you mentioned you are going to :)
[16:50:46] * bstorm_ vanishes again
[17:06:53] <SQL>	 Whois tool (https://tools.wmflabs.org/whois/) is giving a 500 error. I've had to restart the webservice on a couple of my tools recently. Might need done there too?
[20:38:11] <bd808>	 !log tools.replag Restarted webservice. NFS hiccup had broken ability of PHP process to read database credentials.
[20:38:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.replag/SAL
[20:44:55] <legoktm>	 !log tools.ci migrated to stretch grid/webservice
[20:44:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ci/SAL
[20:51:11] <legoktm>	 !log tools.coverage migrated to stretch grid
[20:51:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.coverage/SAL
[20:52:23] <legoktm>	 !log tools.dump-torrents migrated to k8s webservice
[20:52:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dump-torrents/SAL
[20:54:18] <legoktm>	 bd808: is gridengine-status purposely running its webservice on the grid still?
[20:54:26] <legoktm>	 trusty grid*
[20:54:28] <bd808>	 legoktm: yes
[20:54:55] <bd808>	 the only way to read the status of the trusty grid is to be attached to it
[20:55:04] <legoktm>	 got it
[20:57:42] <legoktm>	 !log tools.extreg-wos migrated to stretch grid
[20:57:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.extreg-wos/SAL
[21:05:20] <legoktm>	 !log tools.ninthcircuit migrated to stretch grid
[21:05:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ninthcircuit/SAL
[21:10:12] <legoktm>	 !log tools.phpcs migrated to stretch grid
[21:10:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phpcs/SAL
[21:32:21] <legoktm>	 !log tools.slow-parse disabled slow-parse tool, moved to k8s backend
[21:32:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.slow-parse/SAL
[21:35:30] <legoktm>	 !log tools.scotus-recordings migrated to stretch grid
[21:35:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.scotus-recordings/SAL
[21:37:09] <legoktm>	 !log tools.vendor migrated to stretch grid
[21:37:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.vendor/SAL
[21:37:13] <JJMC89>	 legoktm: Since you're around, are you able to restart the webservice for the whois tool?
[21:37:29] <legoktm>	 lemme see
[21:38:05] <legoktm>	 huh
[21:38:09] <legoktm>	 I think it's the ldap failure
[21:39:11] <legoktm>	 !log tools.whois restarting webservice, outputting HTTP 500
[21:39:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.whois/SAL
[21:39:51] <legoktm>	 JJMC89: looks operational now
[21:40:03] <JJMC89>	 Thanks legoktm!
[21:40:22] <xinbenlv>	 Should i ask lua question here?
[21:40:31] <legoktm>	 bit weird that it's a python tool running on php5.6
[21:40:38] <legoktm>	 xinbenlv: sure? 
[21:42:55] <JJMC89>	 ¯\_(ツ)_/¯ don't know anything about the tool, just a frequent user
[21:44:44] <legoktm>	 !log tools.wikitext-deprecation migrated to stretch grid
[21:44:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikitext-deprecation/SAL
[21:45:19] <legoktm>	 all of my tools are migrated over, now it's time for the bots
[21:46:06] <legoktm>	 bd808: most of my jsub commands still have -l release=trusty in them, is that OK to leave in?
[21:47:20] <bd808>	 legoktm: they will run with it, but I think it will put a warning in the the err log at job start
[21:48:01] <legoktm>	 ok
[21:49:34] <legoktm>	 !log tools.tfaprotbot migrated to stretch grid
[21:49:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tfaprotbot/SAL
[21:51:24] <legoktm>	 !log tools.referencebot disabled crontab, bot hasn't edited since 2017. archived to ~/crontab.backup
[21:51:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.referencebot/SAL
[21:52:50] <legoktm>	 :(