[09:03:07] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/408979 (owner: 10L10n-bot)
[13:49:37] <chasemp>	 Puppet is failing and we know about it :)
[13:53:21] <arturo>	 !log tools deploy https://gerrit.wikimedia.org/r/#/c/407465/ which is causing some puppet issues. Investigating.
[13:53:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:22:34] <arturo>	 !log tools it was some kind of transient error. After a second puppet run across the fleet, all seems fine
[14:22:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:52:39] <thedj>	 hmmm
[15:52:39] <thedj>	 -rwxr-xr-x 1 root root 2245 Oct 22  2013 /usr/share/npm/bin/npm-cli.js
[15:52:45] <thedj>	 maybe time to update npm ? :)
[15:53:08] <thedj>	 node -v
[15:53:08] <thedj>	 v0.10.25
[15:53:10] <thedj>	 wow...
[15:53:51] <chicocvenancio>	 Do we have .deb more current? 
[15:55:25] <thedj>	 yeash debian... npm: package is 3 years old (consider removal?)
[15:56:17] <thedj>	 i think we should have newer versions in wmf repo...
[15:57:41] <chicocvenancio>	 Do we already have a task about this? thedj 
[15:57:55] <thedj>	 no, i intend to write one.
[15:58:54] <chicocvenancio>	 Cool, I will look into it later today if no one else has taken it up by then
[16:01:13] <paravoid>	 pretty sure there's one somewhere (for prod)
[16:02:16] <thedj>	 oh wait, but tools is ubuntu right ?
[16:47:15] <chicocvenancio>	 thedj: bastion hosts are 
[16:49:15] <arturo>	 thedj: you can see here the base images for each host: https://tools.wmflabs.org/openstack-browser/project/tools (cc chicocvenancio )
[16:49:18] <thedj>	 hmm, https://phabricator.wikimedia.org/T169451 is related for kubernetes
[16:50:18] * chicocvenancio is too slow on mobile while burping the sleepy baby
[16:52:09] <chicocvenancio>	 thedj: are there npm jobs in grid? 
[16:52:51] <thedj>	 npm jobs i don't think so, that's usually for installation mostly. node jobs.. possibly
[16:53:48] * chicocvenancio nods 
[16:56:15] <thedj>	 https://tools.wmflabs.org/admin/tools#!/search/node
[17:01:15] <zhuyifei1999_>	 thedj: yeah for newer node jobs use k8s for the time being
[17:01:42] <zhuyifei1999_>	 they run on debian jessie
[17:05:54] <bd808>	 supporting NodeJS/NPM on Trusty is not something we are going to do. We have Jessie with newer versions in the Kubernetes cluster today, and hopefully we will make time to add Stretch with even newer versions "soon"
[17:06:19] <bd808>	 the Foundation's nodejs usage is on jessie/stretch
[17:07:07] <thedj>	 i want it on bastion :)
[17:07:34] <thedj>	 as in, i want my npm install to work, without having to write a script around everything to run it as a job :)
[17:07:56] <thedj>	 a onetime job..
[17:09:28] <bd808>	 not incredibly intuitive, but `webservice --backend=kubernetes nodejs shell` will get you an interactive session in a kubernetes pod that is running jessie and a not-as-ancient nodejs + npm
[17:09:59] <bd808>	 that session will have your tool's $HOME mounted jsut like the bastion does
[17:11:01] <arturo>	 !log tools depool tools-exec-1401.tools.eqiad.wmflabs to do some package upgrades
[17:11:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:15:46] <arturo>	 !log tools aborrero@tools-exec-1401:~$ sudo apt-upgrade upgrade trusty-wikimedia -vy
[17:15:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:20:07] <arturo>	 !log tools aborrero@tools-exec-1401:~$ sudo apt-upgrade upgrade trusty-updates -vy
[17:20:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:23:10] <arturo>	 !loog tools aborrero@tools-exec-1401:~$ sudo apt-upgrade upgrade trusty-tools -vy
[17:29:47] <arturo>	 !log tools repool tools-exec-1401.tools.eqiad.wmflabs. Package upgrades were OK.
[17:29:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:02:29] <arturo>	 !log tools cordon tools-paws-worker-1019 to do some package upgrades
[18:02:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:06:25] <arturo>	 !log tools aborrero@tools-paws-worker-1019:~$ sudo apt-upgrade upgrade stretch-wikimedia -v
[18:06:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:08:59] <arturo>	 !log tools aborrero@tools-paws-worker-1019:~$ sudo apt-upgrade upgrade stable -v
[18:09:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:10:16] <arturo>	 !log tools uncordon tools-paws-worker-1019. Package upgrades were OK.
[18:10:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:28:24] <arturo>	 !log tools cordon & drain tools-worker-1002.tools.eqiad.wmflabs
[18:28:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:32:42] <chasemp>	 tx arturo 
[18:33:07] <arturo>	 👍
[18:33:52] <arturo>	 !log tools aborrero@tools-worker-1002:~$ sudo apt-upgrade -u upgrade oldstable -v
[18:33:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:35:53] <arturo>	 !log tools aborrero@tools-worker-1002:~$ sudo apt-upgrade -u upgrade jessie-wikimedia -v
[18:35:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:38:14] <arturo>	 !log tools aborrero@tools-k8s-master-01:~$ sudo kubectl uncordon tools-worker-1002.tools.eqiad.wmflabs
[18:38:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:02:14] <Zoranzoki21>	 !help Can you please do on my vps sync; echo 3 > /proc/sys/vm/drop_caches
[22:02:14] <wm-bot5>	 Zoranzoki21: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[22:02:47] <paladox>	 that cleans the ram.
[22:02:52] <bd808>	 Zoranzoki21: which vm?
[22:03:05] <Zoranzoki21>	 bd808: command is sync; echo 3 > /proc/sys/vm/drop_caches 
[22:03:14] <Zoranzoki21>	 zoranzoki21@tools-bastion-03
[22:03:38] <bd808>	 heh. not quite your VPS then
[22:04:05] <bd808>	 why do you want us to do that to the shared bastion?
[22:05:05] <Zoranzoki21>	 bd808: nothing than. Thanks anyway
[22:05:26] <bd808>	 that was ... weird
[22:07:44] <chicocvenancio>	 Indeed
[22:08:07] <bd808>	 there is a ton of crap running there that shouldn't be
[22:09:18] <bd808>	 including celery! ffs
[22:09:36] <chicocvenancio>	 Yes
[22:09:55] <chicocvenancio>	 I've been trying to prod user into stopping it
[22:10:08] <bd808>	 I just killed it
[22:10:11] <chicocvenancio>	 And killing the most resource intensive ones
[22:10:48] <chicocvenancio>	 I still see one pwb bot
[22:11:41] <chicocvenancio>	 Never minf
[22:11:42] <bd808>	 ps list looks a bit better now
[22:17:20] <harej>	 We should have a counter every time we have to kill processes on bastion or reboot the bastion
[22:17:37] <paladox>	 harej, starting now, that's one :)
[22:17:46] <harej>	 Or like a swear jar. But for bastion job killing.
[22:18:36] <paladox>	 lol.
[22:18:45] <paladox>	 harej who will be counting though?
[22:18:54] <bd808>	 I'm about ready to write a chaos bot that just kills anything that looks suspicious
[22:19:56] <paladox>	 bd808 if you write one of those, that will free up time that would have been spent killing processes or looking into them :)
[22:20:26] <bd808>	 but take up time with people jumping in here to say that their tools are broken ;)
[22:20:41] <paladox>	 lol
[22:21:52] * bd808 goes back to annual planning
[22:36:01] <primefac>	 !help I'm having some issues logging into wmflabs with putty
[22:36:01] <wm-bot5>	 primefac: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[22:36:30] <primefac>	 I've got the keygen set up, am using Pageant, but after I put in my passphrase it just gives me a 01"Server unexpectedly closed network connection"01 fatal error
[22:37:10] <bd808>	 primefac: what server are you trying to log into?
[22:37:17] <primefac>	 bastion
[22:38:35] <chicocvenancio>	 primefac: what's your shell username? 
[22:38:40] <primefac>	 primefac
[22:39:13] <bd808>	 it looks like your account is connected to Toolforge, but not any other VPS projects. That means you should connect to tools-login.wmflabs.org or tools-dev.wmflabs.org 
[22:40:10] * bd808 guesses that isn't clear in the docs on wikitech
[22:40:16] <primefac>	 thanks!
[22:40:40] * chicocvenancio goes check the docs
[22:40:57] <SQL>	 [[Help:Putty]] does say to use bastion heh
[22:41:37] <bd808>	 yeah, it used to be easy to do that, but we stopped adding everyone to the bastion group
[22:41:59] <bd808>	 Help:Putty could use a complete revamp
[22:42:09] <chicocvenancio>	 Help:Putty is for Cloud VPS, no?
[22:42:29] <bd808>	 its got screen shots for the datacenter we shutdown 4 years ago :)
[22:42:40] <SQL>	 I'm working on one for toolforge at the moment
[22:42:47] <bd808>	 chicocvenancio: its certainly written that way today
[22:42:54] <bd808>	 SQL: oh? nice
[22:43:37] <bd808>	 most of the stuff is the same so maybe we can refactor things and just call out the bits that are different
[22:43:40] <SQL>	 Yeah, shouldn't take very long. Is pagent really needed? I can login fine without it (I don't usually use puTTY, I just use ssh from linux commandline)
[22:44:05] <bd808>	 having one maintained page is better than 2 unmaintained ones :)
[22:44:43] <bd808>	 pagent is helpful if you are using putty's cli to do scp transfers
[22:44:50] <SQL>	 ah
[22:44:52] <bd808>	 (or at least it used to be)
[22:45:08] <chicocvenancio>	 also needed for accessing instances
[22:45:19] <bd808>	 I have to confess to not using Windows in a very long time. The things I know may be very stale
[22:45:21] <SQL>	 Usually if I'm on windows I'll use SmarTTY which is way way way way better IMO
[22:45:32] <chicocvenancio>	 at least how it is written at the moment
[22:46:05] <chicocvenancio>	 (I think you could "hardcode" the key in the proxy command, but it is written to use pagent
[22:46:19] <paladox>	 native openssh is heading to windows, so that should at least make things more compatible
[22:46:32] <chicocvenancio>	 don't know which option is more user friendly
[22:46:54] <bd808>	 ssh tunnels are kind of an "advanced" use case
[22:47:18] <bd808>	 useful for sure, but maybe not needed in the basic "how to use ssh" page
[22:47:57] <chicocvenancio>	 go to bastion and ssh into VPS instance from there?
[22:48:14] <paladox>	 https://blogs.msdn.microsoft.com/powershell/2017/12/15/using-the-openssh-beta-in-windows-10-fall-creators-update-and-windows-server-1709/
[22:48:55] <bd808>	 chicocvenancio: oh. that kind of proxy command. right
[22:49:28] <bd808>	 that bit is not needed for Toolforge users. We don't have jump servers in front of the Toolforge submit hosts
[22:49:44] * chicocvenancio nods
[22:50:08] <bd808>	 maybe that means that 2 pages would be useful after all :)
[22:50:10] <Krenair>	 why did we stop adding everyone to bastion?
[22:50:24] <bd808>	 Krenair: I .. don't know.
[22:51:15] <bd808>	 I know it moved from OpenStackManger to Keystone triggers, but I don't know why we stopped putting tools members in there. Maybe jsut to keep the member list smaller?
[22:52:21] <bd808>	 there have been grand debates about the utility vs security of making the tools submit hosts public rather than behind a jump server
[22:53:07] <Krenair>	 submit hosts being tools-bastion machines?
[22:53:36] <bd808>	 Krenair: yeah
[22:54:34] <Krenair>	 the utility is that people who haven't yet figured out how to proxy SSH can still get to them
[22:55:25] <Krenair>	 a disadvantage of this is that now you have two sets of bastion servers lying around, some outside the bastion project, and people get confused
[22:55:52] <Krenair>	 an advantage of this is that you don't have people who couldn't set up SSH proxying causing problems on the proper bastions
[22:59:09] <SQL>	 https://wikitech.wikimedia.org/wiki/User:SQL/ToolforgePutty Well, it's a start at least
[22:59:42] <chicocvenancio>	 SQL: we have https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP
[23:00:03] <SQL>	 haha welp
[23:00:22] * chicocvenancio is sorry for not pointing it out sooner
[23:00:25] <SQL>	 Thought we just had the way older ones that he was (and I also did the first time) tried
[23:00:28] <SQL>	 no worries
[23:01:04] <chicocvenancio>	 I can't make up my mind on what to do with Help:Putty though
[23:01:31] <chicocvenancio>	 because, to me https://wikitech.wikimedia.org/wiki/Help:Access_to_Cloud_VPS_instances_with_PuTTY_and_WinSCP is better
[23:01:51] <Krenair>	 I use a Windows client at work and prefer working with Git Bash (with the normal OpenSSH tools etc. - ssh, ssh-agent, scp) than PuTTY and friends
[23:04:26] <paladox>	 +1 to ^^ (that's when i used windows)
[23:05:01] <chicocvenancio>	 I'm stuck on windows due to the Brazilian health Ministry producing crappy software
[23:05:24] <bd808>	 chicocvenancio: its "interesting" that the pages have similar start dates and share many authors
[23:06:12] * chicocvenancio goes with putty, has cygwin installed as well
[23:06:22] <SQL>	 I needed something tabbed since I tend to have a lot of connections to machines, and terminals in each machine going at a time. https://i.imgur.com/TscFgdu.png
[23:06:25] <paladox>	 chicocvenancio which version of windows?
[23:07:20] <chicocvenancio>	 windows 10, thankfully the software is no longer crappy enough to demand outdated versions
[23:07:52] <bd808>	 heh. I think we can safely remove the "PuTTY 0.63 or later" box. 0.63 released in August 2013
[23:08:03] <paladox>	 chicocvenancio heh, you could use ubuntu on windows
[23:08:04] <paladox>	 or the other flavours
[23:08:20] <SQL>	 ^^^
[23:08:23] <paladox>	 (ubuntu on windows is a bash thingy that allows you to run ubuntu at the same time as windows so it's basically not a vm)
[23:10:11] <chicocvenancio>	 I use cygwin, or full VMs
[23:10:29] <paladox>	 heh
[23:10:52] <paladox>	 chicocvenancio i never really used linux in a vm unless i wanted to test something. It was always slow for me.
[23:11:16] <chicocvenancio>	 when cygwin is not enough it is usually because there is some reason I need a vm
[23:11:34] <chicocvenancio>	 like, cloning the operations/puppet repo
[23:11:38] <paladox>	 I never used cygwin either
[23:11:52] <paladox>	 as the ubuntu on windows replaced that
[23:11:58] <paladox>	 (also there's git for windows)
[23:12:07] <chicocvenancio>	 (there are files with asterisk in them, windows can not abide by that)
[23:12:22] <paladox>	 oh
[23:12:33] <paladox>	 symnlinks now work on windows so that's a + :)
[23:12:53] <chicocvenancio>	 mingw, cygwin and git went haywire when I tried to clone the repo
[23:13:21] <paladox>	 chicocvenancio heh?
[23:13:25] <paladox>	 What error did you get?
[23:14:26] <paladox>	 since migrating to a mac, everything became alot faster + i could run tests that i coulden't on windows. (Ie i could run things that would on run in linux)
[23:14:35] <chicocvenancio>	 don't remember, something that is was impossible to create the path, and git stopped in a weird state
[23:15:05] <paladox>	 which version of git?
[23:15:13] <chicocvenancio>	 https://stackoverflow.com/questions/26780189/cant-clone-git-repo-on-windows-when-some-file-names-contain-asterisk
[23:15:21] <Krenair>	 paladox, you realise labs instances are linux in a VM right?
[23:15:29] <paladox>	 Krenair yep
[23:15:31] <Krenair>	 ok
[23:16:25] <chicocvenancio>	 `git version 2.11.0.windows.1`
[23:16:37] <paladox>	 that's an old version
[23:16:45] <chicocvenancio>	 is it? 
[23:16:56] <chicocvenancio>	 at any rate it is a NTFS thing
[23:17:01] <paladox>	 chicocvenancio https://github.com/git-for-windows/git/releases
[23:17:10] <paladox>	 chicocvenancio version 2.16 is out
[23:17:24] <paladox>	 git for windows matches the releases for git for linux and other distos
[23:17:37] * chicocvenancio starts download
[23:17:57] <paladox>	 heh
[23:17:57] <chicocvenancio>	 will keep my vagrant boxes and cygwin configured though
[23:20:52] <paladox>	 chicocvenancio you should give ubuntu on bash a try. :)