[00:42:36] <shilad>	 Hi all, I am having trouble connecting to my open cloud instance and wonder if somebody can help me troubleshoot.
[00:42:50] <shilad>	 The instance was created in https://phabricator.wikimedia.org/T161554
[00:43:29] <shilad>	 I've uploaded my ssh keys and can connect to other WMF servers (not in my opencloud project).
[00:50:30] <shilad>	 ssh session debug info follows: 
[00:50:47] <shilad>	 https://www.irccloud.com/pastebin/CNI0azkb/ssh-session-logs
[00:50:48] <Krenair>	 shilad, what other servers can you connect to?
[00:51:13] <shilad>	 stat1004.eqiad.wmnet, for example.
[00:51:20] <shilad>	 My ssh config and key are different for those.
[00:51:28] <Krenair>	 wmnet stuff is a separate administrative domain
[00:51:36] <shilad>	 But I am not sure if the problem is with my client ssh config or my instance ssh config.
[00:51:50] <Krenair>	 debug1: Executing proxy command: exec ssh -a -W wikibrain-embeddings-01.wikibrain.eqiad.wmflabs:22 bast1001.wikimedia.org
[00:51:53] <Krenair>	 this can not work
[00:52:23] <shilad>	 ??
[00:52:25] <Krenair>	 bast1001 has no privileged access to the labs network
[00:52:49] <Krenair>	 your bastion for *.wmflabs should be a host like bastion.wmflabs.org
[00:52:57] <shilad>	 Aha! Thanks. I'm using the wrong proxy.
[00:54:06] <shilad>	 Still doesn't quite work: 
[00:54:13] <shilad>	 https://www.irccloud.com/pastebin/IepZha8X/ssh-session-log2
[00:54:24] <Krenair>	 ok
[00:54:24] <shilad>	 I checked to make sure I can ssh to bastion directly.
[00:54:33] <shilad>	 bastion is fine
[00:54:39] <shilad>	 BTW, thanks for the help, Krenair!
[00:55:03] <Krenair>	 krenair@bastion-01:~$ id a558989
[00:55:03] <Krenair>	 id: a558989: no such user
[00:55:24] <Krenair>	 do you have a user configured for bastion.wmflabs.org in your SSH config?
[00:55:26] <shilad>	 in my ssh config I have User shiladsen
[00:55:31] <shilad>	 yes!
[00:55:44] <Krenair>	 can you paste the wikimedia section of your ssh config file?
[00:57:34] <shilad>	 Apparently I *do* need my username to connect to bastion. Sorry about that. So something is wonky about my config.
[00:57:46] <shilad>	 Here's the part of my ssh config:
[00:57:52] <shilad>	 https://www.irccloud.com/pastebin/avwMmLCh/shilads-ssh-config
[00:58:10] <Krenair>	 yeah you'll need another part that's:
[00:58:14] <Krenair>	 Host bastion.wmflabs.org
[00:58:22] <Krenair>	 User shiladsen
[00:58:23] <shilad>	 BTW, thanks for the help, Krenair. I fought with this for 45 minutes before giving up!
[00:58:28] <Krenair>	 etc.
[00:58:34] <Krenair>	 or just set the username in that ProxyCommand
[00:59:03] <shilad>	 Success!! Thank you so much.
[00:59:29] <Krenair>	 no problem
[01:01:10] <Krenair>	 I've seen quite a few people miss bastion when setting up SSH config
[01:01:22] <Krenair>	 or making it get into a mess with trying to get to the bastion... via the bastion
[01:01:43] <Krenair>	 though I don't think I've seen many people who can SSH into wmnet but haven't done wmflabs before :)
[01:03:11] <shilad>	 Hah! Glad I could show you something new. Could I ask one follow up question?
[01:03:15] <Krenair>	 sure
[01:03:18] <shilad>	 In the ticket is says "Note that by default most of the disk space is not mounted for a new VM. To partition that space you'll need to apply something like the role::labs::lvm::srv puppet class."
[01:03:31] <shilad>	 Could you point me in some direction that would help me understand how to do this.
[01:03:43] <Krenair>	 there's a couple of ways
[01:04:03] <shilad>	 I found the puppet config in horizon.
[01:04:14] <Krenair>	 oh okay
[01:04:25] <Krenair>	 you're pretty much there then :)
[01:04:32] <Krenair>	 you should see an option for role::labs::lvm::srv on there?
[01:04:37] <shilad>	 Do I just need the single puppet class, or do I also need the mount poitn?
[01:05:38] <Krenair>	 hmm
[01:06:39] <Krenair>	 I think it just goes to /srv ?
[01:06:49] <shilad>	 Awesome. I will try that.
[01:09:25] <shilad>	 Krenair: I don't see anything big mounted... maybe I have access to the device but I have to mount it myself?
[01:09:40] <Krenair>	 did you run puppet on the instance?
[01:09:56] <shilad>	 Oh boy... I don't know, I guess is the right answer?
[01:10:17] <Krenair>	 it should run automatically from time to time, but you can run it manually with: sudo puppet agent -tv
[01:10:21] <shilad>	 I think I need to go read about puppet...
[01:11:03] <shilad>	 That did it, though. Hooray!
[01:11:09] <shilad>	 I'm all set. Thanks again for your help!
[01:11:18] <Krenair>	 puppet is the configuration management system in use here
[01:12:12] <Krenair>	 one writes some files that basically describe what should be set up - install package w, file at path x with contents y, ensure service z is running
[01:13:03] <shilad>	 I've used docker, so I get it conceptually. Thanks!
[01:13:18] <Krenair>	 you include this stuff in a type of class we call a role (or profile, turns out it's more complicated since I frequently wrote this stuff), and attach the role to the target node (instance, in the case of labs)
[01:13:41] <Krenair>	 then puppet running on the instance will pick up the new stuff
[01:15:02] <Krenair>	 useful for running lots of servers that are identical, or servers that should be easily replaceable (i.e. you can easily spin up a new one configured in exactly the same way), etc. etc.
[01:15:51] <Krenair>	 there's not (necessarily) any containers involved
[01:16:13] <shilad>	 That's great. Thank you!!
[12:08:13] <Samwalton9>	 Anyone around? I've started a webservice, which didn't start up properly (segmentation fault, looking into it), but I now can't get its status or stop it; I get a similar traceback to T156605.
[12:08:13] <stashbot>	 T156605: "webservice shell" fails with "No such file or directory" (with php5.6) - https://phabricator.wikimedia.org/T156605
[12:16:13] <Samwalton9>	 Nevermind, think I've sorted it.
[12:30:23] <arturo>	 Samwalton9: hey :-) if you have any additional information, it would be great if you follow-up in that ticket
[12:32:01] <Samwalton9>	 It was a PEBKAC error ;)
[14:08:50] <arturo>	 !log Tools T181647 aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --test'
[14:08:50] <stashbot>	 arturo: Unknown project "Tools"
[14:08:50] <stashbot>	 T181647: create 'attended' upgrade workflow for cloud with Toolforge as canonical case - https://phabricator.wikimedia.org/T181647
[14:09:47] <arturo>	 !log tools T181647 aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --test'
[14:09:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:04:50] <andrewbogott>	 !log tools depooling exec-manage tools-exec-1430.  Experimenting with purge-old-kernels
[15:04:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:15:40] <andrewbogott>	 !log tools repooling exec-manage tools-exec-1430. 
[15:15:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:15:52] <andrewbogott>	 !log tools running purge-old-kernels on all Trusty exec nodes
[15:15:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:33:36] <Nudin>	 Technical Advice IRC meeting starting in 30 minutes in channel #wikimedia-tech, hosts: @addshore & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[16:28:03] <wikibugs>	 (03PS1) 10Ottomata: Update secrets/certificates with deployment-prep certs for TLS Kafka [labs/private] - 10https://gerrit.wikimedia.org/r/404706 (https://phabricator.wikimedia.org/T121561)
[16:29:20] <wikibugs>	 (03PS2) 10Ottomata: Update secrets/certificates with deployment-prep certs for TLS Kafka [labs/private] - 10https://gerrit.wikimedia.org/r/404706 (https://phabricator.wikimedia.org/T121561)
[16:29:31] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Update secrets/certificates with deployment-prep certs for TLS Kafka [labs/private] - 10https://gerrit.wikimedia.org/r/404706 (https://phabricator.wikimedia.org/T121561) (owner: 10Ottomata)
[16:29:53] <Sagan>	 can somebody help me with puppet?
[16:30:01] <Sagan>	 luke081515@xenon:~$ sudo puppet agent -tv
[16:30:02] <Sagan>	 Error: Could not initialize global default settings: Cannot set modulepath settings in puppet.conf
[16:33:38] <paladox>	 Sagan hi, is this on a puppetmaster?
[16:33:54] <Sagan>	 paladox: no, a normal instance with default master, not special rule
[16:34:00] <paladox>	 oh
[16:34:09] <paladox>	 Could you paste your puppet.conf please
[16:34:14] <paladox>	 in /etc/puppet/puppet.conf
[16:34:59] <chasemp>	 to my knowledge the puppetmasters for cloud are not having issues fwiw
[16:35:32] <Sagan>	 paladox, chasemp: https://pastebin.com/mE6Wgjde
[16:35:48] <paladox>	 that looks like a puppet master
[16:35:51] <paladox>	 config Sagan
[16:36:08] <Sagan>	 paladox: it was one, once, but that role got disabled long time ago IIRC
[16:37:28] <paladox>	 Sagan try https://phabricator.wikimedia.org/P6605
[16:37:42] <Sagan>	 paladox: you mean just replacing that?
[16:37:46] <paladox>	 yep
[16:41:08] <bd808>	 Sagan: I think paladox is on the right track there. The puppet.conf you pasted looks like it got messed up at some point. Switching back from a project local/self-hosted puppetmaster to the global ones always takes some manual steps.
[16:41:08] <Sagan>	 paladox: it's running now, thx
[16:41:20] <paladox>	 Your welcome :).
[16:41:26] <Sagan>	 :)
[16:43:24] <chasemp>	 tx paladox 
[16:43:33] <paladox>	 Your welcome :) :)
[17:55:57] <arturo>	 !log tools aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo report-pending-upgrades -v' | tee pending-upgrades-report.txt
[17:56:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:01:09] <arturo>	 andrewbogott: ^^^ when running the report script, I see some instances which are doing weird dpkg operations, I would say configuring half-configured pending packages 
[18:08:08] <arturo>	 andrewbogott: this is what i mean
[18:08:15] <arturo>	 https://www.irccloud.com/pastebin/Y9MOR6pe/
[18:08:46] <arturo>	 its very weird, as I think the terminal is mixing stderr/stdout? I'm not sure
[18:09:30] <arturo>	 my fear is that by means of this some instances have been put in any inconsistent state
[18:09:33] <arturo>	 chasemp: what do you think?
[18:09:55] <arturo>	 some instances could have been put*
[18:14:11] <andrewbogott>	 Arturo, i can't make any sense of that paste on my phone but will be back at my laptop soon
[18:14:27] <arturo>	 ok andrewbogott no
[18:14:29] <arturo>	 np*
[18:23:24] <chasemp>	 arturo: I'm entirely sure what I'm looking at, can you filter out the odd nodes?
[18:25:17] <arturo>	 let me try, since the output was not stored with tee (stderr?)
[18:26:57] <chasemp>	 arturo: it's entirely possible some batch of nodes will be an crazy state and we'll ahve to do a round of cleanup
[18:27:08] <chasemp>	 but I'm not sure what I'm looking at there
[18:27:20] <andrewbogott>	 arturo: what's an example of a host I should look at?
[18:28:04] <andrewbogott>	 probably related:  I ran a 'purge-old-kernels' batch on all exec nodes which seems to be hanging.   So anywhere that that's still in process would look bad to you.
[18:28:46] <arturo>	 andrewbogott: example node: tools-webgrid-generic-1402.tools.eqiad.wmflabs
[18:29:41] <andrewbogott>	 ok, so that's unrelated then
[18:30:13] <andrewbogott>	 arturo: what command did you run to get all that output above?
[18:30:49] <arturo>	 andrewbogott: aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo report-pending-upgrades -v' | tee pending-upgrades-report.txt
[18:31:33] <chasemp>	 'sudo report-pending-upgrades -v' should be fine on the local instance andrewbogott 
[18:31:37] <chasemp>	 afaiu
[18:32:19] <arturo>	 chasemp: I got a file: https://phabricator.wikimedia.org/P6606 lines with /usr/bin/dpkg
[18:34:06] <andrewbogott>	 ok, I guess I don't understand how to read this.  It's reporting lots of available packages to install (which is true on lots of hosts) but unclear to me if that's an error condition or just a suggestion that I run apt-get upgrade
[18:35:08] <arturo>	 andrewbogott: there were actual dpkg runs with --unpack --auto-deconfigure and stuff going on. I guess the script triggered some previous aborted updates?
[18:35:21] <arturo>	 the host list seems to be something like this:
[18:35:26] <arturo>	 https://www.irccloud.com/pastebin/KR4HS7LF/
[18:36:01] <arturo>	 (some names where mangled I guess because races in the output)
[18:37:17] <andrewbogott>	 arturo: can you give me an example command and what you'd expect it to do vs. what it actually does?
[18:37:27] <andrewbogott>	 for tools-webgrid-generic-1402
[18:39:30] <chasemp>	 ran 'sudo report-pending-upgrades -v' on bastion-03 and saw '/usr/bin/dpkg --force-confdef --force-confold --status-fd 9 --configure libc-bin:amd64' so I ran apt-get install libc-bin
[18:39:33] <chasemp>	 which seemed to work out
[18:40:00] <arturo>	 andrewbogott: since the dpkg operations were completed the time I run the script (above), now there are no pending dpkg operations, so it can't be reproduced? no idea
[18:40:08] <andrewbogott>	 ah, ok
[18:40:19] <andrewbogott>	 so that sounds like a win, then?  The script fixed things :)
[18:40:50] <andrewbogott>	 I think finding VMs with dpkg in an inconsistent state is not itself cause for alarm — there are lots of non-serious things that could get us there.
[18:40:51] <arturo>	 andrewbogott: well :-) I'm not sure. Why did we have half-configure packages?
[18:41:00] <chasemp>	 I'm sure it was a mistake somewhere along teh lines
[18:41:13] <chasemp>	 the apt situation here has been awful, stale, and not consistent for a long time
[18:41:52] <chasemp>	 arturo: in theory it looks like 'apt-upgrade trusty-wikimedia -v' does puppet since that's the only thing pending from trusty-wikimedia
[18:41:55] <chasemp>	 is that right?
[18:42:43] <chasemp>	 here = tools
[18:42:58] <arturo>	 andrewbogott: for the record, comparision: https://phabricator.wikimedia.org/P6606#37216
[18:43:15] <arturo>	 chasemp: in which host?
[18:44:20] <arturo>	 chasemp: in the case of tools-webgrid-generic-1402 these are the pending upgrades from trusty-wikimedia:
[18:44:28] <arturo>	 https://www.irccloud.com/pastebin/wNOlwqwP/
[18:44:47] <chasemp>	 arturo: ok so it's not even consistent across fun times, I was looking at bastion-03
[18:44:50] <arturo>	 so if you run 'apt-upgrade trusty-wikimedia -vs' it should report to upgrade all 3
[18:45:04] * chasemp nods
[18:45:22] <chasemp>	 arturo: can you run a report for pending trusty-wikimedia packages then?
[18:46:33] <arturo>	 sure
[18:46:50] <arturo>	 but we don't have a script for that, so it should be this oneliner: apt-show-versions | grep upgradeable | grep trusty-wikimedia
[18:47:18] * arturo running it
[18:47:57] <arturo>	 !log tools aborrero@tools-clushmaster-01:~$ clush -w @all 'apt-show-versions | grep upgradeable | grep trusty-wikimedia' | tee pending-upgrades-report-trusty-wikimedia.txt
[18:48:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:49:50] <arturo>	 chasemp: this is the report you asked for: https://phabricator.wikimedia.org/P6606#37217
[18:52:41] <arturo>	 I just discover that some instances have such an old version of python3-apt that my apt-upgrade script won't work (missing attributes in some classes)
[18:52:44] <chasemp>	 arturo: tx my mangling says https://phabricator.wikimedia.org/P6606#37219
[18:52:57] <chasemp>	 arturo: ah, do a clush mass upgrade of that then?
[18:53:04] <chasemp>	 hopefully it's availablef or trusty :)
[18:54:11] <arturo>	 chasemp: will try tomorrow after https://gerrit.wikimedia.org/r/#/c/404736/ is merged
[18:54:58] <arturo>	 does that works for your?
[18:55:06] <chasemp>	 arturo: cool, once you land that, clush out python3-apt
[18:55:16] <chasemp>	 then I think a apt-upgrade trusty-wikimedia
[18:55:24] <chasemp>	 will work out based https://phabricator.wikimedia.org/P6606#37219
[18:55:41] <chasemp>	 nothing I'm afraid of there 
[18:55:49] <chasemp>	 madhuvishy: hhvm on tools-checker...any reason not to upgrade that?
[18:56:34] <arturo>	 ok chasemp, will do tomorrow
[18:56:42] <chasemp>	 arturo: have a good night
[18:57:51] <arturo>	 thanks :-)
[19:04:38] <madhuvishy>	 chasemp: hhvm?
[19:05:00] <chasemp>	 madhuvishy: pending upgrade for hhvm package from trusty-wikimedia on tools-checker hosts in Tools.  does that concern you at all?
[19:05:09] <madhuvishy>	 aah
[19:05:21] <madhuvishy>	 no should be fine
[19:05:29] <chasemp>	 if not we'll hit the entire list at https://phabricator.wikimedia.org/P6606#37219
[19:05:37] <chasemp>	 madhuvishy: kk, just checking in
[19:08:33] <Zppix>	 !log git starting Maint. Window, shut down all hosts
[19:08:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[19:11:17] <Zppix>	 !log git maint. Window over hosts back up
[19:11:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[20:21:19] <andrewbogott>	 In about 90 minutes (at 16:00 PST) I'm going to reboot some more WMCS hosts.  This will interrupt nodepool and CI for a few minutes and also break wikitech and Horizon.
[21:27:59] <halfak>	 !log ores staging ores-wmflabs-deploy:96d7f12
[21:28:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[21:30:51] <halfak>	 !log ores deploying ores-wmflabs-deploy:96d7f12
[21:30:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[21:42:21] <halfak>	 !log ores deleted ores-misc-01
[21:42:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[21:44:46] <halfak>	 !log ores created ores-misc-01 as Debian Stretch instance
[21:44:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[22:00:10] <andrewbogott>	 system reboots happening NOW, horizon and wikitech will be down briefly
[22:07:21] <Zppix>	 ??
[22:07:25] <Zppix>	 Why?
[22:10:51] <halfak>	 Zppix, bug fix
[22:11:03] <halfak>	 Same thing that caused the alert wave in #wikimedia-ai yesterday
[22:11:26] <Zppix>	 I think yesterday was for meltdown kernal updates
[22:11:44] <Krenair>	 think this is just the same thing for different physical hosts
[22:11:52] <Zppix>	 Ah
[22:12:22] <Krenair>	 from the list it sounds like labnodepool1001, californium, silver
[22:13:02] <Krenair>	 possibly contint1001
[22:13:32] <Zppix>	 <3 wikimedia's naming schemes
[22:14:05] <chasemp>	 we targeted hosts with the most exposure first, and on down the list
[22:31:45] <andrewbogott>	 labnodepool looked to me like it was already done so I didn't reboot it. But otherwise, yes :)
[22:32:18] <wikibugs>	 (03Abandoned) 10Zppix: Wikimedia-AI host migration/cleanup [labs/icinga2] - 10https://gerrit.wikimedia.org/r/404584 (owner: 10Zppix)
[22:39:00] <cam11598>	 Hey I'm having problems getting toolforge to let me input a new ssh
[22:40:48] <madhuvishy>	 cam11598: are you trying to change your ssh key? Where are you trying this?
[22:41:31] <cam11598>	 Trying to add a new one and on https://toolsadmin.wikimedia.org/profile/settings/ssh-keys I generated it using PuttyGen (First time using Putty Gen)
[22:41:53] <cam11598>	 I keep getting " Invalid public key. " when inputting it
[22:43:59] <madhuvishy>	 cam11598: I see, couple ideas - one, may be the type of key (RSA/DSA/and a few other types we support) is mismatched, or may be you are copying your private key instead of the public key by mistake
[22:44:13] <cam11598>	 Let me try again one sec...
[22:45:07] <cam11598>	 Got it -.- putty gen's exxport pasting doesn't seem to like me
[22:45:12] * cam11598 is dumb
[22:45:14] <madhuvishy>	 aah
[22:45:28] <cam11598>	 I usually use mac so this is easily avoided -.-
[22:46:11] <madhuvishy>	 right :) Putty contributes to a high number of our support questions :D
[23:01:54] <cam11598>	 !log loltrs killed & restarted
[23:01:55] <stashbot>	 cam11598: Unknown project "loltrs"
[23:01:55] <stashbot>	 cam11598: Did you mean to say "tools.loltrs" instead?
[23:02:05] <cam11598>	 !log tools.loltrs killed & restarted
[23:02:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.loltrs/SAL
[23:07:55] <wikibugs>	 (03PS1) 10Gerrit Patch Uploader: Migrate wikimedia-ai hosts and remove some [labs/icinga2] - 10https://gerrit.wikimedia.org/r/404879
[23:07:58] <wikibugs>	 (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/icinga2] - 10https://gerrit.wikimedia.org/r/404879 (owner: 10Gerrit Patch Uploader)
[23:08:31] <wikibugs>	 (03CR) 10Zppix: [V: 032 C: 032] Migrate wikimedia-ai hosts and remove some [labs/icinga2] - 10https://gerrit.wikimedia.org/r/404879 (owner: 10Gerrit Patch Uploader)