[09:41:45] <Samwalton9>	 Hi! Is anyone around? I'm having some issues SSHing to a new horizon instance. I can connect to bastion and other projects no problem, but this is the first I've set up from scratch so I've probably missed something. Help:Access says to look for "“Finished puppet run”, BEGIN SSH HOST KEY FINGERPRINTS..." in the log, and I'm not seeing that. The project is wikicitevis :)
[10:10:18] <arturo>	 hi Samwalton9 
[10:11:05] <arturo>	 which is the error you see in your ssh client?
[10:11:44] <arturo>	 i.e, connection refused, timeout, etc
[10:15:07] <Samwalton9>	 Permission denied (publickey)
[10:15:46] <Samwalton9>	 I get it when going via the ProxyCommand method, or when connecting with agent forwarding via bastion (I can connect to bastion no problem)
[10:16:18] <arturo>	 2018-07-05T10:13:14.779200+00:00 wikicitevis-prod nslcd[432]: [45e146] <passwd="samwalton9"> (re)loading /etc/nsswitch.conf <--- I see this
[10:19:18] <Samwalton9>	 I'll be honest I'm not sure what that means :D
[10:23:40] <arturo>	 same for me :-)
[10:25:19] <Samwalton9>	 All I've done is set up the instance, add a web proxy, and tried to ssh via the two methods outlined in Help:Access
[10:25:30] <arturo>	 ok
[10:25:34] <arturo>	 I'm checking
[10:25:38] <Samwalton9>	 Thanks :)
[10:27:46] <arturo>	 I can't jump using ssh either :-/
[10:29:07] <arturo>	 [   22.463321] rc.local[385]: + puppet agent --onetime --verbose --no-daemonize --no-splay --show_diff --waitforcert=10 --certname=wikicitevis-prod.WikiCiteVis.eqiad.wmflabs --server=puppet2018-07-05T09:04:00.398176+00:00 wikicitevis-prod rc.local[385]: + puppet agent --onetime --verbose --no-daemonize --no-splay --show_diff --waitforcert=10 --certname=wikicitevis-prod.WikiCiteVis.eqiad.wmflabs --server=puppet
[10:29:07] <arturo>	 [   23.097142] rc.local[385]: [1;31mError: Could not initialize global default settings: Certificate names must be lower case; see #1168[0m
[10:29:15] <arturo>	 ok, this could be a clue
[10:29:34] <arturo>	 puppet didn't run because the project name
[10:29:40] <arturo>	 which must be all lowercase
[10:30:00] <arturo>	 we may require to delete the project and create it again
[10:30:42] <Samwalton9>	 ahh
[10:30:44] <Samwalton9>	 That
[10:30:48] <Samwalton9>	 *That's no problem
[10:32:55] <arturo>	 Samwalton9: deleting instances and project now
[10:33:03] <Samwalton9>	 Thanks!
[10:35:34] <arturo>	 did you create a proxy? I think I forgot to delete it
[10:35:44] <Samwalton9>	 Yeah
[10:36:00] <arturo>	 and I deleted the project already... hopefully openstack is smart enough to delete related resources
[10:36:04] <arturo>	 by himself
[10:36:08] <arturo>	 or no? no idea :-)
[10:36:28] <arturo>	 by itself*
[10:36:34] * Samwalton9 shrugs and hopes
[10:39:06] <arturo>	 Samwalton9: try again, I just created the new project
[10:43:24] <Samwalton9>	 Starting up a new instance, seeing a lot of log spam about certificate issues, not sure if that's expected or not
[10:46:32] * arturo looking
[10:47:39] <arturo>	 doesn't look good
[10:47:44] <arturo>	 let's wait a bit more
[10:48:16] <arturo>	 not sure how reliable is removing a project and then creating a new one with the same name just modifying the case
[10:49:07] <arturo>	 in the meantime, we could create a `wikicite-vis` project :-P
[10:49:25] <Hauskatze>	 'bis'
[10:49:50] <arturo>	 Hauskatze: is currently named `wikicitevis`
[10:50:06] <Hauskatze>	 then wikicitevis-bis :D
[10:50:10] <Samwalton9>	 yeah that's fine, the project name doesnt matter that much :D
[10:50:53] <arturo>	 Samwalton9: let's wait for 5 more minutes? or we could wait for andrewbogott to be awake in case he has some other idea on how to fix this
[10:52:59] <Hauskatze>	 I see labslogbot created on wikitech a page called 'wikicitevis' so as far as I can see the instance is correctly created
[10:53:26] <Hauskatze>	 wikicitevis-prod.wikicitevis.eqiad.wmflabs
[10:55:07] <Samwalton9>	 arturo, if you think renaming the project would work then I'm fine with that, but if you'd like to fix the problem more fully I'm happy to wait, there's no rush!
[10:57:52] <Hauskatze>	 Samwalton9: on openstack I see the instance is called wikicitevis-prod, can you try to connect there maybe? just guessing
[10:58:11] <Hauskatze>	 https://tools.wmflabs.org/openstack-browser/server/wikicitevis-prod.wikicitevis.eqiad.wmflabs
[10:58:40] <Samwalton9>	 Yeah I'm trying to connect to wikicitevis-prod.wikicitevis.eqiad.wmflabs and still getting the same issue
[11:08:19] <arturo>	 there is probably some issue in the labs-puppetmaster server with the renaming
[11:09:36] <arturo>	 but I see the cert in the puppetmaster
[11:09:36] <arturo>	   "wikicitevis-prod.wikicitevis.eqiad.wmflabs"           (SHA256) D7:A0:FB:DD:52:B6:82:67:8F:CD:F2:B1:88:68:9D:36:12:E8:B4:66:94:5A:79:CE:35:48:B7:97:89:FC:80:A8
[11:09:37] <stashbot>	 D7: Testing: DO not merge - https://phabricator.wikimedia.org/D7
[11:11:44] <arturo>	 I just ran `aborrero@labpuppetmaster1001:~$ sudo puppet cert sign wikicitevis-prod.wikicitevis.eqiad.wmflabs`
[11:11:50] <arturo>	 and things are now moving
[11:12:26] <Samwalton9>	 ooh
[11:12:27] <arturo>	 Samwalton9: can you connect now?
[11:12:49] <Samwalton9>	 Same error :(
[11:12:55] <arturo>	 ok wait
[11:13:01] <arturo>	 I will clean the certificate
[11:13:08] <arturo>	 please, remove the VM and create it again
[11:13:44] <arturo>	 Samwalton9: ^^^
[11:14:16] <Samwalton9>	 Launching...
[11:15:02] <arturo>	 hey andre__ :-)
[11:16:28] <arturo>	 Samwalton9: puppet seems to be doing actual things now
[11:16:34] <Samwalton9>	 Hooray!
[11:17:16] <arturo>	 Samwalton9: try now! hopefully it will work this time :-)
[11:17:21] <Samwalton9>	 I'm connected :D
[11:17:26] <arturo>	 :-D
[11:17:26] <Samwalton9>	 Thank you so much!
[11:17:27] <arturo>	 great!
[11:17:32] <arturo>	 thanks you Samwalton9 !!
[11:17:58] <Samwalton9>	 I'll make a note of this on the original project request task
[11:18:27] <arturo>	 ack
[11:19:00] <arturo>	 I also added a note in our docs: https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Projects_lifecycle&diff=1796215&oldid=1796056
[11:19:19] <Samwalton9>	 Awesome :)
[11:19:32] * arturo feeling useful today *^^*
[11:23:17] * andre__ waves
[12:15:00] <Amir1>	 !log wmam deleting the wikikids node due to request from WMAM and WMF legal
[12:15:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmam/SAL
[13:35:25] <andrewbogott>	 arturo, Samwalton9, did you delete all the VMs in the project before deleting the project?
[13:35:51] <arturo>	 andrewbogott: I would say so
[13:36:01] <arturo>	 Im not sure about the proxy
[13:36:35] <andrewbogott>	 and the project moved from lowercase to MixedCase?   or the other way around?
[13:36:41] <arturo>	 it is possible to have orphaned VM?
[13:36:50] <arturo>	 from mixed to loser
[13:36:53] <arturo>	 lower*
[13:37:05] <andrewbogott>	 arturo: yeah, I think it is possible to orphan a VM
[13:37:16] <andrewbogott>	 but I'd expect all lowercase to work better than CamelCase
[13:37:29] <andrewbogott>	 Are there still problems with the puppetmaster or is it mostly better now?
[13:37:38] <arturo>	 it should be solves now
[13:37:45] <arturo>	 solved*
[13:38:01] <andrewbogott>	 ok then :)
[13:38:07] <arturo>	 the issue with mixed case was with the puppetmaster
[13:38:15] <arturo>	 (certificate issues)
[13:38:27] <arturo>	 not with openstack itself
[13:41:02] <andrewbogott>	 ok, that's (potentially) less complicated
[15:16:25] <musikanimal>	 hello friends! I am trying to delete a continuous grid job with `qdel`, but it's a very resilient job that's just refusing to die... can an admin force kill it for me? the ID is 243564
[15:16:50] <Oshwah>	 musikanimal: Like... murder it?
[15:17:02] <Oshwah>	 Also, hi! Long time no chat, musikanimal :-)
[15:17:40] <musikanimal>	 maybe humane euthanasia, but if it comes down to a brutal murder, than so be it
[15:24:35] <bd808>	 musikanimal: I force deleted 243564 for you
[15:24:49] <musikanimal>	 thank you!