[16:36:39] <halfak>	 !log deployment-prep deploying ores 7d45b80
[16:36:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[19:26:17] <paladox>	 hi, i get this error "Error: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:///var/lib/puppet/client/ssl/certs/ca.pem" after setting up a new instance.
[19:26:35] <paladox>	 it's phabricator-09.phabricator.eqiad.wmflabs
[19:28:57] <bd808>	 paladox: thanks for the report. That sounds like it could be related to the Puppet work that andrewbogott did this week.
[19:29:10] <bd808>	 paladox: have you tried a second puppet run to see if it clears itself?
[19:29:22] <paladox>	 bd808 yup
[19:29:33] <bd808>	 and is this instance using a project local puppetmaster or the shared puppetmaster?
[19:29:53] <paladox>	 it's using the puppet master from git.
[19:30:06] <paladox>	 i quickly trying to test buster for gerrit (then i'm deleting it).
[19:30:16] <bd808>	 paladox: "yup" it cleared itself or "yup" related to puppetmaster changes?
[19:30:33] <bd808>	 project local puppetmaster. good data point
[19:30:39] <paladox>	 the yup was to the second run.
[19:31:00] <paladox>	 hmm, i wonder if this may be due to the outdated master? (i think only a few days)
[19:31:30] <bd808>	 cleared on second run, so maybe a missing dependency in the puppet manifests
[19:31:57] <paladox>	 bd808 sorry i was not clear, it didn't work on the second round :(
[19:32:22] <bd808>	 ah.
[19:32:58] <bd808>	 so if the project local puppemaster is even just 24h behind things could be in all kinds of weird states
[19:33:03] <paladox>	 ah
[19:33:10] <paladox>	 i'll resolve the conflicts
[19:33:14] <bd808>	 because there was a lot of puppetmaster/cert work this week
[19:33:30] <paladox>	 bd808 thanks!
[19:34:05] <bd808>	 all part of moving the shared puppetmasters into a Cloud VPS project instead of the old bare metal and trying to make switching from the shared puppetmasters to project local work better
[19:34:17] <bd808>	 that last part is still a work in progress
[19:34:59] <paladox>	 ah, ok.
[19:34:59] <mutante>	 paladox: let's just use the normal puppetmaster instead of the local one in this case
[19:35:04] <paladox>	 ok
[19:35:05] <paladox>	 sure
[19:35:25] <paladox>	 bd808 what's the new name for the new puppet masters?
[19:35:31] <mutante>	 we dont need the gerrit patches at this point
[19:40:15] <paladox>	 ah puppetmaster.cloudinfra.wmflabs.org
[19:41:06] <paladox>	 bd808 andrewbogott would someone be able to remove phabricator-09 cert on puppetmaster.cloudinfra.wmflabs.org please?
[19:41:15] <paladox>	 On the master:
[19:41:15] <paladox>	   puppet cert clean phabricator-09.phabricator.eqiad.wmflabs
[19:41:49] <mutante>	 paladox: maybe easier to destroy and create again?
[19:41:56] <paladox>	 ah
[19:41:57] <paladox>	 yeh
[19:41:59] <paladox>	 guess so
[19:42:10] <mutante>	 just do -10
[19:42:22] <mutante>	 we'll delete it soon anyways.. "cattle not pets" or something
[19:48:22] <andrewbogott>	 paladox: sorry, was afk
[19:48:36] <andrewbogott>	 If you can reproduce that let me know, it's similar to an issue I've been following
[19:48:41] <paladox>	 andrewbogott ok :) (i've recreated the instance and pointed it at the cloud instance)
[19:48:49] <andrewbogott>	 did it happen after you moved the instance to a different puppetmaster?
[19:49:06] <bd808>	 paladox: still need a cert removal? or did delete do that automatically?
[19:49:36] <andrewbogott>	 In theory cert cleanup on instance deletion is working
[19:49:37] <paladox>	 bd808 i've deleted the instance, so nope :)
[19:49:39] <andrewbogott>	 …in theory :)
[19:50:37] <andrewbogott>	 paladox: did you by chance create that failed VM 4 or 5 hours ago?
[19:51:00] <paladox>	 within the last 40 minutes
[19:51:03] <andrewbogott>	 hm
[19:51:09] <andrewbogott>	 but when you tried again it worked?
[19:51:43] <paladox>	 nope, it still failed if you mean by re running puppet :(. But creating a new instance and using the new cloud puppet masters worked.
[19:51:54] <paladox>	 andrewbogott by any chance does the new puppet masters use puppet5?
[19:52:23] <andrewbogott>	 so, hang on, which puppetmaster was it using before?
[19:53:02] <paladox>	 andrewbogott puppet-paladox
[19:53:18] <andrewbogott>	 ah, ok
[19:53:26] <paladox>	 i'm now trying with the cloud puppet masters since mutante says we doin't need any patches i did locally.
[19:53:28] <andrewbogott>	 so  in that case you are almost certainly running into T232428
[19:53:29] <stashbot>	 T232428: Resolve local commits on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs and cloud-puppetmaster-02.cloudinfra.eqiad.wmflabs - https://phabricator.wikimedia.org/T232428
[19:53:49] <andrewbogott>	 So at least we know about it!  Sorry for the time-waster :(
[19:53:52] <paladox>	 oh
[19:53:56] <paladox>	 andrewbogott it's ok :)
[19:54:03] <andrewbogott>	 (The patches listed there are reverted on the official masters but likely present on any project-local masters)
[19:54:26] <paladox>	 andrewbogott do the cloud puppet masters run puppet 5?
[19:55:26] <andrewbogott>	 I doubt it — checking
[19:55:32] <paladox>	 ok
[19:56:01] <andrewbogott>	 https://www.irccloud.com/pastebin/7VnlLvwG/
[19:56:12] <andrewbogott>	 So assuming package version ~= puppet version
[19:56:14] <paladox>	 ah so puppet4, hmm
[19:56:46] <paladox>	 hmm does puppet5 work with a puppet4 master?
[19:57:08] <paladox>	 actually ignore me, stupid question as puppet just ran successfully :)
[20:46:06] <bd808>	 !log tools.bridgebot Restarting matterbridge. Bot AWOL from channels
[20:46:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[20:48:58] <phamhi>	 !log tools Deleted tools-puppetdb-01.tools as it is no longer in used
[20:49:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:18:50] <halfak>	 Hey folks.  I've got a new VM I can't log into again.  ores-web-04.ores.eqiad.wmflabs
[21:19:01] <halfak>	 I'm getting a public key denied. 
[21:19:51] <bd808>	 halfak: let me see if superpowers can get there...
[21:19:56] <halfak>	 thanks bd808 
[21:20:45] <bd808>	 !log ores Forcing puppet run on ores-web-04.ores.eqiad.wmflabs to see if I can fix ssh access for halfak 
[21:20:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[21:21:50] <bd808>	 halfak: try again please? Pupept run was clean, but I'm tailing auth.log now
[21:22:07] <halfak>	 It works.  
[21:22:08] <halfak>	 Thank you. 
[21:22:13] <halfak>	 Should I have just waited longer? 
[21:22:20] <halfak>	 I waited a good 5 minutes after horizon gave the OK
[21:22:27] <bd808>	 I think I only helped by giving you something to do until Puppet finished running :)
[21:22:42] <halfak>	 haha.  OK.  I'll wait longer next time.  
[21:22:56] <halfak>	 Sorry to pull you in prematurely.  I appreciate your help anyway. 
[21:22:57] <bd808>	 It said last puppet run 0 minutes ago when I got in
[21:23:11] <bd808>	 no worries! better to ask than to be stuck for hours
[21:25:10] <halfak>	 :)