[00:07:36] I'm not finding the docs on how to regenerate puppet certs after rebuilding an instance and giving the same name as an old box [00:08:09] awight puppet agent -tv should regenerate it if you rm -rf /var/lib/puppet/ssl [00:08:19] but the cert needs to be removed from the puppet master too [00:08:28] and also signed if you have the auto sign setting off [00:22:35] paladox: sudo puppet agent -tvd mysteriously hangs without doing anything, maybe you've seen that before? [00:22:48] awight -d is a deamon [00:22:53] awight try puppet agent -tv [00:22:58] deployment-puppetmaster02.deployment-prep.eqiad.wmflabs resolves fine from that machine [00:23:20] I think -d is --debug. -tv gives the same behavior, unfortunately. [00:26:19] awight it hangs? [00:27:01] yah only one line of debug output, [00:27:06] Debug: Runtime environment: puppet_version=4.8.2, ruby_version=2.3.3, run_mode=user, default_encoding=UTF-8 [00:27:10] hmm [00:27:56] awight ah ok [00:28:01] awight do the following [00:28:03] puppet agent -tv [00:28:05] without -d [00:28:53] remember to stop the preveous puppet run [00:29:19] Nothing magical happening yet... [00:29:30] strange and bad [00:31:06] hmm [00:34:25] paladox: hahaha so I've been running "puppet apply -tv" furiously. [00:34:32] lol [00:34:43] Glad there was an explanation. [00:34:45] puppet apply -tv is the wrong one :) [00:34:49] puppet agent -tv [11:15:33] (03PS1) 10Volans: Add dummy ECDSA private key for debmonitor.discovery.wmnet [labs/private] - 10https://gerrit.wikimedia.org/r/432361 (https://phabricator.wikimedia.org/T191299) [11:16:13] (03CR) 10Volans: [V: 032 C: 032] Add dummy ECDSA private key for debmonitor.discovery.wmnet [labs/private] - 10https://gerrit.wikimedia.org/r/432361 (https://phabricator.wikimedia.org/T191299) (owner: 10Volans) [11:22:23] (03PS1) 10Volans: Debmonitor: add dummy Django secret key [labs/private] - 10https://gerrit.wikimedia.org/r/432362 (https://phabricator.wikimedia.org/T191299) [11:22:52] (03CR) 10Volans: [V: 032 C: 032] Debmonitor: add dummy Django secret key [labs/private] - 10https://gerrit.wikimedia.org/r/432362 (https://phabricator.wikimedia.org/T191299) (owner: 10Volans) [11:33:47] (03PS1) 10Volans: Debmonitor: add dummy keyholder deployment keys [labs/private] - 10https://gerrit.wikimedia.org/r/432363 (https://phabricator.wikimedia.org/T191299) [11:34:20] (03CR) 10Volans: [V: 032 C: 032] Debmonitor: add dummy keyholder deployment keys [labs/private] - 10https://gerrit.wikimedia.org/r/432363 (https://phabricator.wikimedia.org/T191299) (owner: 10Volans) [13:33:32] !log rcm Xenon: Phab update [13:33:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [13:33:37] !log rcm Xenon: Package updates [13:33:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [13:34:37] !log rcm Tin: Package updates [13:34:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [13:36:10] !log codereview Argon: Package updates [13:36:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codereview/SAL [13:36:19] !log codereview Radon: Package updates [13:36:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codereview/SAL [18:55:54] !log tools depooling, rebooting, repooling tools-exec-1401 to test a kernel update [18:55:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:15:56] !log ores staging ores-wmflabs-deploy:d30df05 [19:15:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [19:52:20] !log ores deploying ores-wmflabs-deploy:04103d2 [19:52:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [21:01:09] !log ores staging ores-wmflabs-deploy:346e198 [21:01:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [21:02:55] (03PS1) 1020after4: add phabricator_files dummy key to swift::params::account_keys: [labs/private] - 10https://gerrit.wikimedia.org/r/432531 [21:03:32] !log ores deploying ores-wmflabs-deploy:346e198 [21:03:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [21:06:19] (03CR) 1020after4: [C: 032] "self-merging because I need to test this on beta." [labs/private] - 10https://gerrit.wikimedia.org/r/432531 (owner: 1020after4) [21:09:22] (03CR) 1020after4: [V: 032 C: 032] add phabricator_files dummy key to swift::params::account_keys: [labs/private] - 10https://gerrit.wikimedia.org/r/432531 (owner: 1020after4) [21:18:50] !log wikilabels staged wikilabels-wmflabs-deploy:2759b7d [21:18:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [21:31:43] !log wikilabels deployed wikilabels-wmflabs-deploy:2759b7d [21:31:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL