[12:26:33] !log openstack T219626 create VM cloudcontrol-codfw1dev-test-01 for testing codfw1dev cloudcontrol puppet role [12:26:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [12:26:38] T219626: codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch - https://phabricator.wikimedia.org/T219626 [14:51:46] !log maps shutting down maps-wma1 as per T204506 [14:51:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL [14:51:49] T204506: cloudvps: maps project trusty deprecation - https://phabricator.wikimedia.org/T204506 [14:54:30] !log tools moving tools-static-12 to eqiad1-r (for real this time maybe) [14:54:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:53:52] !log tools cleared E state on 6 grid queues [16:53:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:43:16] !log tools Shutdown tools-checker-02 via Horizon (T219243) [19:43:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:43:20] T219243: Migrate tools-checker system to Stretch - https://phabricator.wikimedia.org/T219243 [19:44:03] !log tools Deleted tools-checker-02 via Horizon (T219243) [19:44:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:09:09] Hello everyone, quick question. How can I apply a puppet change on horizon? I added a hiera config parameter but it didn't applied automatically. [21:09:27] hi mateusbs17 [21:09:41] what hiera config did you try to add where, and what happened? [21:10:37] I set use_nodejs10:true on `deployment-maps04`, the configuration changed but logging in the vm via ssh showed that the last puppet run was before my change. [21:11:05] And dpkg -l nodejs still shows v6.11.x [21:11:17] well that's normal [21:11:22] changing hiera won't trigger a puppet run [21:11:40] puppet runs on cron, you could trigger it manually though [21:11:55] though, I just went to check and it shouldn't be 90 minutes ago, that's a bit much [21:12:18] ah [21:12:30] mateusbs17, it does not like your change [21:12:34] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Profile::Maps::Apps]: parameter 'use_nodejs10' expects a Boolean value, got String on node deployment-maps04.deployment-prep.eqiad.wmflabs [21:13:14] interesting [21:13:30] horizon/encapi has been known to muck with types of parameters/hiera [21:14:30] Hmm, I see. Not sure why it considers as string though, I will take a look at it. Where can I find the puppet compiler feedback? [21:15:03] mateusbs17, oh you didn't just do it in hiera, it's in the puppet parameters thing [21:15:04] yeah I think this is a known bug [21:15:54] Yes, I edited on the parameters thing after a weird capital letter change on the value [21:16:38] looks like this may be part of https://phabricator.wikimedia.org/T210513 though I thought there was another one [21:20:01] It seems that there is no workaround other than be less strict on puppet types [21:22:17] let's see if I can work around it for this case [21:22:52] mateusbs17, ew okay so I tried to set this in hiera and it's got clever [21:22:52] it overrides the parameter instead [21:22:53] might explain the uppercase you mentioned [21:22:54] re puppet compiler: I don't think we have a working puppet compiler for instances right now [21:22:55] andrewbogott was working on something [21:22:56] mateusbs17, well with the True value shown it behaves with types, new error: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Apt::Repository[wikimedia-node10] is already declared in file /etc/puppet/modules/service/manifests/node.pp:228; cannot redeclare at [21:22:57] /etc/puppet/modules/service/manifests/node.pp:228 at /etc/puppet/modules/service/manifests/node.pp:228:13 at /etc/puppet/modules/tilerator/manifests/ui.pp:103 on node deployment-maps04.deployment-prep.eqiad.wmflabs [21:23:31] Krenair: hopefully in a few days we'll have something stable. In the meantime you can /probably/ compile things that use the main cloud puppetmaster (labpuppetmaster1001) [21:25:01] so service::node and tilerator::ui both try to set up apt::repository { 'wikimedia-node10': ? [21:25:03] Krenair: thanks, that is an evolution. [21:26:28] Honestly, I am not sure what is happening. I will ask gehel for help on that one. Thanks [21:29:34] * gehel is in bed already, but will have a look tomorrow [21:33:04] no [21:33:05] is it multiple service::node classes both trying to create the same apt::repository? [21:33:21] mateusbs17, yeah it's upset about multiple service::node resources existing and therefore them both trying to set up this apt::repository resource [21:33:24] in this case, tileratorui, tilerator, and kartotherian [21:39:31] My poor ZNC seems to be having issues [21:39:45] Okay so I've been sending messages but not receiving them from others, and stuff just reconnected. Based on wmbot's channel logs my messages went through though [21:40:19] mateusbs17, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/500615 [21:41:08] * mateusbs17 Krenair looking [21:46:13] It's weird, I was trying to replicate https://github.com/wikimedia/puppet/blob/production/modules/aqs/manifests/init.pp#L94 [21:46:15] per https://phabricator.wikimedia.org/T213077#5014778 [21:48:57] But yeah, kartotherian, tilerator and tileratorui altogether are triggering multiple apt::repository [21:50:30] Thanks Krenair! [21:50:41] np [21:50:53] I cherry-picked this on deployment-puppetmaster03 so -maps04 will work [22:51:10] bd808: we can revert if you think thats best, i've seen numerous funny things happen to cirrus because of this so i disabled it [22:51:21] mwv certainly isn't just cirrus [22:52:26] ebernhardson: right now the things that it breaks are super low priority. But changes like this are nice to discuss in a task first I guess so we can actively decide that [22:52:59] so, I'd say leaving the change is ok, but we may have to revisit that [22:53:03] ok [22:53:31] I don't want to scare you off though! Thank you for using MW-V :) [22:53:42] the funny things in cirrus might only be silly dev things...but if you delete an index and the job queue is still doing something it creates crazy indice s:) [22:55:04] ebernhardson: do you not install index templates that would make them get recreated as expected? [22:55:20] bd808: no, cirrus creates indices explicitly without templates [22:55:32] ah, nod [22:55:49] templates would require 40+, as the configuration depends on language