[09:39:31] (03CR) 10MarcoAurelio: "recheck" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/423010 (owner: 10MarcoAurelio) [09:39:36] (03CR) 10jerkins-bot: [V: 04-1] Reinstate "build: Updating mediawiki/mediawiki-codesniffer to 17.0.0" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/423010 (owner: 10MarcoAurelio) [09:56:31] (03CR) 10Hashar: "recheck" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/423010 (owner: 10MarcoAurelio) [09:59:10] (03CR) 10MarcoAurelio: [C: 032] Reinstate "build: Updating mediawiki/mediawiki-codesniffer to 17.0.0" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/423010 (owner: 10MarcoAurelio) [09:59:51] (03Merged) 10jenkins-bot: Reinstate "build: Updating mediawiki/mediawiki-codesniffer to 17.0.0" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/423010 (owner: 10MarcoAurelio) [20:42:23] (03Draft1) 10MarcoAurelio: hieradata: fix for deployment-tin/mira lack of ::git_owner [labs/private] - 10https://gerrit.wikimedia.org/r/423178 [20:42:26] (03PS2) 10MarcoAurelio: hieradata: fix for deployment-tin/mira lack of ::git_owner [labs/private] - 10https://gerrit.wikimedia.org/r/423178 [20:44:38] (03PS3) 10MarcoAurelio: hieradata: fix for deployment-tin/mira lack of ::git_owner [labs/private] - 10https://gerrit.wikimedia.org/r/423178 [20:50:52] (03PS4) 10MarcoAurelio: hieradata: fix for deployment-tin/mira lack of ::git_owner [labs/private] - 10https://gerrit.wikimedia.org/r/423178 [20:51:49] (03CR) 10Dzahn: "looks good now, just add the Bug: line to that ticket plz" [labs/private] - 10https://gerrit.wikimedia.org/r/423178 (owner: 10MarcoAurelio) [20:52:26] (03PS5) 10MarcoAurelio: hieradata: fix for deployment-tin/mira lack of ::git_owner [labs/private] - 10https://gerrit.wikimedia.org/r/423178 (https://phabricator.wikimedia.org/T191110) [20:53:22] (03CR) 10Dzahn: [V: 032 C: 032] hieradata: fix for deployment-tin/mira lack of ::git_owner [labs/private] - 10https://gerrit.wikimedia.org/r/423178 (https://phabricator.wikimedia.org/T191110) (owner: 10MarcoAurelio) [22:40:09] !log toolsbeta copied over many prefix puppet configuration in horizon from toolforge T190893 [22:40:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [22:40:12] T190893: Setup the webservice-related instances in toolsbeta - https://phabricator.wikimedia.org/T190893 [22:44:02] In openstack browser I can see a hiera value being applied to a deploment-prep host I'm interested in. Now I want to know where that value is set, so I cycled through all the places named in https://wikitech.wikimedia.org/wiki/Puppet_Hiera#In_Labs (wikitech, ops/puppet repo, labs/private repo), but still can't find it. [22:44:18] Might it be that https://wikitech.wikimedia.org/wiki/Puppet_Hiera#In_Labs is outdated and the hiera value is set in Horizon? [22:45:05] * chicocvenancio checks [22:46:57] eddiegp: definitely, horizon is not listed there but it does have 3 places where Hiera settings may be placed [22:47:46] 'it' means Horizon (having 3 places to set hiera)? [22:48:35] Yes [22:48:58] Project puppet, prefix puppet and instance puppet settings [22:49:22] That last one is under instance details [22:49:28] So then my next question would be if there's any way to see the hiera settings set in horizon for a project I'm not a member of. [22:49:31] or the puppet repo [22:50:05] paladox: yes, or Wikitech, but those are listed on the page [22:50:12] yep [22:50:56] eddiegp: I am not sure that it definitely is not possible, but I think it isn't. [22:51:08] paladox: Yeah, I've looked through all of them already (wikitech Hiera:, puppet and labs/private repo). [22:51:15] oh [22:51:36] I would do it by elimination, If it is not on the other backends it should be in horizon [22:52:01] openstack-browser might have some information [22:52:18] The more common approach is using the prefix puppet setting [22:52:42] Well openstack-browser only tells me everything that's applied, not where it is applied. [22:52:43] zhuyifei1999_: it is present there, but he asked where it would be set [22:52:53] oh [22:53:07] * chicocvenancio is, again, slow [22:53:13] Yeah, it has to be in horizon then. [22:54:01] Thanks chicocvenancio! [22:54:18] You're welcome [22:54:44] eddiegp: hmmm... that's an interesting question. I think by the time that openstack-borwser knows a hiera setting the meta data about where that datum came from has been discarded. [22:54:45] Maybe I should add that to my list of reasons why it seems like a good idea to request access to deployment-prep ;) [22:56:25] Projects like deployment-prep which have a project local puppetmaster can also have local patches that inject hiera data. That type of data would not show up in openstack-browser at all. [22:56:54] bd808: We should just start killing some of all these origins of hiera data. I bet there's a task for that. :) [22:56:55] * chicocvenancio will add horizon to the list in Wikitech once his son allows. (if it isn't there by then) [22:57:07] eddiegp: there is ;) [22:57:48] the one that can most easily be eliminated is the wikitech Hiera: namespace [22:58:30] the major advantage of it however is that it has a version history and author comments which are not available in Horizon's system [22:59:03] Wikitech Hiera namespace is basically the tool data half of Striker for Cloud VPS, or something? [22:59:37] wait, that's Nova Resource [22:59:40] no, it is the precursor to the hiera settings tab(s) in Horizon [23:00:04] and we would shun it and eventually get rid of it, except it has version controls which we like [23:00:07] harej: ah, yes. [23:00:13] yup [23:00:51] so a.ndrew and I talk occasionally about how to version the data from Horzon but then never find time to work on a solution [23:01:13] how much work would it take to put in a version history / audit trail in Horizon? [23:02:09] "some" [23:02:40] drawn out battle with upstream, or someone sits down for a couple of weeks to extend horizon? [23:03:12] some version of the latter. the hiera/puppet stuff is all home grown [23:03:36] I don't remember if the data is in a local db, or only stored via an API [23:03:54] if it's in a db with django model management history is not too hard [23:04:27] but I think it may be all in an sqlite db elsewhere [23:04:35] which makes it more work [23:05:50] wikitech is just so special [23:07:43] its less special than it once was :) [23:08:05] So, the wikitech->horizon migration is one thing, but it's just two places to look. I think what's worse is the ton of places in git (the docs currently list 6, though hierarchy settings on the puppet master suggest even more). Would be glad if we could get rid of some of those. [23:08:17] but, yes, like commons and wikidata it has some unique features [23:09:12] eddiegp: that's... less likely to happen. The core SRE team generally likes all that flexibility. For a production host there are even more places in git that the data could come from. [23:11:25] But none on wikitech or horizon ;) [23:13:30] If we asked the core SRE team I'm pretty sure they would vote to only have this data in the operations/puppet.git repo. That turns out to make setting up Cloud VPS projects hard for non-SRE team folks though [23:13:40] It's not that I don't like git, I just miss the point of having a public labs/private repo as well as the main public puppet repo and storing hiera in both. [23:13:51] so we add complexity for convenience [23:14:36] the labs/private.git repo is really a bandaid. [23:15:21] there are things in the operations/puppet.git repo that assume a "private" repo exists and contains certain files [23:16:11] most (all?) of those assumptions come from before the introduction of hiera to our environment [23:16:34] but they persist because the references haven't been changed to be hiera lookups instead [23:17:22] but I hear you that the whole thing is complex and hard to debug [23:18:58] Killing the whole labs/private repo is a different thing than killing the hieradata folder in it though. As far as I understood hiera not much would change if we moved the files from there to the respective folders in the puppet repo (and merged with existing ones there, if necessary). [23:21:03] eddiegp: true. that would be possible. I haven't looked at the hiera data there, but it is likely based on what would be in the unpublished 'private' repo that is used in production [23:21:45] so the difference is basically making it easier for SREs to keep in sync with the core network config or making it less confusing for volunteers to track down data [23:23:12] we really could also get rid of per project/host hiera settings for Cloud VPS things in operations/puppet.git too [23:23:34] true. Making it easy for both is probably near impossible or at least needs "some" time ;) [23:23:35] deployment-prep is probably the main project using that. [23:25:12] We could at least get rid of "labs/hosts/%{::hostname}". It's an empty folder anyway. [23:25:45] eddiegp: quick! post a patch! [23:26:14] Meh, I just found a .gitignore file in there: [23:26:18] # Exclude all yaml files here from git, as they should be created on-demand for any instance. [23:26:20] *.yaml [23:26:49] hmmm... on demand? That doesn't seem likely in practice [23:27:18] unless... let me look at the cloud vps shared puppetmaster [23:28:34] Yeah, but removing it from the hierarchy shouldn't happen unless we know for sure that no puppet master uses it. [23:29:16] it's empty on labpuppetmaster1001. It could be used via local patches on some project local puppetmaster I guess [23:30:07] that .gitignore is from the first commit adding hiera [23:30:18] I think its an option that never got used (for good reason) [23:30:51] I'd ask whether there's an easy way to check that, but I already assume "no". [23:31:08] for files on other puppetmasters? [23:31:21] Yes, whether it's used on any local puppetmasters [23:31:24] there is ... if I can remember how to use cumin [23:32:13] cumin was this "run shell command on hosts based on puppet facts/roles" thing, right? [23:32:47] yeah. its basically a wrapper around clush which is a parallel ssh libarary [23:36:37] * bd808 mashes buttons on a cumin query [23:42:36] eddiegp: cumin says that `ls /var/lib/git/operations/puppet/hieradata/labs/hosts/*.yaml` returns an error on all 750 Cloud VPS instances. [23:42:58] so as far as I can tell that feature is unused [23:44:21] bd808: Can you do the same for the labs/private repo hieradata/labs/ folder please? [23:45:16] There's only /hieradata/labs/tools/common.yaml in there right now, which we can move to /hieradata/tools.yaml (that's in the hierarchy as well). [23:47:28] * bd808 mashes buttons again [23:56:17] eddiegp: cumin seems to say that `ls -1 /var/lib/git/labs/private/hieradata/labs/|grep -v tools` is empty or fails everywhere [23:57:14] Nice! [23:58:21] We could also probably merge labs.yaml and common.yaml in the private repo.