[11:24:19] so, moritzm pointed out that /srv/private on puppetmaster1001 got out of sync for some reason [11:24:47] indeed on puppetmaster1001 I see e434b8c8f57d9b674ce1596ced1e1588b9eac0f4 and 527c5881b0c8e219af0538533a52dfca0babe1eb - but they're not on 2001 [11:25:54] per https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet they should get sync by a hook, did that maybe fail when commited e434b8c8f57d9b674ce1596ced1e1588b9eac0f4? [11:26:09] I see that .git/hooks/post-commit does some pushing [11:26:41] to be honest I don't remember seeing any error after committing e434b8c [11:27:51] trying the # Now refresh the other masters # part of the hook by hand [11:28:41] ah yes, that of course fails [11:28:51] moritzm: did you get a failure message when committing your last change? [11:29:58] oh, I see that the sha of sukhe's commit differs on puppetmaster1001 compared to puppetmaster1002 [11:30:12] 03fda4cf28e188855200571425e3304f1a5629dc on puppetmaster1001, 9ed73c9490cbc0608a62c8c2f900c12dd8070644 on puppetmaster1002 [11:30:46] I don't have that terminal history anymore unfortunately, I had an error about " empty ident name <(null)> not allowed", but that might have been a red herring as my commit ultimatately had the jmm in the commit message [11:31:47] ah, maybe Sukhbir amended a commit [11:32:07] /home/sukhe/.bash_history:sudo git commit --amend [11:32:10] indeed [11:32:21] there's a notice not to do that in /srv/private/README, but unfortunately not in https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet [11:33:05] oops [11:33:12] there's no README on 1002 BTW [11:33:12] sorry! [11:33:29] how do I fix that? [11:33:41] sukhe: you did not do anything wrong, don't worry about it [11:34:39] I think we apparently should reset --hard the last 3 commits and re-commit them individually [11:34:49] at least that's what the README suggests [11:36:33] also the diff seems weird on puppetmaster1003 [11:37:13] perhaps due to the fact that it has a different git version /o\ [11:37:22] wierd how? puppetmaster1003 is running buster [11:37:26] yes :) [11:37:45] ah, fair enough [11:37:56] "weird" diff as in, different from all other puppetmasters [11:38:27] this could easily be the beginning of a yak shaving session [11:39:43] moritzm, jbond42, sukhe: if you all agree, I'll reset the last 3 commits on puppetmaster1001 [11:40:39] ema: it's an obvious yes from me :) [11:40:49] ok if its just cosmetic then can be ignored. as to rest i think that sounds good. [11:42:36] jbond42: yeah see `cumin 'puppetmaster1002*,puppetmaster1003*' 'cd /srv/private && git log -p -1'` [11:42:38] ema: sounds good! [11:42:39] it's quite interesting actually :) [11:43:21] the diff produced by git installed on 1003 makes more sense [11:43:32] if that worked fine, let's add it to the wikitech page, I vaguely remember we had the same issue a while back when someone used --amend [11:44:16] oh yes i see thats a nice improvment [11:46:13] OK now HEAD on 1001 is 94b9e1f6c1c5a0afa2203e933e56d972276cbeac [11:46:47] while on all other puppetmasters it's 9ed73c9490cbc0608a62c8c2f900c12dd8070644 (one commit after 1001's head) [11:47:41] perhaps I can try to push from 1002 to 1001 at his point, they should get in sync like that? [11:48:16] alternatively, reset 9ed73c9490cbc0608a62c8c2f900c12dd8070644 on all other puppetmasters [11:49:42] reset all others sounds the safest to me but im git novice [11:50:14] jbond42: too late, pushed from 2001 to 1001 and that worked [11:50:20] namely: [11:50:23] :D [11:50:24] root@puppetmaster2001:/srv/private# su -c "export GIT_SSH=/srv/private/.git/ssh_wrapper.sh ; git push ssh://puppetmaster1001.eqiad.wmnet/srv/private master" gitpuppet [11:51:43] alright, now cumin shows the same output on all puppetmasters for `cumin 'A:puppetmaster' 'cd /srv/private && git log -1'` [11:52:11] I'm gonna re-do my commit on 1001 [11:52:31] ack, when you're done, I'll redo mine [11:52:58] moritzm: done and it seems to have worked [11:57:30] ack, yes. I don't see a mail to private yet, but the commit is on puppetmaster2001 [11:57:40] and the mail arrived [12:01:56] very well then [12:03:13] I'll update wikitech to document how to behave in case someone is so reckless to run --amend [12:04:05] * sukhe pretends to look away [12:07:05] sukhe: just to make sure we're 100% clear: there is absolutely no way you could have know this, *and* no damage was done :) [12:07:56] yeah, totally [12:09:09] our private repo is a bit like juggling with hand grenades and ema is adding some docs which tell people not to pull the trigger pin while juggling [12:09:14] ema: with great power comes great responsibility (such as reading the README file) :D [12:09:42] but yeah, I should probably update the onboarding docs as well so that someone else doesn't make the same mistake [12:14:29] added a fat warning to https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet - let me know if you think it's ok or if any further clarification is needed [12:19:53] yeah I think it's good [12:20:23] thanks! [12:24:55] haha, I almost did the same thing on my first commit to the private repo [12:58:14] * cdanis idly thinking about how hard it might be to write a script that noticed about-to-expire icinga downtimes [13:05:15] cdanis: we can have an alert for about-to-expire-downtimes :-D [13:05:20] What if we downtime that? :) [13:05:22] ahaha [13:06:08] it would be very easy to write an irc bot that pinged people some time before a downtime expired, but it would be somewhat harder to make it ping people only when it's a downtime regarding stuff that is still critical [13:08:03] cdanis: and also hard to correlate downtimes usernames with people's IRC nicks, as those might not be the same [13:08:11] indeed [13:08:24] cdanis: but it could also be a script that sends an email to ops@ with about to expire downtimes summary once per week or something [13:34:34] <_joe_> cdanis: https://gerrit.wikimedia.org/r/529358 [13:34:45] \o/ [13:35:57] <_joe_> elukey: what is a server that runs a lot of analytics systemd timers? [13:36:20] an-coord1001 [13:39:43] <_joe_> damn I forgot - is in \W [13:42:59] _joe_: can you do (?[ \W - [-] ]) [13:43:02] ? [13:43:26] <_joe_> cdanis: '^[\w-]' I guess :P [13:43:44] <_joe_> '^[\w\-]' to be precise [13:46:50] <_joe_> the ^ in the parens, ofc. jeez [13:47:10] looks like not even puppet6 supports extended bracketed character classes [13:54:41] <_joe_> I think your expectations towards puppet (the company) and puppet (the language) and puppet (the configuration management system) still need some tuning cdanis [13:55:09] ahaha [13:55:12] it looks like it is Ruby that does not support a pretty deep Perl-ism [13:55:22] <_joe_> I think the true zen is reached when you venture the first time in the codebase to see how "file" is implemented [13:55:43] <_joe_> then you realize your dayjob depends on that beauty [13:55:49] I am yet to look at Puppet's implementation [13:55:54] <_joe_> simple, terse, well tested code [13:56:10] and you'll not snipe me into it today [13:56:33] <_joe_> it's a reading better done while sipping a hot beverage [13:56:37] the snipee is awake [13:56:54] <_joe_> possibly chocolate, it would at least offset some of the misery [13:56:56] _joe_: re the file type tell me about it, thought thios would be a simple fix https://github.com/puppetlabs/puppet/pull/6908 [13:57:32] I am trying to make the puppet debugger of visual studio to work [13:57:43] I think it is a waste of time, but what if it isnt [13:58:35] jijiki: i have also been looking at https://www.puppet-debugger.com/ [13:58:57] and musing that it might be nice to be able to log into the CI docker image and run that gem [13:59:13] oh dear now I have to look into that as well [13:59:24] the demo at http://demo.puppet-debugger.com/ is nice, but uses a hopelessly up-to-date Puppet version ;) [13:59:30] <_joe_> cdanis: I think it's more interesting as an idea to work on running the compiler locally [13:59:34] cdanis: it was funnier when we were simply snipping you [14:00:22] _joe_: as someone who has so far cargo-culted their way through Puppet even at the syntax level, I think both are useful 🤭 [15:48:13] _joe_: haha, I just now realized, there's been an optional 'comment' field in the instance schema all along https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/conftool/+/refs/heads/master/conftool/tests/fixtures/dbconfig/schemas/dbconfig/instance.schema#27 [15:50:28] <_joe_> cdanis: and i forgot it was already there [15:50:50] <_joe_> I now remember thinking "we can add comments later" [15:50:53] I was thinking to have a 'note' field at the instance level, not at the instance-section level [15:51:33] <_joe_> I thought It allowed to add a comment per section could be useful [15:51:55] <_joe_> cdanis: did I tell you how fun a schema change is? [15:51:56] yeah, I could imagine both being useful [15:51:58] no [15:52:02] <_joe_> hah! [15:52:18] in my local testing so far with schema changes I've been doing 2>/dev/null [15:52:23] as it is quite noisy otherwise [15:52:39] <_joe_> it will be yeah, but it failed to do anything in the past [15:52:44] <_joe_> did I fix that too? [15:53:06] WARNING:conftool:Setting note to the default value [15:53:10] <_joe_> it would need manual patching of the existing objects in etcd, which was asinine [15:53:12] once for every instance [15:53:13] <_joe_> ok [15:53:24] <_joe_> it makes sense it does though, it's an unusual situation :P [15:53:28] eh, I was thinking to automate it [15:54:11] add a schema-update subcommand, which will just read and re-write each object without making any explicit changes [15:54:17] hide it behind some sort of --expert flag [15:54:59] we'll need it more than once with the changes we have lined up [15:55:00] <_joe_> i was thinking an option of the syncer [15:55:25] the syncer scares me [15:55:32] <_joe_> ahah :P [15:55:57] <_joe_> yeah it's also simpler to do the other way [15:56:30] <_joe_> it's basically a select 'name=.*' read-and-write-back [15:56:40] yeah [15:57:29] <_joe_> for obj in entity.query(name='.*'): obj.write() [15:59:01] in quasi-related griping, I wish that the yaml syntax for "a key with an empty string value" wasn't so ugly, but also I have no constructive ideas on something better [15:59:09] I assume that's why you've gone with 'PLACEHOLDER' [15:59:20] <_joe_> yes [16:00:06] <_joe_> also I tried to stay away from types conversions as much as possible with php/python interacting [16:00:43] it's all JSON what could possibly go wrong [16:02:29] * fsero cough cough protbouf [16:02:39] something is going badly inside me [16:02:41] :P