[00:12:25] TimStarling: I want to ask you a few things about swift/ms7 when you have the time [00:12:38] https://wikitech.wikimedia.org/view/Swift/Open_Issues_Aug_-_Sept_2012/Cruft_on_ms7 specifically [00:18:37] TimStarling: No, it's because Aaron added a $wgMathDirectory usage to filebackend.php [00:45:35] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [00:48:51] !log starting nagios on spence [00:49:01] Logged the message, Master [00:49:52] TimStarling: the original patchset was the script importing a sql dump [00:50:25] TimStarling: which would allow toolserver to run arbitrary commands on one of our production systems [00:50:26] ACKNOWLEDGEMENT - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours daniel_zahn its a Dell C2100 [00:52:09] Ryan_Lane: that's still the case as far as I can see [00:52:37] it's using tab delimited data [00:52:52] or it should be anyway [00:54:07] how's DROP TABLE; ALTER TABLE ... RENAME TO isn't racy? [00:54:20] lol @ Daniel's acknowledgement "it's a Dell C2100" [00:54:29] what happens on the hits in the middle of the two statements? [00:54:42] apparently the app will handle that situation magically [00:56:12] what happens if toolserver.org gets hacked and attempts to fill our database with bogus data? [00:56:19] we're fucked [00:56:26] come on, this is terrible in so many ways [00:56:29] yep [00:56:41] we've complained a number of times about this [00:56:46] apparently it's only running for a number of months [00:56:50] that code is pretty dense [00:56:51] then it'll be fixed for next time [01:00:31] RoanKattouw: :) [01:00:43] RoanKattouw: does umask for wikidev users still matter after switch from svn to git? [01:01:07] re: "Simply running svn up with the wrong umask can put our SVN checkout in a nasty broken state" [01:01:08] I think so [01:01:17] I believe git pull can have the same effet [01:01:43] alright [01:02:19] I've seen screwed-up ownership in the .git directory at least once, but I don't remember if it was due to a bad umask or not [01:07:53] so the data imported by that script is actually used for something? [01:12:17] why is the toolserver used at all? [01:13:42] my question exactly [01:14:24] because it's doing the computation on toolserver and it's somehow too much work to move that over too [01:14:43] I think (and have said) this is a bad idea [01:40:38] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 242 seconds [01:40:56] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 262 seconds [01:46:47] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 613s [01:59:23] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 1s [01:59:32] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 10 seconds [02:00:17] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 8 seconds [02:00:35] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [02:10:20] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [02:10:20] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [02:12:17] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: Puppet has not run in the last 10 hours [02:57:26] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: Puppet has not run in the last 10 hours [02:58:20] PROBLEM - Puppet freshness on ms-be11 is CRITICAL: Puppet has not run in the last 10 hours [03:03:35] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Thu Aug 30 03:03:25 UTC 2012 [03:05:19] "cd /var/wlm/data/ && rm.txt" [03:05:37] I think there's a typo in that comment. [03:38:22] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: Puppet has not run in the last 10 hours [03:51:24] PROBLEM - Puppet freshness on cp1023 is CRITICAL: Puppet has not run in the last 10 hours [03:56:22] PROBLEM - Puppet freshness on cp1022 is CRITICAL: Puppet has not run in the last 10 hours [04:05:21] PROBLEM - Puppet freshness on ms-fe4 is CRITICAL: Puppet has not run in the last 10 hours [04:06:55] paravoid: i think i addressed racy in my comment? maybe good enough, maybe perfectly (we could ask domas ;) ) [04:08:18] (that's gerrit 17964) [04:22:27] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [04:22:27] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [04:22:27] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [04:22:27] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [04:22:27] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [04:22:28] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [04:22:28] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [04:22:29] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [04:22:29] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [04:22:30] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [04:22:30] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [04:22:31] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [04:22:31] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [05:29:05] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [07:08:10] PROBLEM - Puppet freshness on ms-be8 is CRITICAL: Puppet has not run in the last 10 hours [07:31:20] PROBLEM - Puppet freshness on ms-be1 is CRITICAL: Puppet has not run in the last 10 hours [07:42:17] PROBLEM - Puppet freshness on ms-be2 is CRITICAL: Puppet has not run in the last 10 hours [07:52:29] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [07:53:59] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [07:58:29] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [08:02:46] hello [08:22:11] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [08:22:11] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [08:22:11] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [08:22:11] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [08:23:23] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [08:28:11] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [08:34:02] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [08:38:41] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [08:40:11] PROBLEM - Puppet freshness on palladium is CRITICAL: Puppet has not run in the last 10 hours [08:56:14] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [09:06:08] PROBLEM - Puppet freshness on ms-be9 is CRITICAL: Puppet has not run in the last 10 hours [09:46:07] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 185 seconds [09:46:07] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 186 seconds [09:46:25] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 199 seconds [09:46:25] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 199 seconds [09:55:07] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [09:55:07] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [10:46:34] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [10:46:52] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [10:47:46] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 0 seconds [10:48:58] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 0 seconds [10:53:54] !log gerrit-wm irc bot is apparently no more sending anything to #mediawiki :( IRC user has been inactive since midnight UTC. Opened {{bug|39797}} [10:54:07] Logged the message, Master [10:54:12] !root [10:54:22] any ops around to restart irc echo on manganese ( https://bugzilla.wikimedia.org/show_bug.cgi?id=39797 ) [10:54:32] gerrit-wm is no more sending notification to IRC channel [10:54:35] started occurring at midnight [10:54:43] so might be a cron job that killed it / caused issue [10:54:55] apergos: paravoid: ^^^ [12:11:34] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [12:11:34] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [12:13:31] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: Puppet has not run in the last 10 hours [12:21:26] apergos [12:24:20] mark: [12:24:50] that rsync cron spam [12:24:52] why is that happening? [12:25:24] which? [12:25:43] nice [12:25:46] so you don't even see it [12:25:52] dataset1001 [12:25:55] I looked at cronspam today [12:26:00] but I ddn't see a huge amount [12:26:08] i don't want any from that box [12:26:10] I can't do shit about it [12:26:15] and it's cluttering up my email [12:26:18] whereas you're not even seeing it [12:26:30] it makes me want to turn off the damn cron job every day [12:26:47] I just said I looked at cronspam today [12:26:51] I look at it every day [12:26:58] hi; can you please to restart ircecho on manganese? gerrit-wm is no more sending notification ( https://bugzilla.wikimedia.org/show_bug.cgi?id=39797 ) [12:27:00] well apparently you're not fixing it then? [12:27:11] the dataset messages are very few compared to everything else [12:27:32] yeah there's virt stuff which I'm complaining to ryan about [12:27:38] but I don't see the point of getting them [12:27:48] if it's something you can't fix, why not send it to devnull? [12:27:50] or yourself only [12:28:02] I can send em just to me, that's fine [12:28:13] it's not like anyone else is gonna do anything with it [12:28:19] I do want to get notifications, not for these files, maybe I can filer a little better [12:28:26] yeah that's true [12:28:34] so are these there because the files change during the rsync? [12:28:40] well they complete [12:29:02] or they are temp files that get tossed [12:29:10] I got rid of some of those but not all I guess [12:29:15] perhaps those shouldn't get rsynced then [12:29:20] or perhaps LVM snapshots would help there [12:30:20] we really don't need em rsynced, that's the best [12:30:54] but for now I'll probably mailifonly to the alias for dumps (which I think has only me on it :-P) [12:31:02] thanks [12:31:05] yw [12:52:56] RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000 [12:53:41] RECOVERY - check_job_queue on neon is OK: JOBQUEUE OK - all job queues below 10,000 [12:57:53] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: Puppet has not run in the last 10 hours [12:58:56] PROBLEM - Puppet freshness on ms-be11 is CRITICAL: Puppet has not run in the last 10 hours [13:16:17] woot sound the klaxon. first successful payment test through the new frack payments rig! [13:17:43] congrats [13:38:59] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: Puppet has not run in the last 10 hours [13:52:02] PROBLEM - Puppet freshness on cp1023 is CRITICAL: Puppet has not run in the last 10 hours [13:57:07] PROBLEM - Puppet freshness on cp1022 is CRITICAL: Puppet has not run in the last 10 hours [14:06:16] PROBLEM - Puppet freshness on ms-fe4 is CRITICAL: Puppet has not run in the last 10 hours [14:06:27] apergos: here? [14:06:43] yes [14:06:53] are you staying or about to leave? [14:07:05] I'll be here for a while [14:07:08] okay [14:07:10] what's up? [14:07:19] we have a disk replaced in ms-be7 [14:07:30] that needs to be formatted and reenabled [14:07:41] ah ha [14:08:08] the box probably needs a reboot too; the disk was sdg but hte system now sees it as "sdo" [14:08:20] ugh [14:08:22] we can always mv the node, but I prefer clean solutions [14:09:08] !log authdns-update for helium/potassium (poolcounter servers) [14:09:21] Logged the message, RobH [14:09:22] I guess chris swapped it in last night? [14:09:28] yes [14:10:11] I think formatting etc. is being handled by puppet (scary) [14:10:22] and the rest is the ring builder, which I've done once before [14:10:23] so, [14:10:25] paravoid: apergos: mark: we have lost gerrit notification from ircecho on manganese ? Can one of you look at it please ? https://bugzilla.wikimedia.org/show_bug.cgi?id=39797 [14:10:28] would you like to do this one? :-) [14:10:32] yeah I was looking [14:10:34] (maybe I should get a simple shell access on that server) [14:10:44] but I don't see what's wrong yet [14:11:02] I restarted it and have been loking a bit at the code and at the logs [14:11:18] next I might shoot it and run it from the command line and see if I get anything useful (since it's broken anyways) [14:11:44] paravoid: sure, but it will be a little bit later. I wanna try to make headway on the irc bot (or reach the giving up point) [14:11:59] okay [14:12:02] I have a meeting in 20' [14:12:05] ok [14:12:15] and another one an hour after that [14:12:31] well do your meetings, it's fine [14:13:44] anything tricky about the ringbuilder piece? [14:20:33] there are instructions [14:20:52] but I don't think they're complete [14:21:02] I don't remember the details; catch up with me when you're about to do it [14:21:09] ok [14:21:18] or when you hit a wall [14:21:57] great [14:23:13] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [14:23:13] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [14:23:13] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [14:23:13] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [14:23:13] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [14:23:14] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [14:23:14] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [14:23:15] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [14:23:15] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [14:23:16] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [14:23:16] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [14:23:17] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [14:23:17] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [14:25:02] apergos: looks like gerrit-wm notify again thx! [14:25:09] it is? [14:25:12] but uh [14:25:17] got a notification [14:25:26] have you done anything? [14:25:28] where? [14:25:34] in #mediawiki [14:25:35] oh [14:25:37] huh how weird [14:25:41] ok well um [14:25:48] got restarted 10 minutes ago apparently [14:25:54] I restarted it much before that [14:25:57] and it didn't help [14:26:13] guess some magic stuff appeared that fixd it :) [14:26:15] I closed the bug! [14:26:16] it's running directly from the command line and not in screen or anything so I'm going to shoot it and let it run again [14:26:18] thx! [14:26:22] ok [14:26:48] I found a bunch of [14:27:05] ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to gerrit2@formey.wikimedia.org:/var/lib/gerr [14:27:05] it2/review_site/git/operations/puppet.git [14:27:05] and [14:27:19] TransportException: gerrit2@formey.wikimedia.org:/var/lib/gerrit2/review_site/git/operations/puppet.git: session is down [14:27:20] in the logs [14:27:32] but I did not get anywhere close to figuring out what the cause was/is [14:28:59] I don't even know that this is the issue [14:45:41] hey makr [14:45:42] mark [14:45:45] can you look at this today? [14:45:51] it has been waiting since monday [14:45:57] https://gerrit.wikimedia.org/r/#/c/21749/ [15:01:15] hashar: I think it was premature to close the bug, sorry [15:02:47] dohh [15:03:10] i think formey is a backup / slave [15:03:10] I see reviews and patchsets being added to the logs [15:03:17] and nothing going to the channel [15:03:24] ^demon: apparently gerrit replication from manganese to formey has some issues [15:03:45] apergos: if it is written in the log, at least the gerrit hooks are working [15:03:50] so that let us with the irc bot [15:04:08] could be either ircecho that no more track the file [15:04:10] I do see that gerrit on manganese was restarted yesterday [15:04:13] hey opsen, is there someone who has some spare cycles to take care of https://rt.wikimedia.org/Ticket/Display.html?id=2970 (Redirect all .mobile requests to .m)? [15:04:26] well ircecho has them open, I checked that [15:04:32] <^demon> hashar: What sort of issues? [15:04:49] ^demon: ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to gerrit2@formey.wikimedia.org:/var/lib/gerrit2/review_site/git/operations/puppet.git [15:04:52] session is down [15:04:56] or at least it opens them and then does some inotify thing on them [15:04:58] <^demon> apergos: Could you please merge https://gerrit.wikimedia.org/r/#/c/21965/? I can't get at the gerrit logs until that goes back in. [15:05:03] spotted by apergos in the logs [15:05:14] bah, it doesn't let me look at it [15:05:17] oh [15:05:20] my client took the ? [15:07:56] what host do you need that on? [15:08:05] <^demon> manganese & formey [15:10:20] on formey: [15:10:22] err: Could not prefetch ssh_authorized_key provider 'parsed': Could not parse line "ssh-rsa gerrit2" at /var/lib/gerrit2/.ssh/authorized_keys:2 [15:10:32] dir perms are fixed [15:10:45] <^demon> Eh that's not surprising. Permissions is all I needed. [15:10:46] <^demon> Thanks. [15:11:34] ottomata: it's not strange that reviews take a long time if you do these huge commits eh [15:11:45] it's not something you can review in a minute [15:12:05] why do we need xinetd? [15:12:19] I didn't write this, this was taken from puppetlabs [15:12:24] paravoid suggested I commit the files directly [15:12:39] <^demon> hashar: This is why I want 2.5. I could just /stop replication/ by unloading the plugin :) [15:12:44] xinetd is a way to run an rsync daemon [15:12:45] https://github.com/puppetlabs/puppetlabs-rsync [15:12:48] ^demon: :)))))))) [15:12:50] the rsync module uses it [15:12:52] (but doesn't have to) [15:12:56] it can use init.d/ scripts too [15:13:05] apergos: so I guess the gerrit-wm not responding is due to ircecho :/ [15:13:07] <^demon> Oh herp derp. I know what's wrong. [15:13:17] so at least split those up in separate commits [15:13:20] <^demon> I'll have a fix for replication real soon now. [15:13:22] i don't think we need xinetd [15:13:26] what, rsync and xinetd? [15:13:28] yes [15:13:36] well, the rsync module uses the xinetd module [15:13:43] it doesn't need it [15:13:47] you can get around it, but you won't be able to just include rsync::server [15:13:51] you'd ahve to [15:14:01] class { "rsync::server": use_xinetd => false } [15:14:10] hashar: I'm sure but what I don't know is that happened to make it stop working [15:14:14] code is the same old code [15:14:26] i tested the xinetd stuff on my VM, and it works great [15:14:29] oohhh maybe demon's fix will help [15:14:35] doesn't mean we necessarily want it [15:14:48] is there a reason you don't want it? [15:14:56] yes, yet more cruft in our repo [15:15:05] xinetd is cruft? [15:15:08] i think so [15:15:27] really? its a pretty standard thing, no? [15:15:31] we don't use it [15:15:42] is there a reason why not? [15:15:44] it's standard in redhat land I think [15:15:54] more like, is there a reason why we would use it? [15:16:22] <^demon> apergos: https://gerrit.wikimedia.org/r/#/c/22031/ + a puppet re-run on formey will fix replication and the puppet error [15:16:43] * ottomata googling for internet opinions on xinetd on debian/ubuntu [15:16:49] so how did it break? [15:17:02] <^demon> SSH public key wasn't installed on formey [15:17:04] <^demon> Just the private. [15:17:10] can you split it up in separate commits? [15:17:13] rsync might go in soon [15:17:22] we can add xinetd later if it's really needed [15:17:27] ottomata: I'm with mark on this, there is nothing "wrong with" xinetd per se but why use it if we don't need to, it's just one more thing to keep track of [15:18:06] i dislike big commits in general [15:18:19] incremental is good [15:18:43] yeah, i understand that, and what I would've prefered to do is not commit this at all in puppet [15:18:47] since I didn't write it [15:18:51] there are lots of small commits [15:18:59] you can go read history from upstream if you want to review all the small commits :) [15:19:22] as is, if you try to use rsync module without xinetd, you'll get puppet errors unless you try to get around it [15:19:33] you just said rsync doesn't need xinetd per se [15:19:41] right, but you have to manually do it [15:19:42] and [15:19:46] then do that? [15:19:49] the code is there in rsync module to use xinetd [15:19:50] or change the default parameter? [15:20:02] so it seems wrong to have the code existing in a non working state [15:20:05] ja was about to suggest that [15:20:10] i'm ok with that, if we default to init.d [15:20:19] init.d? [15:20:23] what's wrong with running rsync in daemon mode? [15:20:32] oh sorry [15:20:35] I think the more important question is what to do with external modules [15:20:36] i misread that as inetd [15:20:40] aye ja [15:20:46] yeah that is a bigger q [15:21:00] paravoid: just don't use em? :) [15:21:06] haha [15:21:11] import them as they come into our puppet repo with a danger of having it blow up in size [15:21:14] and cruft [15:21:19] you want us to rewrite everything ourselves when someone else has done hte work? [15:21:25] yes if it's not exactly what we need [15:21:59] i don't particularly like many 3rd party modules [15:22:05] have you looked at this one? [15:22:09] not yet [15:22:15] I looked at external rsync modules when I was going to write one (still plan to) [15:22:21] they didn't really cover our cases well [15:22:27] which is why I was going to write one :-/ [15:22:34] i was thikning about writing one too [15:22:34] I'm afraid we'll end up with a NIH syndrome [15:22:35] but if you start like "we need X extra modules because module Y requires it even if we don't need it" then I already start to dislike it ;) [15:22:39] our template duplication right now is really bad [15:22:46] but otoh I've seen a lot of crappy modules as well [15:23:03] yeah, for example, that's why I wrote the generic::mysql define (it should bea module) [15:23:10] so ^demon do we need to restart anything anywhere (on manganese)? [15:23:11] because there was nothign good that I could find [15:23:22] but this rsync one is really good, it does everything I want it too [15:23:35] specificically the ability to puppetize rsync daemon modules separately [15:23:57] <^demon> apergos: Shouldn't need to, but I'll check. [15:23:57] so you don't have to have an rsync.conf template for every machine type [15:24:19] so mark, ok, I will change this commit, I will remove xinetd and use init.d by default [15:24:42] <^demon> apergos: I'm not getting replication errors anymore. [15:24:48] ok well that's good [15:24:54] now I wonder about gerrit-wm [15:24:59] guess I'll restart it again :-/ [15:28:16] New review: Demon; "Test" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/21961 [15:28:22] <^demon> Seems to be back :) [15:28:35] yay [15:28:38] well that was painful [15:28:58] I was waiting for someting to show up in one of the logs. :-D [15:29:19] so why would formey replication being broken also break ircbot? [15:29:30] <^demon> Unrelated, probably. [15:29:32] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [15:29:45] but [15:29:50] I have restarted the bot twice already [15:29:53] ottomata: if you change the module in any way (like that default parameter), do that in a separate commit [15:30:05] no, three times [15:30:19] and now after you touch things it suddenly works? [15:30:20] <^demon> Dunno. [15:30:32] <^demon> I'm just that good? ;-) [15:30:36] :-D [15:31:08] ^demon knows where it's /special/ place is [15:31:34] <^demon> Oh, I know all of gerrit's special places. [15:31:37] ewww [15:31:41] this is a family channel [15:32:29] ok mark, I can do that [15:32:43] buuuut, gotta ask, is that really better? is it better to have a commit that is sorta kinda 'broken' [15:32:43] ? [15:32:53] upstream rsync module w/o xinetd? [15:32:53] it's not broken, that's bs [15:33:16] i mean, doesn't really matter since no one will revert to that commit, but meh? [15:33:19] but it can be helpful to have a commit which is exactly like upstream [15:33:20] seems weird to me [15:33:28] ja i could see that [15:33:35] yeah, i guess they have them in different repos anyway [15:33:36] ok ok ok [15:34:05] but the rsync module works, you just can't use it with certain parameters [15:34:16] it's unfortunate that that is the default parameter, but ok, we'll change that [15:34:27] we're essentially forking the module that way though [15:34:33] yes [15:34:42] means it's a one-off import, any further enhancements/bugfixes to that module we'll have to three-way merge [15:34:48] and it can be handy to be able to revert that fork commit [15:34:54] for this reason too [15:35:41] mark, just curious what you think about this, I had originally committed this as git submodules, instead of fully importing [15:35:55] in general, do you prefer manually commits of 3rd party modules like this? [15:36:05] or would a gerrit mirror + git submodule be better? [15:36:11] we're not setup for git submodules currently [15:36:16] ottomata: submodules with third-party repos (github) is not going to happen [15:36:18] so you can't use them now, it's as simple as that [15:36:21] and indeed [15:36:25] right, which is why I said gerrit mirror [15:36:27] 3rd party repos are completely out of the question [15:36:33] submodules within our gerrit, it's something we should talk about [15:36:37] how are we not setup? meaning people don't now how to use them? [15:36:37] yes [15:36:57] ottomata: our processes have "git fetch", which won't fetch them afaik [15:37:01]