[00:15:07] !log imported jouncebot from github - https://gerrit.wikimedia.org/r/#/q/project:wikimedia/bots/jouncebot,n,z [00:15:11] Logged the message, Master [00:20:08] mwalker: https://gerrit.wikimedia.org/r/#/c/149201/1 [00:43:32] (03PS4) 10Dzahn: wikimedia.ee - own zonefile and set external MX [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 [00:50:10] mutante, if you have the permissions; we should probably add a 'Jouncebot' component to the Wikimedia bugzilla product [00:52:24] (03PS4) 10Dzahn: add index.html pages for various directories on dataset hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/144640 (owner: 10ArielGlenn) [00:55:53] mwalker: i don't, andre__ wants tickets for component changes, but https://bugzilla.wikimedia.org/show_bug.cgi?id=68549 [00:56:14] hahah! https://bugzilla.wikimedia.org/show_bug.cgi?id=68548 [00:56:18] jinx! [01:13:44] (03PS5) 10Jeremyb: wikimedia.ee - own zonefile and set external MX [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [01:17:56] (03PS1) 10Ori.livneh: Disable LuaSandbox's profiling feature, to isolate bug 68413 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149211 [01:18:36] (03CR) 10Ori.livneh: [C: 032] Disable LuaSandbox's profiling feature, to isolate bug 68413 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149211 (owner: 10Ori.livneh) [01:18:40] (03Merged) 10jenkins-bot: Disable LuaSandbox's profiling feature, to isolate bug 68413 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149211 (owner: 10Ori.livneh) [01:20:22] the commit message subject is misleading -- the change above is scoped to labs [01:21:57] !log ori Synchronized wmf-config/CommonSettings.php: Ic29ae11fa: On Labs, disable LuaSandbox's profiling feature to isolate bug 68413 (duration: 00m 04s) [01:22:04] Logged the message, Master [01:33:28] (03CR) 10Ori.livneh: "..but it's okay, because destBuffer doesn't get used, in that case. Removing my -1" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149119 (owner: 10BBlack) [01:35:41] (03PS6) 10Jeremyb: wikimedia.ee - own zonefile and set external MX [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [01:36:36] (03CR) 10Ori.livneh: [C: 031] "Tested this by extracting the code into a self-standing program (https://gist.github.com/atdt/3296446ed6e854a44f08) and it appears to work" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149119 (owner: 10BBlack) [01:37:14] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [01:53:08] (03CR) 10Jeremyb: [C: 031] "the source that you had copied from (wikimedia.com) had been retabbed in the meantime. update to be closer to the new version." [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [01:59:14] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 1 failures [02:10:58] (03PS1) 10Springle: Make config changes applied on labsdb1002 with SET GLOBAL permanent. [operations/puppet] - 10https://gerrit.wikimedia.org/r/149215 [02:14:09] (03CR) 10Springle: [C: 032] Make config changes applied on labsdb1002 with SET GLOBAL permanent. [operations/puppet] - 10https://gerrit.wikimedia.org/r/149215 (owner: 10Springle) [02:16:14] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:33:12] (03PS1) 10Springle: Remove mysql_multi_instance from labsdb1002 after migration. [operations/puppet] - 10https://gerrit.wikimedia.org/r/149218 [02:34:05] (03CR) 10Springle: [C: 032] Remove mysql_multi_instance from labsdb1002 after migration. [operations/puppet] - 10https://gerrit.wikimedia.org/r/149218 (owner: 10Springle) [02:48:20] !log LocalisationUpdate completed (1.24wmf14) at 2014-07-25 02:47:17+00:00 [02:48:28] Logged the message, Master [03:23:52] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 1 failures [03:31:37] !log LocalisationUpdate completed (1.24wmf15) at 2014-07-25 03:30:33+00:00 [03:31:42] Logged the message, Master [03:38:12] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [03:41:52] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [04:19:51] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 25 04:18:45 UTC 2014 (duration 18m 44s) [04:19:57] Logged the message, Master [05:02:55] PROBLEM - RAID on virt1009 is CRITICAL: CRITICAL: Active: 14, Working: 14, Failed: 2, Spare: 0 [05:13:33] (03PS2) 10Chmarkine: Make lists.wikimedia.org HTTPS only [operations/puppet] - 10https://gerrit.wikimedia.org/r/145616 (https://bugzilla.wikimedia.org/68553) (owner: 10JanZerebecki) [05:38:55] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [05:47:18] bblack, around? Will you have time in the next 10+ hrs to try switching a large partner to unified? [05:47:29] i'm in russia, will be testing on the ground [05:48:32] I'm somewhat around if you wanna ping me for easy reviews and merges, but not for deep stuff [05:50:19] bblack, nothing in-depth, just need to be able to roll back quickly if it fails :) [05:50:23] (03PS1) 10Ori.livneh: $wgImageMagickTempDir: /a/magick-tmp => /tmp/magick-tmp [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149253 [05:50:46] when would be a good time? I'm boarding a plane right now, shoud be with a proper sim card in about 3-4 hours [05:51:05] (03CR) 10Ori.livneh: [C: 032] $wgImageMagickTempDir: /a/magick-tmp => /tmp/magick-tmp [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149253 (owner: 10Ori.livneh) [05:51:09] (03Merged) 10jenkins-bot: $wgImageMagickTempDir: /a/magick-tmp => /tmp/magick-tmp [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149253 (owner: 10Ori.livneh) [05:51:32] than i will push a varnish patch and start testing [05:52:03] if it fails, might have to roll back, since i can't do a php depl today [05:52:07] bblack, ^ [05:52:23] yeah ok [05:52:41] well, ideal would be like 8 hours from now [05:52:49] I may or may not sleep sometime between now and then [06:28:55] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:15] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:15] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:25] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:25] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:35] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:35] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:45] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:55] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:55] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:55] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:44:59] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:45:28] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:45:29] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:45:38] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:45:38] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:59] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:46:59] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:47:29] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:47:38] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:52:59] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: Puppet has 1 failures [07:09:59] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:11:27] (03Abandoned) 10Ori.livneh: wmflib: add safe_filename() [operations/puppet] - 10https://gerrit.wikimedia.org/r/148723 (owner: 10Ori.livneh) [07:12:38] (03Abandoned) 10Ori.livneh: wmflib: add apt_version() [operations/puppet] - 10https://gerrit.wikimedia.org/r/148512 (owner: 10Ori.livneh) [07:20:54] anyone awake in here? [07:21:39] akosiaris, are you around? [07:21:58] I'm looking for a root to shutdown tantalum for me [07:27:04] apergos, ? [07:28:31] mwalker: shutdown or restart? [07:28:48] just shut it down -- it's fighting with the "real" ocg servers [07:28:53] ok [07:29:01] thanks! :) [07:29:23] !log shutdown tantalum per mwalker request [07:29:29] Logged the message, Master [07:29:53] :'( its been a fun ride tantalum, but the big boys have replaced you [07:30:01] :) [07:30:48] PROBLEM - Host tantalum is DOWN: PING CRITICAL - Packet loss = 100% [07:36:33] https://integration.wikimedia.org/ci/job/cxserver-npm/235/console - no space? needs hashar :) [07:39:48] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [07:53:44] hi :) [07:53:58] mwalker: you know LVS is still pointed at tantalum and not ocg100[123] right? [07:54:30] switch it over? [07:54:38] yes please :) [07:55:05] traffic should start moving shortly [07:55:09] (it's not urgent though; nothing is actively using that fqdn unless I poke it with a script) [07:59:28] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 1 failures [08:00:22] (03CR) 10Hashar: [C: 031] "Hopefully the scap target will still work :-]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146091 (owner: 10BBlack) [08:02:39] (03PS5) 10BBlack: beta::natfix removal step 2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146091 [08:03:36] (03CR) 10jenkins-bot: [V: 04-1] beta::natfix removal step 2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146091 (owner: 10BBlack) [08:04:34] (03PS6) 10BBlack: beta::natfix removal step 2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146091 [08:04:46] yeah yeah I know jenkins :P [08:05:36] (03CR) 10BBlack: [C: 032] beta::natfix removal step 2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/146091 (owner: 10BBlack) [08:25:57] (03PS1) 10Chmarkine: icinga-admin -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/149267 (https://bugzilla.wikimedia.org/53259) [08:31:28] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [08:53:31] (03PS1) 10Mwalker: Allow world read/write + sticky on ocg tmpfs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149274 [08:54:29] bblack, want to review and merge a puppet change for me? :D [08:55:43] or maybe springle? [08:56:07] * mwalker debates if it might not be better to just go to bed [09:02:19] (03PS1) 10Filippo Giunchedi: add swift account for search backups [operations/puppet] - 10https://gerrit.wikimedia.org/r/149276 [09:05:15] <_joe_> mwalker: if you need assistance, I'm here [09:05:45] _joe_, I have a patch :) it fixes some rendering permissions issues: https://gerrit.wikimedia.org/r/149274 [09:06:39] <_joe_> ok I need to wrap my head around the whole problem [09:06:48] <_joe_> what is the problem you're trying to solve? [09:06:58] <_joe_> a tmpfs only writable by root? [09:07:43] uh no; I have a process that runs in the ocg group that needs to write to the tempfs [09:08:00] right now the tempfs is writeable only by root [09:08:02] <_joe_> so, why not changing the dir permission to match the group? [09:08:17] <_joe_> and why you need it to be 1777? [09:08:34] <_joe_> 777 is a security liability [09:08:39] <_joe_> most of the times [09:08:57] 1777 is the default for /tmp -- I wasn't sure if I could have the tmpfs mount on a folder that wasn't owned by root so I didn't change the owner/group [09:09:02] <_joe_> sorry to ask so many questions [09:09:11] <_joe_> but I've never looked at this subsystem [09:09:14] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] add swift account for search backups [operations/puppet] - 10https://gerrit.wikimedia.org/r/149276 (owner: 10Filippo Giunchedi) [09:09:34] <_joe_> ok is it incredibly late in your TZ? [09:09:43] heh; yep :) but that's not a problem [09:09:50] <_joe_> if this is the case, I'll take a look at the patch, I may modify it and merge it [09:09:58] <_joe_> I understood what's your need [09:10:15] <_joe_> let me understand if there is a better way to do this - I suspect there is [09:10:35] possibly! [09:11:01] (maybe mounting it 1440 with ocg/ocg is possible -- and probably preferred) [09:11:17] I can test in labs; I just took a shortcut [09:11:20] * mwalker looks innocent [09:11:36] <_joe_> :) [09:11:40] <_joe_> don't worry [09:11:53] <_joe_> I'll test this, the requirement is clear [09:12:11] *er, that would need to be 1660 (I need read/write) [09:12:14] <_joe_> btw [09:12:14] awesome; thanks :) [09:12:26] <_joe_> you can specify the mount permissions [09:12:32] <_joe_> when mounting a tmpfs [09:12:45] <_joe_> (reading mount(1) manual and related resources) [09:15:44] <_joe_> (Also, I don't get why you need the sticky bit there, but it adds security so it's good in some respect) [09:16:51] <_joe_> and, you have to set the owner of the tmpfs in mounting it [09:17:26] (sticky bit was mostly if it was open to other users/groups) [09:18:09] (03PS1) 10QChris: Document database pool size parameters [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/149277 [09:18:29] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 1 failures [09:19:19] (03CR) 10QChris: [C: 031] "Looks good to me [1]." [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/149126 (https://bugzilla.wikimedia.org/68534) (owner: 10Milimetric) [09:20:23] <_joe_> mwalker: btw, default mount mode for tmpfs is 1777 [09:20:40] <_joe_> so - where are you having problems at the moment? [09:21:06] <_joe_> tmpfs is by default owned by root and in mode 1777 [09:21:28] I suspect puppet is changing the permissions due to the $file {tmpfs_mountpoint statement [09:21:28] <_joe_> if you don't pass it -o mode=something [09:21:36] but; if you want to see this on a real host; try ocg1001 [09:21:40] .eqiad.wmnet [09:21:43] <_joe_> yes thanks [09:22:18] <_joe_> the permissions of the mountpoint before mounting are irrelevant btw [09:22:50] but can puppet change the permissions back after it's been mounted [09:23:02] (03CR) 10Filippo Giunchedi: [C: 031] contint: setup localhost.qunit vhost on lanthanum [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 (https://bugzilla.wikimedia.org/68529) (owner: 10Hashar) [09:23:20] godog: we can get it merged in :-) [09:23:31] since I am around [09:24:09] <_joe_> mwalker: mmmh *very* strange [09:24:22] <_joe_> ok, puppet is braindead [09:24:57] hashar_: sure [09:25:26] godog: I will force run puppet on the server and verify apache works fine [09:25:52] <_joe_> mwalker: I refuse to think there is no way around this stupidity :| [09:25:59] hah [09:26:21] <_joe_> basically puppet is so idiotic it does not see the dir has become a mountpoint [09:26:23] can you create a file in puppet without specifying the user/group/permissions (so therefore they're implicit and puppet doesn't change them every 20 minutes) [09:26:35] <_joe_> mwalker: AFAIR no [09:26:41] *facedesk* [09:26:43] <_joe_> but lemem check [09:27:22] <_joe_> http://docs.puppetlabs.com/references/latest/type.html#file-attribute-mode [09:27:50] <_joe_> ok obviously this is not clear [09:27:56] * _joe_ testing [09:28:44] <_joe_> the real gold nugget is "When setting numeric permissions for directories, Puppet sets the search permission wherever the read permission is set" [09:28:59] <_joe_> which is them assuming puppet users are morons [09:29:47] (03PS1) 10QChris: Enable Apache's headers module [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/149279 [09:30:36] hashar_: kk, ping me here when done [09:30:57] godog: just need the patch to be merged in Gerrit and on palladium :] [09:31:15] (03CR) 10QChris: "In case you find this change when searching how to add" [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/144761 (owner: 10Milimetric) [09:32:10] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] contint: setup localhost.qunit vhost on lanthanum [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 (https://bugzilla.wikimedia.org/68529) (owner: 10Hashar) [09:32:34] hashar_: we'll need another +1 from a human, _joe_ by any chance? :) [09:32:59] <_joe_> mwalker: ok removing the mode line and fixing it by hand on the servers should be enough [09:33:04] <_joe_> I did some tests [09:33:09] shiny [09:33:29] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [09:33:42] its ocg100[1-3] that will need to be scrubbed [09:34:30] <_joe_> mwalker: we would just need to remount the tmpfs [09:34:33] godog: I am sure it is harmless. Apache is not used on that server, it just run tests :] [09:34:39] <_joe_> mwalker: are those in production already? [09:34:46] nope [09:34:55] <_joe_> godog: 1 minute please :) [09:34:57] godog: the change is there to enable the vhost we are using for testing. just be bold! [09:35:48] (03CR) 10QChris: [C: 04-1] "In addition to the inline comments, the wikimetrics" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/149127 (https://bugzilla.wikimedia.org/68534) (owner: 10Milimetric) [09:36:00] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Your change will accomplish what you need but in the wrong way. Just remove the line assigning explicit permissions. Mount will then mount" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/149274 (owner: 10Mwalker) [09:36:12] <_joe_> mwalker: If you want to rest, I can take care of this [09:36:38] <_joe_> godog: which change needs my attention? I have 50 of them :P [09:36:38] my goal tonight is to get the servers rendering documents again! I shall not rest! [09:36:46] <_joe_> ok ok [09:36:51] <_joe_> I know the deal [09:37:00] <_joe_> so, go on remove that line [09:37:13] <_joe_> and in ~20 minutes they'll be up and runnign [09:38:06] _joe_: https://gerrit.wikimedia.org/r/#/c/149105/ :D [09:38:26] <_joe_> hashar_: taking a look - you're just moving things around basically [09:38:29] adds some Apache vhost on lanthanum, the second jenkins slave in prod. Apache is unused right now, that will let us use it [09:38:30] <_joe_> and changing permissions [09:38:49] and puppet catalog compiler detects no change on gallium (the other server) [09:38:53] I am sure it is going to work :-] [09:38:57] <_joe_> not even permissions [09:39:01] else I will write somewhere I owe you both a beer [09:39:03] <_joe_> it's good for me [09:39:10] (03PS2) 10Mwalker: Allow world read/write + sticky on ocg tmpfs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149274 [09:39:29] (03CR) 10Giuseppe Lavagetto: [C: 031] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 (https://bugzilla.wikimedia.org/68529) (owner: 10Hashar) [09:39:49] (03PS3) 10Giuseppe Lavagetto: Allow world read/write + sticky on ocg tmpfs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149274 (owner: 10Mwalker) [09:39:58] (03CR) 10Giuseppe Lavagetto: [C: 032] Allow world read/write + sticky on ocg tmpfs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149274 (owner: 10Mwalker) [09:40:14] (03CR) 10Giuseppe Lavagetto: [V: 032] Allow world read/write + sticky on ocg tmpfs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149274 (owner: 10Mwalker) [09:40:25] <_joe_> I hate you jenkins [09:40:48] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [09:41:23] heh; I typically imagine jenkins as a crotchety old man hobbling around on a walker [09:43:01] (03PS3) 10Filippo Giunchedi: contint: setup localhost.qunit vhost on lanthanum [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 (https://bugzilla.wikimedia.org/68529) (owner: 10Hashar) [09:43:13] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] contint: setup localhost.qunit vhost on lanthanum [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 (https://bugzilla.wikimedia.org/68529) (owner: 10Hashar) [09:43:51] oops, _joe_ good to puppet-merge ? [09:45:40] <_joe_> yes [09:45:52] <_joe_> sorry [09:46:50] ugh... "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: File[/var/log/syslog] is already declared in file /etc/puppet/modules/ocg/manifests/init.pp:195; cannot redeclare at /etc/puppet/modules/base/manifests/init.pp:106 on node i-00000396.eqiad.wmflabs" [09:46:55] I think I will just have to go to bed [09:47:09] <_joe_> mwalker: ugh [09:47:10] _joe_: kk merged both [09:47:45] <_joe_> mwalker: I'm confirming how puppet works, I'm applying the change on all nodes [09:47:50] <_joe_> 5 mins and we're ok [09:48:11] *crossing fingers* things work in prod better than in beta [09:48:31] godog: thanks :-) [09:48:48] though I broke puppet now haha [09:48:49] <_joe_> wow ocg1001 is not in salt, :( [09:48:57] hashar: np [09:49:16] _joe_, should be... I deploy OCG using trebuchet [09:49:43] <_joe_> mwalker: ok it gave no output for test.ping [09:50:10] <_joe_> mwalker: running puppet to confirm [09:50:59] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 1 failures [09:52:00] <_joe_> mwalker: it is fixed now [09:52:02] (03PS1) 10Hashar: contint: /srv/localhost should belong to jenkins-slave [operations/puppet] - 10https://gerrit.wikimedia.org/r/149281 [09:52:04] yaaaay [09:52:07] thanks so much [09:52:10] \O/ [09:52:16] * mwalker proceeds to torture test OCG some more [09:52:35] <_joe_> glad to help [09:53:37] godog: _joe_ : and some lame follow up to my previous patch https://gerrit.wikimedia.org/r/#/c/149281/ 'jenkins' user does not exist everywhere :-D [09:54:12] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] contint: /srv/localhost should belong to jenkins-slave [operations/puppet] - 10https://gerrit.wikimedia.org/r/149281 (owner: 10Hashar) [09:54:15] hashar: hahah okay [09:54:31] done [09:54:59] <_joe_> ok good [09:55:03] <_joe_> back to hiera [09:56:07] sorry :-( [09:56:29] _joe_: does hiera bring support for puppet environments or is that unrelated? [09:56:30] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 1 failures [09:56:47] <_joe_> mmmh checking tin [09:57:05] <_joe_> hashar: what do you mean with puppet environments? [09:57:42] _joe_: apparently you can shared nodes in different 'envs', much like the $::realm we are using [09:57:48] and each env can come with its own set of manifests [09:58:29] not hiera :-) forget me [09:58:41] <_joe_> oh ok, yes, I thought we used that [09:58:41] some basic doc at http://docs.puppetlabs.com/puppet/latest/reference/environments.html but that will divert you from working on hiera [09:58:54] <_joe_> realm is basically our version of environment [09:58:58] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [09:59:03] <_joe_> we do that by ourselves :) [09:59:19] <_joe_> hashar: I know what environments are [09:59:45] <_joe_> directory environments, never used them [10:00:15] I am wondering whether it could be used instead of puppetmaster::self for labs :D [10:00:29] and have each labs project use its own git repo in addition of operations/puppet. [10:00:32] but that is digressing [10:00:46] (03PS2) 10QChris: Use wikimetrics' new configurable pool sizes [operations/puppet] - 10https://gerrit.wikimedia.org/r/149127 (https://bugzilla.wikimedia.org/68534) (owner: 10Milimetric) [10:00:48] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:20] (03CR) 10QChris: Use wikimetrics' new configurable pool sizes (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/149127 (https://bugzilla.wikimedia.org/68534) (owner: 10Milimetric) [10:03:56] (03CR) 10QChris: "Since it seems we want to press this hard into production," (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/149127 (https://bugzilla.wikimedia.org/68534) (owner: 10Milimetric) [10:04:16] (03PS1) 10Hashar: contint: qunit local vhost needs mod_rewrite [operations/puppet] - 10https://gerrit.wikimedia.org/r/149283 [10:06:38] PROBLEM - Disk space on ocg1003 is CRITICAL: DISK CRITICAL - free space: /mnt/tmpfs 2 MB (0% inode=99%): [10:08:42] (03CR) 10QChris: [C: 04-1] "Bumping the wikimetrics module is still missing." [operations/puppet] - 10https://gerrit.wikimedia.org/r/149127 (https://bugzilla.wikimedia.org/68534) (owner: 10Milimetric) [10:09:38] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 53278 bytes in 0.452 second response time [10:09:42] (03CR) 10Hashar: "Puppet compilation: http://puppet-compiler.wmflabs.org/184/change/149283/html/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149283 (owner: 10Hashar) [10:09:58] PROBLEM - Disk space on ocg1001 is CRITICAL: DISK CRITICAL - free space: /mnt/tmpfs 64 MB (1% inode=99%): [10:09:58] PROBLEM - Disk space on ocg1002 is CRITICAL: DISK CRITICAL - free space: /mnt/tmpfs 35 MB (0% inode=99%): [10:13:35] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] contint: qunit local vhost needs mod_rewrite [operations/puppet] - 10https://gerrit.wikimedia.org/r/149283 (owner: 10Hashar) [10:13:46] sorry that is messy :( [10:14:20] hashar: not to worry :) [10:14:52] the contint manifests are not very robust [10:15:00] and there is a bunch of leftover tech debt :( [10:21:44] as long as we are decreasing the debt I think we're fine :)) [10:26:47] yeah [10:26:53] apache all happy on both servers \O/ [10:27:11] some paperwork / lunch and will be back later this afternoon [10:33:09] I am off [10:37:28] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:41:06] hashar: How do you trigger puppet-compiler in labs? And how does it know for which nodes to try identical/diff ? [10:41:16] e.g. yours only ran for gallium and lanthanum [10:41:38] <_joe_> Krinkle: what do you mean with 'in labs'? [10:41:59] <_joe_> Krinkle: you want to run it against labs hosts? I guess that's not supported at the moment [10:44:46] grrrit-wm: (CR) Hashar: "Puppet compilation: http://puppet-compiler.wmflabs.org/184/change/149283/html/" [operations/puppet] - https://gerrit.wikimedia.org/r/149283 (owner: Hashar) [10:44:48] _joe_: [10:45:13] That puppet-compiler thing runs in labs, it doens't just run willy nilly for every puppet change simulating every prorduction node we hae [10:45:14] it's selective [10:46:57] !log integration-slave1001.eqiad.wmflabs is out of disk space ( / /dev/vda1) [10:47:02] Logged the message, Master [10:47:57] <_joe_> Krinkle: oh ok you want to run it [10:47:59] <_joe_> sorry :) [10:48:13] <_joe_> Krinkle: https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ [10:48:20] <_joe_> ->> build with params [10:48:29] <_joe_> or look at the parameters of the last runs [10:48:38] Well, no, I just want to know how it works, and perhaps tell someone to document it because neither individual runs nor the root dir http://puppet-compiler.wmflabs.org/ show any kind of information [10:48:45] <_joe_> when you trigger the build, look at the console output [10:49:13] <_joe_> Krinkle: I should be the one documenting it [10:49:22] <_joe_> I was pretty sure I had updated wikitech [10:49:38] Is it for detecting changes beteeen puppet 2 and 3, or also for the effective diff within one puppet version before/after this change? [10:50:00] s/this change/the given change/ [10:50:07] <_joe_> It is born for puppet 2 -> puppet 3, now we use it just for diffs between prod and a change [10:50:13] cool [10:50:30] <_joe_> it's not documented in that "website" as that is just a static collection of outputs :P [10:50:47] <_joe_> It's intended only to be run through jenikns nowadays [10:50:56] !log contint: manually cleared /tmp on the 3 labs jenkins slaves. [10:51:00] <_joe_> or in vagrant if you prefer but that is untested [10:51:01] Logged the message, Master [10:51:14] Sure, but could make an index.html with a link to https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ and/or a link to the instance project page on wikitech. [10:52:07] there isn't a puppet-compiler wmflabs project, I guess it is puppet3-diff? [10:52:07] https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet3-diffs [10:52:15] <_joe_> it is [10:52:23] <_joe_> I may change the name of the project [10:52:30] <_joe_> and I have to document it a little there [10:52:55] <_joe_> but - I should just write a page on wikitech about how to use the compiler to test your changes [10:53:12] * _joe_ adds to the endless queue of very important things to do [10:53:29] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 1 failures [10:56:44] _joe_: I added an initial description [10:57:26] _joe_: Could you add me as member? One or more of its hosts are connected to Jenkins, and in order to perform general maintenance it's probably useful to be able to connect to the instance myself as well. [10:57:44] <_joe_> Krinkle: of course I can [10:57:49] <_joe_> that host needs some love btw [10:57:53] <_joe_> I should work on it [10:58:43] The only thing I expect to do at this point is dump my dotfiles template and be able in the future to possibly stop/start jenkins-slave connection if needed. [11:00:34] hashar: Hm.. as of today all links on pages like https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ have an underline. Makes the text a bit unreadable. Do you know anything about that and/or other updates? [11:01:21] no clue [11:01:55] Krinkle: jenkins has a plugin that loads https://integration.wikimedia.org/jenkins.css [11:02:04] should be in integration/docroot.git [11:02:11] as for why the underline suddenly appeared I have no idea [11:02:59] I am off for lunch [11:22:35] (03CR) 10Giuseppe Lavagetto: "nitpick but apart from that, LGTM" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/149066 (owner: 10Ori.livneh) [11:25:09] <_joe_> lunch. bbl. [11:25:46] (03PS1) 10Reedy: Add export-0.8 to index.html [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149289 (https://bugzilla.wikimedia.org/68561) [11:26:14] (03CR) 10Reedy: [C: 032] Add export-0.8 to index.html [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149289 (https://bugzilla.wikimedia.org/68561) (owner: 10Reedy) [11:26:18] (03Merged) 10jenkins-bot: Add export-0.8 to index.html [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149289 (https://bugzilla.wikimedia.org/68561) (owner: 10Reedy) [11:26:45] !log reedy Synchronized docroot and w: (no message) (duration: 00m 13s) [11:26:51] Logged the message, Master [11:36:58] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.014 second response time [11:39:58] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.359 second response time [11:41:48] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [11:59:29] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 2 failures [12:11:29] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [12:51:54] bblack, around? [12:54:00] (03PS3) 10Giuseppe Lavagetto: mediawiki/apache: use ports.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/148098 [12:57:29] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 1 failures [13:03:25] <_joe_> !log stopping puppet on all appservers - will reactivate after testing [13:03:31] Logged the message, Master [13:04:15] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki/apache: use ports.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/148098 (owner: 10Giuseppe Lavagetto) [13:08:59] <_joe_> !log re-enabling puppet, test run on the test host was fine. [13:09:04] Logged the message, Master [13:12:28] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:16:43] _joe_ Coren: you do know that puppet does not merge complex hierarchies on hiera ? [13:17:06] akosiaris: Yeah, I know it's first found. [13:17:49] Coren: kind of worse. IIRC it will not drill down to a complex hierarchy [13:18:07] complex == multilevel [13:18:13] for what I am saying [13:18:33] I stumbled across that with a friend of mine while playing with it [13:18:51] let me see if I can find the exact case [13:19:09] <_joe_> akosiaris: I know [13:19:38] <_joe_> akosiaris: the only way to create complex hierarchies it to create them in the backend directly, which kinda sucks [13:26:32] (03PS1) 10Ottomata: Install hive on hadoop workers to get hive-hcatalog packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/149308 [13:26:47] (03CR) 10Ottomata: [C: 032 V: 032] Install hive on hadoop workers to get hive-hcatalog packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/149308 (owner: 10Ottomata) [13:28:21] _joe_: (think of RUBY 1.8 scanning and parsing yaml files for every puppet compilation on the master, then cry). <-- priceless [13:29:34] oh hey, akosiaris, yt? [13:30:05] ottomata: yes [13:30:24] _joe_: matanya that got me thinking. Our new puppetmaster packages do not depend on ruby1.8 anymore [13:30:28] so, there was a use case a couple of weeks ago where it would have been nice to have a kafka-client package [13:30:38] which is really just the same package [13:30:43] but without the init.d script [13:30:45] we could switch to 1.9 ruby (I hope) [13:30:59] dan wanted to check the kafka logs for something [13:31:09] and i couldn't get him an easy way to do it [13:31:17] logs ? [13:31:18] without either giving him shell access to a broker [13:31:19] soryr [13:31:21] data [13:31:24] data in kafka [13:31:27] he wanted to consume on the cli [13:31:32] akosiaris: we can't we still have some .rb thing that where removed from 1.9 iirc [13:31:33] ok [13:31:40] so yeah, without giving him shell acecss to broker, or installing the kafka package on stat1002 [13:31:43] which is fine, iguess [13:31:48] PROBLEM - puppetmaster backend https on strontium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:55] but then there'd be the init script and default left at KAFKA_START=no [13:32:17] i think I tried to make a kafka client package before, a while ago [13:32:24] ottomata: sorry looking at the alert, gimme 5 [13:32:27] np [13:32:38] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.025 second response time [13:32:47] and you and faidon didn't like it...mainly because I made 'kafka' package have everythign and 'kafka-server' was just the init scripts [13:35:00] akosiaris: e.g modules/stdlib/spec/unit/puppet/parser/functions/range_spec.rb has to_a which was removed from 1.9 [13:35:33] matanya: we should anyway upgrade that module [13:35:39] not that it is going to be easy [13:35:44] agreed [13:35:57] rsync module has the same issue [13:37:18] and to all ops here: Happy sysadmin day! [13:38:08] ottomata: so I figured out nothing obvious from the alert ... sigh... [13:38:34] (03PS1) 10Yurik: Moved 250-99 to unified config [operations/puppet] - 10https://gerrit.wikimedia.org/r/149310 [13:38:55] bblack, ^, but ping me when you merge it so that i can test it right away [13:38:58] anyway from what I see now the kafka package has everything inside (client/server/common code - about 99% of it) [13:39:35] If you do want it can be split up the to 3 binary packages (server, client, common) [13:39:38] aye [13:39:40] the source package will be one again [13:39:58] yes, source package one, could we just have two packages? [13:40:05] kafka-client kafka-server? [13:40:09] or, just kafka and kafka-server? [13:40:19] kafka would be evertying but init scripts [13:40:26] I don't think it would be wise [13:40:35] it would work but not clear [13:40:42] so common should have all the jars [13:40:43] ok, what would be in client and common? [13:40:44] ah [13:40:52] client the script you wrote [13:40:57] and server the init scripts [13:41:13] so basically 99% of the package will go into common [13:41:17] hm, ok, that's fine with me, just curious though, why is that more clear? [13:41:36] because my way kafka-client would have stuff that coudl be used for server too [13:41:37] ? [13:41:44] yes and vice versa [13:41:54] hm, ok sure [13:41:55] which one would depend on which ? [13:42:06] you answer that easily but still [13:42:25] plus it is already been done by other packages out there so let's follow the stream [13:42:35] but does it really add something ? [13:42:43] doing it like that I mean [13:42:48] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [13:43:09] I get dan's problem and use case but he can just install the kafka package, can't he ? [13:43:24] it is not like he will get a daemon running [13:43:45] my point being - I don't think I have the time to do that now really [13:43:59] so feel free [13:44:14] ah! I forgot to turn puppet back on an27, that's my fault [13:44:16] (03PS1) 10Chmarkine: planet.wikimedia.org -- fix https redirects to http [operations/puppet] - 10https://gerrit.wikimedia.org/r/149311 (https://bugzilla.wikimedia.org/68554) [13:44:27] well, he can't, I can though [13:44:36] but it feels weird to puppetize it like that, [13:44:37] ottomata: btw I finished today the kafka reorganization I was telling you about [13:44:39] yeah, akosiaris i would do it [13:44:43] oh ok awesome [13:44:58] its kind of a big difference though [13:45:09] not codewise but I changed branch names etc [13:45:17] to conform to gbp's stuff [13:45:34] cool [13:45:47] ah, did you adopt the gbp convention of using master as debian branch? [13:45:51] yes [13:45:53] dawwww [13:45:56] i don't like that convention :/ [13:46:02] it works for kafka [13:46:07] neither did I [13:46:08] because their master is 'trunk [13:46:09] ' [13:46:28] but it conflicts with most other repos if you want to turn them in to gbp repos [13:46:46] yes because that is the wrong way to do it [13:47:07] howso? [13:47:15] turns out the best way is to not have local branches of the remotes [13:47:22] and have upstream for the upstream code [13:47:39] and tags? [13:47:49] upstream/ [13:47:55] debian/ [13:47:58] works nicely [13:48:00] hm, but then you have to maintain it all yourself! [13:48:11] you anyway have to [13:48:17] and it is not difficult [13:48:25] I have a readme ready [13:48:30] hm ok... [13:48:42] oh and debian/ is maintained by gbp [13:49:00] yes, i meant you have to maintain the upstream branch/tag mappings yourself [13:49:14] rathern than just git pull and git checkout......hmm or do you have to do that anyway too [13:49:27] i dunno, i mean, ooook!, it isn't a big deal either way [13:49:40] wait, so [13:49:41] master [13:49:43] has debian/ [13:49:44] only [13:49:48] and debian/ [13:49:51] not anymore [13:49:52] has tag + debian/ [13:49:53] ? [13:50:10] i mean, master has debian/ dir only [13:50:11] right? [13:50:13] no [13:50:15] no? [13:50:15] oh [13:50:17] it has everything [13:50:25] so you mostly just do [13:50:36] git tag upstream/ [13:50:45]