[00:21:19] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [00:39:19] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:00:29] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 28 Jul 2014 00:00:02 UTC [02:16:04] !log LocalisationUpdate completed (1.24wmf14) at 2014-07-28 02:15:00+00:00 [02:16:14] Logged the message, Master [02:26:38] !log LocalisationUpdate completed (1.24wmf15) at 2014-07-28 02:25:34+00:00 [02:26:44] Logged the message, Master [02:40:09] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jul 28 02:40:04 UTC 2014 [03:00:41] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 28 02:59:35 UTC 2014 (duration 59m 34s) [03:00:47] Logged the message, Master [03:26:10] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:59] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.004 second response time [03:32:04] (03CR) 10Vogone: "Undelete only covers viewing deleted page revisions through Special:Undelete. What it does not cover are actions through Special:RevisionD" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149637 (https://bugzilla.wikimedia.org/68612) (owner: 10Withoutaname) [03:58:29] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 28 Jul 2014 01:57:28 UTC [04:26:56] (03PS2) 10Ori.livneh: wmflib: add ordered_yaml() [operations/puppet] - 10https://gerrit.wikimedia.org/r/149775 [05:16:49] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 28 05:16:43 UTC 2014 [05:40:35] (03PS2) 10Ori.livneh: wmflib: add ensure_service() [operations/puppet] - 10https://gerrit.wikimedia.org/r/149778 [05:40:37] (03PS1) 10Ori.livneh: Nutcracker: move declaration to role::mediawiki; parametrize [operations/puppet] - 10https://gerrit.wikimedia.org/r/149800 [06:06:23] <_joe_> morning [06:06:49] <_joe_> ori: in stdlib there is ensure_resource [06:07:10] <_joe_> oh ok nevermind [06:07:12] <_joe_> :) [06:07:49] hey, morning [06:08:09] i'm just reading over aaron's patch to dispatch jobs via http requests [06:08:24] he sort of cheated :P [06:08:38] instead of shelling out to php...... it shells out to curl [06:08:40] <_joe_> I love cheating as long as it yields results [06:08:45] <_joe_> ewww [06:09:02] yeah, i'm not a fan [06:09:03] <_joe_> I thought he was using fastcgi [06:09:07] <_joe_> and not http [06:09:09] well "he is" [06:09:17] <_joe_> :) [06:09:18] because http -> apache -> fastcgi [06:09:22] <_joe_> yes, ok :) [06:09:29] but yeah, not the way to go imo [06:09:48] https://gist.github.com/wofeiwo/3720207 looks nice [06:09:52] <_joe_> what change are you talking about? [06:09:55] (fastcgi client library for python) [06:10:03] https://gerrit.wikimedia.org/r/#/c/149216 [06:10:55] the one thing i do like about it is that it provides an extremely easy way to compare the two approaches [06:11:09] <_joe_> ori: OTOH the good thing here would be [06:11:19] <_joe_> the jobrunners become services [06:11:30] <_joe_> and you can curl them from wherever you want [06:11:58] <_joe_> my fear is that very short jobs will lose time in shelling out curl, maybe [06:12:00] yes, i'm in favor of that part, but that's in different patches (going into mediawiki core) [06:12:33] <_joe_> not sure it's better than the current approach [06:12:43] yeah, i think the reason he did it is that it lets him keep the current implementation more or less in tact [06:12:50] <_joe_> (we're pulling jobs from the queue, which is smart and reliable) [06:13:00] because the assumption that you're spawning a subprocess and then waiting on it holds true [06:13:02] <_joe_> ori: which is a +1 from me for now [06:13:16] so it's a very cheap way to test the approach [06:13:45] yes, but a +1 with an ewww, as you said :) [06:13:45] <_joe_> couldn't we use curl from within php there? it's 10 lines of code more :) [06:13:58] but then it's not a subprocess [06:14:45] <_joe_> oh ok so you can't compare the relative speed of hhvm [06:15:02] <_joe_> but it could be faster just because of not shelling out [06:15:15] but it is shelling out [06:15:22] <_joe_> spawning a process is exensive in general, from php in particular [06:15:29] <_joe_> so I agree [06:15:54] <_joe_> this is the right approach to see how large are the benefits due to using fastcgi [06:16:05] yeah, but not something to keep [06:16:26] <_joe_> ori: when I have the new packages, I'll try them on mw1053 for a while [06:16:30] <_joe_> to see if it crashes [06:16:37] <_joe_> crash dumps are in /tmp right? [06:16:54] yep [06:17:28] this is the only unmerged PR we need: https://github.com/facebook/hhvm/pull/3249 [06:18:23] brett updated it ~14h ago, it'll probably get merged within the next 12-24h [06:19:10] <_joe_> brett works on weekends as well [06:19:42] <_joe_> I guess he likes this work :) [06:20:05] <_joe_> I've just had an almost-computer-less we, so I'm fresh :P [06:20:27] <_joe_> I hope to merge one or two of the apache patches today [06:26:01] ok cool, hope the weekend was fun [06:27:02] the nutcracker patch doesn't do anything we need urgently btw [06:27:51] so feel free to ignore that one [06:33:49] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:29] <_joe_> ori: it seems nice and goes in the right direction IMO [06:37:25] ah thanks, i'm glad you think so [06:39:57] <_joe_> ok, coffee, then hhvm [06:44:09] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:44:44] * Nemo_bis throws some sugar into _joe_'s hhvm cup [06:51:49] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:01:42] ori: will you be at wikimania? [07:03:09] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:07:01] YuviPanda: he will [07:07:05] legoktm: ah, cool [08:10:39] YuviPanda: (yes) [08:12:22] good morning [08:15:48] morning hashar [08:18:48] (03PS3) 10Ori.livneh: wmflib: add ensure_service() [operations/puppet] - 10https://gerrit.wikimedia.org/r/149778 [08:20:27] <_joe_> ciao hashar [08:22:30] * hashar ori: sleep sleep! :-D [08:23:02] _joe_: all the Zuul puppet patches I had got reviewed/merged/deployed last week with Alexandros :-] Thank you for the preliminary reviews [08:23:16] we got everything done in like half an hour. Gotta love well prepared patches [08:30:40] <_joe_> hashar: :) [08:34:23] (03PS1) 10Giuseppe Lavagetto: [wikimedia] Add patches by Tim [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149808 [08:34:25] (03PS1) 10Giuseppe Lavagetto: [wikimedia] update changelog [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149809 [08:34:27] (03PS1) 10Giuseppe Lavagetto: [wikimedia] Add init scripts, bump changelog [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149810 [08:34:29] (03PS1) 10Giuseppe Lavagetto: Imported Upstream version 3.1+20140723 [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149811 [08:34:31] (03PS1) 10Giuseppe Lavagetto: [wikimedia] Remove patches integrated in the tree, add PR #3121 and PR #3249 [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149812 [08:34:33] (03PS1) 10Giuseppe Lavagetto: [wikimedia] Remove merged patches [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149813 [08:34:48] <_joe_> eww [08:34:54] <_joe_> ok, some fake merging to do [08:34:55] <_joe_> :) [08:35:24] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] [wikimedia] Add patches by Tim [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149808 (owner: 10Giuseppe Lavagetto) [08:35:40] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] [wikimedia] update changelog [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149809 (owner: 10Giuseppe Lavagetto) [08:36:40] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] [wikimedia] Add init scripts, bump changelog [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149810 (owner: 10Giuseppe Lavagetto) [08:37:07] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Imported Upstream version 3.1+20140723 [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149811 (owner: 10Giuseppe Lavagetto) [08:37:43] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] [wikimedia] Remove patches integrated in the tree, add PR #3121 and PR #3249 [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149812 (owner: 10Giuseppe Lavagetto) [08:38:31] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] [wikimedia] Remove merged patches [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149813 (owner: 10Giuseppe Lavagetto) [08:40:10] labs-vagrant was already using 3.1+20140723-1+wmf1, nothing to do right? [08:40:59] <_joe_> Nemo_bis: nevermind these commits [08:41:12] <_joe_> it's me getting back the hhvm repo into track [08:41:19] <_joe_> with my work offline [08:41:28] :) [08:42:10] <_joe_> well, really with my tiny slice of work on top of paravoid's one [08:42:29] what's the most useful bit to mention when stating what version I'm using? 3.1+20140723-1+wmf1, 3.3.0-dev, heads/wikimedia-0-g8b842db4e2db664a9b4d543047ae154a6dd59de6 [08:42:36] , ce469da81c1d8ec23f3a4aa889afadad8df5a759 [08:42:52] <_joe_> the version of the package [08:43:01] <_joe_> dpkg -l hhvm [08:43:19] <_joe_> the next package will be using 3.3.0-dev [08:44:04] ok [08:44:11] <_joe_> the next package will be out in ~ 3-4 hours [08:57:29] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 28 Jul 2014 06:57:12 UTC [08:57:29] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 28 08:57:25 UTC 2014 [08:57:54] (03PS18) 10Legoktm: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) (owner: 10Yuvipanda) [08:58:18] (03CR) 10Legoktm: "PS18: Pass --all to nightly.py" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) (owner: 10Yuvipanda) [09:18:29] legoktm: doesn't need flask either, no [09:18:47] yeah :D [09:19:13] no dependencies now [09:19:32] legoktm: can you change config file format to be json or yaml? [09:19:37] oh [09:19:37] sure [09:19:40] json [09:19:43] legoktm: ok [09:20:29] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 28 Jul 2014 07:19:45 UTC [09:20:43] YuviPanda: ummmm, how do I know where conf.json is? :P [09:21:29] legoktm: ah, so you can check in /etc/extdist then dirname(__file__) [09:21:59] /etc/extdist/conf.json and then dirname(__file__)/conf.json? [09:22:51] legoktm: no, /etc/extdist.conf and then dirname(__file__)/conf.json? [09:22:59] gotcha [09:23:03] legoktm: make the log file path be configurable from the config as well [09:23:09] and I'll setup logging on /var/log/extdist [09:23:11] uhh, [09:23:20] right now it just uses whatever python's logging is defaulting to? [09:23:28] I haven't set a log dir anywhere [09:23:39] legoktm: right, I'm asking you to add code that sets that also ::P [09:23:45] oh ok :P [09:25:56] YuviPanda: er, what's the python version of __FILE__? [09:26:08] legoktm: __file__ [09:26:23] :P [09:26:43] oh, it's not set in interactive [09:26:49] oh, yeah [09:40:19] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jul 28 09:40:17 UTC 2014 [09:57:16] (03PS19) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [09:57:54] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) (owner: 10Yuvipanda) [10:00:44] (03PS20) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [10:04:20] <_joe_> eww I screwed up badly it seems with my git commits [10:04:38] <_joe_> ok let's do that again, from scratch [10:06:25] (03PS21) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [10:07:41] (03CR) 10Alexandros Kosiaris: [C: 032] wmflib: add ensure_service() [operations/puppet] - 10https://gerrit.wikimedia.org/r/149778 (owner: 10Ori.livneh) [10:25:20] (03PS22) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [10:27:23] (03PS23) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [10:33:15] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/149003 (owner: 10Matanya) [10:33:27] thanks godog [10:34:34] matanya: thank you! [10:38:07] (03PS1) 10Giuseppe Lavagetto: Imported Upstream version 3.3-dev+20140728 [operations/debs/hhvm] (upstream) - 10https://gerrit.wikimedia.org/r/149826 [10:41:34] (03PS24) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [10:41:57] (03PS25) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [10:43:19] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [10:45:29] <_joe_> godog: ^^ happened again [10:46:53] fascinating, git on strontium didn't know who I was [10:46:58] *** Please tell me who you are. [10:47:18] (I think it was on strontium) [10:47:53] http://paste.debian.net/hidden/4a58ee06/ [10:48:57] _joe_: what's the quickest fix? [10:49:23] <_joe_> godog: do git-merge on strontium [10:49:28] <_joe_> but we have to solve thisd [10:49:41] <_joe_> it's some issue local to strontium [10:54:19] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [10:56:30] mark: ? [10:56:41] (03Abandoned) 10Giuseppe Lavagetto: Imported Upstream version 3.3-dev+20140728 [operations/debs/hhvm] (upstream) - 10https://gerrit.wikimedia.org/r/149826 (owner: 10Giuseppe Lavagetto) [11:05:55] (03PS1) 10Giuseppe Lavagetto: Imported Upstream version 3.3-dev+20140728 [operations/debs/hhvm] (upstream) - 10https://gerrit.wikimedia.org/r/149837 [11:07:45] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Imported Upstream version 3.3-dev+20140728 [operations/debs/hhvm] (upstream) - 10https://gerrit.wikimedia.org/r/149837 (owner: 10Giuseppe Lavagetto) [11:32:47] bah what puzzles me about the failure on strontium is that it looks like it is attempting to commit and fail, indeed we do a forced command of git pull && git submodule update --init, the git pull should probably be --ff-only too [11:32:55] akosiaris: ^ [11:41:59] (03CR) 10Filippo Giunchedi: [C: 04-1] "given how short ordered_json I'm more inclined to duplicate it rather than introducing a possibly-confusing dependency (confusing e.g. whe" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149775 (owner: 10Ori.livneh) [11:52:25] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM, would be nice to have a link to the catalog compiler too" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148099 (owner: 10Giuseppe Lavagetto) [11:53:50] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] apache2: rename apache2_test_config => apache2_test_config_and_restart [operations/puppet] - 10https://gerrit.wikimedia.org/r/149344 (owner: 10Ori.livneh) [11:55:01] and now it failed like this: http://paste.debian.net/hidden/fa6fb0f7/ [11:57:55] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] wmflib: add ensure_service() [operations/puppet] - 10https://gerrit.wikimedia.org/r/149778 (owner: 10Ori.livneh) [12:00:16] mhh on https://gerrit.wikimedia.org/r/#/c/149778/ I'm hitting "submit ps3" gerrit thinks for a while and then the button becomes enabled again and nothing happened, have you seen this before? [12:01:18] godog: no, but gerrit is not exactly known for stability [12:01:30] that would also explain the just failed puppet-merge [12:01:38] (03CR) 10Filippo Giunchedi: [C: 031] mediawiki::multimedia: stop managing /a/magick-tmp; provision fontconfig-config [operations/puppet] - 10https://gerrit.wikimedia.org/r/149368 (owner: 10Ori.livneh) [12:02:04] true [12:02:29] godog: the patch has an unmerged dependency [12:02:44] https://gerrit.wikimedia.org/r/#/c/149775/2 [12:03:02] (why would gerrit not tell you that, i have no idea) [12:03:12] Status Submitted, Merge Pending [12:03:21] MatmaRex: ahah! [12:03:35] yeah, you afterward kind of guess why [12:04:10] bah, in retrospect it makes sense but no feedback is a bit meh [12:04:36] akosiaris: anyways, ideas on the merge vs ff above? [12:04:41] (tahnks btw) [12:18:12] (03PS26) 10Yuvipanda: [WIP] Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [12:28:15] !log Upgrading our Jenkins Job Builder fork ( d833015..666e953 ) [12:28:20] Logged the message, Master [12:31:51] (03CR) 10Filippo Giunchedi: swift: monitor object/container availability (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/149019 (owner: 10Filippo Giunchedi) [12:38:29] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 28 Jul 2014 10:37:40 UTC [12:56:20] apergos: Just for your interest: The new json dump is fine, despite your worrying about echo :) [13:06:01] (03PS1) 10Giuseppe Lavagetto: Imported Upstream version 3.3-dev+20140728 [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149850 [13:06:03] (03PS1) 10Giuseppe Lavagetto: Update the patchsets to apply cleanly [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149851 [13:06:05] (03PS1) 10Giuseppe Lavagetto: version bump; add postrm hook [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149852 [13:06:41] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Imported Upstream version 3.3-dev+20140728 [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149850 (owner: 10Giuseppe Lavagetto) [13:06:59] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.001 second response time [13:07:20] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Update the patchsets to apply cleanly [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149851 (owner: 10Giuseppe Lavagetto) [13:07:36] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] version bump; add postrm hook [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/149852 (owner: 10Giuseppe Lavagetto) [13:08:09] (03PS3) 10Giuseppe Lavagetto: Enable hhvm hotprofiler [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/148850 (owner: 10EBernhardson) [13:08:34] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Enable hhvm hotprofiler [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/148850 (owner: 10EBernhardson) [13:10:29] hoo: good to know! congrats [13:10:41] Thanks :) [13:15:59] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.009 second response time [13:36:59] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jul 28 13:36:55 UTC 2014 [13:37:34] (03PS11) 10Physikerwelt: WIP: Draft for Mathoid role [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 [14:08:09] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [14:19:13] (03PS1) 10Manybubbles: Beta builds Cirrus speed up field [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149859 [14:20:23] anyone mind if merge a beta-only change? [14:21:02] manybubbles: Of course, change is evil!!1 :D [14:21:23] (03CR) 10Manybubbles: [C: 032] Beta builds Cirrus speed up field [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149859 (owner: 10Manybubbles) [14:21:28] (03Merged) 10jenkins-bot: Beta builds Cirrus speed up field [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149859 (owner: 10Manybubbles) [14:21:37] hoo: with that attitude we'll get everything done [14:21:47] I'll do swat today any way [14:22:28] * hoo got things in for todays swat... so better stop trolling :P [14:22:37] :P [14:25:04] <^d> manybubbles is back :) [14:25:38] ^d: I'm so back! [14:26:57] <^d> And so you're back...from outer space.... [14:27:09] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:27:17] hello manybubbles [14:27:23] Nemo_bis: hi! [14:28:01] Krenair: would you mind building the submodule updates for your swat changes today? [14:28:04] it saves me some time:) [14:28:18] I'll +2 the backports ifyou'd like [14:29:17] (03CR) 10Manybubbles: [C: 031] Add import sources for bhwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149530 (https://bugzilla.wikimedia.org/68616) (owner: 10Hoo man) [14:30:50] (03CR) 10Manybubbles: [C: 031] Add 'abusefilter-log-detail' to 'rollbacker' and 'patroller' group at eswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149533 (https://bugzilla.wikimedia.org/68319) (owner: 10Vogone) [14:31:19] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There is one unmerged change in mediawiki_config (dir /a/common/). [14:32:31] manybubbles, hey [14:32:51] manybubbles, is there some page on wikitech explaining how to do that? [14:33:05] probably [14:33:16] Krenair: sure! https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule [14:33:23] ah yes [14:33:26] what manybubbles says [14:34:15] Krenair: I just +2ed on the release branches - you can follow the instructions to get them into the core release branch [14:34:20] that'd be super useful for me [14:34:36] I can totally do it but its nice for swat when that is already done [14:37:08] manybubbles, hm. this might take a while [14:37:39] Krenair: oh? If you are super stuck I can get it [14:38:41] It's just going to take a while to download everything [14:40:48] manybubbles, maybe I should have done this on a machine in labs? [14:41:29] Krenair: hmmm - I'm pretty sure we're averse to labs having your ssh private key to propose changes [14:41:34] Yeah, because sticking your private key to labs is a good idea [14:41:37] wait [14:41:58] I believe the issue is that too many people have root on labs [14:42:12] I don't need to put my private key on labs to upload stuff to gerrit, I don't think... [14:42:14] we're not happy that we do key forwarding on tin for deployment and that is just a temporary thing [14:42:30] Krenair: How do you plan to upload it then? [14:42:43] Key forwarding via ssh-agent is also not a good idea if a lot of people have root [14:43:08] Doesn't it allow a temporary password to upload via http? [14:43:27] don't think so [14:43:41] I have an extra core clone for backports [14:43:57] I'm away to eat for a moment... but will be back in time for my changes [14:44:05] so no canceling please :P [14:48:03] hoo|away: no canceling:) - unless your away for like an hour [14:48:43] akosiaris: yt? i'm trying to convert the kafka .deb into a multi binary package like we talked about [14:48:46] got a q... [14:50:41] Krenair: I can build it for you [14:51:36] ottomata: yes [14:52:17] (03CR) 10Helder.wiki: "Isn't "[[wikipedia:]]" supposed to work on en.wikipedia.org?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/144264 (https://bugzilla.wikimedia.org/954) (owner: 10TTO) [14:54:35] manybubbles, is https://gerrit.wikimedia.org/r/#/c/149868/ right? [14:54:47] for https://gerrit.wikimedia.org/r/#/c/149329/ [14:54:50] ok akosiaris [14:54:51] so [14:54:58] Mostly everything is working [14:54:59] except [14:55:11] i think the make install step installs everything into debian/tmp/... [14:55:16] and, for a multi binary package [14:55:35] the files should be installed into directories named by package [14:55:36] like [14:55:44] Krenair: looks fine, yeah [14:55:48] debian/kafka-common/usr/... [14:55:49] or whatever [14:55:59] manybubbles, okay, doing 1.24wmf15 submodule update as well then [14:56:35] is it possible to get the install scripts to do that? should I just DESTDIR to debian/kafka-common in rules or something? [14:56:38] Krenair: thanks [14:56:39] set* [14:56:50] ottomata: https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#multiple-binary [14:57:04] manybubbles, https://gerrit.wikimedia.org/r/149869 for https://gerrit.wikimedia.org/r/#/c/149328/ [14:57:26] so, just move the files around in the temporary trees like you already described :-) [14:57:53] using DESTDIR? [14:58:01] Krenair: got it: https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=121583&oldid=121552 [14:58:10] or using package-name.install? [14:58:16] manybubbles, turns out I don't need to do the whole "git submodule update --init --recursive" just to send those commits [14:59:02] plain install or cp calls in an new install: target in debian/rules [14:59:03] Krenair: you just need your submodule, I imagine [14:59:08] yep. [14:59:34] akosiaris: ? [14:59:38] ottomata: cause we don't see to already have one, using dh $@ --with javahelper for everything [14:59:41] manybubbles, okay. I can't merge on the wmf branches so I have to get a deployer to merge them before I can do the submodule stuff [14:59:51] so override dh_install [15:00:04] manybubbles, Reedy, awight: Sir, Please deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140728T1500), the time has come. At your service [15:00:36] oh, for each file, use install or cp for each file? in override_dh_install? [15:00:39] hm [15:00:41] Krenair: I thought I did do the merging on the submodule branches for you? but, yeah, that is a bit weird [15:01:02] manybubbles, yeah, you did. [15:01:10] ottomata: not that many files anyway.. right ? [15:01:30] the bin shell script for kafka-client and init.d script for kafka-server ? [15:01:33] anything I miss ? [15:01:42] manybubbles, But I mean I have to put the cherry-pick-to-extension-branch gerrit changes on the deployment calendar [15:01:44] awight: hey - you merged your submodule update before the swat deployment so I have to do you first [15:01:46] are you ready? [15:02:01] Krenair: ah! because you can't have built the submodule update yourself [15:02:05] manybubbles: yep, thank you [15:02:07] akosiaris: no, those work, via the .install files [15:02:10] yeah [15:02:12] kafka-server.install, etc. [15:02:18] then ? [15:02:19] yeah, the merge was during my attempted deployment last week... oops! [15:02:20] its the files that your Makefiles install [15:02:24] the compiled .jars [15:02:27] ah [15:02:27] for kafka-common [15:02:31] that go to debian/tmp/... [15:02:45] Krenair: I'll bother folks about that - it is certainly easier on me if you can +2 in the deployment branches and make the submoudle updates [15:03:03] manybubbles, well I can't be a deployer, so... [15:03:19] Krenair: I can help CR, what's the patch? [15:03:29] awight, we sorted it for this one, it's fine [15:03:32] kk [15:03:36] Krenair: I'm more arguing that merging to the deployment branch might be something we let everyone do [15:03:46] ah, yeah [15:04:05] (03PS27) 10Yuvipanda: Add extdist module + role for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) [15:06:36] awight: just checking on your change - it looks like it was merged five days ago [15:06:43] manybubbles: exactly. [15:06:47] are you sure you meant it to go in swat today? [15:06:53] manybubbles: I failed to do the submodule update... [15:06:58] because when you merged it into the deployment branch..... [15:06:59] ah [15:07:19] hehe only realized what I'd done wrong on Friday. [15:07:50] awight: so the change you actually want deployed is this: 95e0324951bb0ebaf48be7a0871e897799d58688 ? [15:07:59] looking... [15:08:13] Hey greg-g. [15:08:19] because it *looks to me* like it was all deployed [15:08:32] manybubbles: yes, that's the underlying change [15:08:32] I checked the version both before and after I did the submodule update and it all looked clean [15:08:43] manybubbles: ok I'll verify [15:08:51] cool - if not I can sync it again [15:08:58] ottomata: I don't have the answer ready. Will need to research it a bit more [15:09:03] manybubbles: it's the 1.24wmf14 branch that was botched [15:09:43] awight: that is what I was looking at. I'll just sync the files again to be sure - but it'd be helpful if you could verify that it all worked [15:10:08] ottomata: got a change I should use ? [15:10:20] !log manybubbles Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow/: SWAT update fundraising to fix botched deploy [15:10:25] Logged the message, Master [15:11:19] Krenair: I'm going to start the process on your wmf15 updates [15:11:31] manybubbles, okay [15:11:48] ok cool, akosiaris i am trying DESTDIR... [15:12:17] manybubbles: confirmed, the change has gone out and the bug is even fixed. Thank you! [15:12:24] awight: wee! [15:12:27] (03PS12) 10Physikerwelt: Mathoid configuration for beta labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 [15:13:41] (03CR) 10Manybubbles: [C: 032] Add import sources for bhwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149530 (https://bugzilla.wikimedia.org/68616) (owner: 10Hoo man) [15:13:51] hoo: I'll do the import source while I wait for jenkins [15:14:00] (03Merged) 10jenkins-bot: Add import sources for bhwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149530 (https://bugzilla.wikimedia.org/68616) (owner: 10Hoo man) [15:14:24] yeah, they're trivial enough [15:14:53] hoo: going out now [15:14:55] !log manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - add import sources to bhwiki (duration: 00m 08s) [15:15:01] Logged the message, Master [15:15:19] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [15:15:23] ^d: there? :) I created the swift account so we can move forward [15:15:28] it seems that there is a problem with http://ganglia.wmflabs.org/latest/ [15:15:56] confiremd [15:15:58] <^d> godog: Yep, I'm here. So I realized Thurs/Fri that we never deployed the swift plugin for ES. [15:16:04] * confirmed [15:16:11] <^d> We did the git-fat dance and got it live, but we still need a rolling cluster restart to pick it up. [15:16:19] <^d> Which I was waiting for manybubbles to come back before doing. [15:16:24] <^d> (And not on a friday) [15:17:00] !log manybubbles Synchronized php-1.24wmf15/extensions/Echo/: SWAT - fix incorrect variable name (duration: 00m 08s) [15:17:07] Logged the message, Master [15:17:18] (03PS1) 10Hashar: Load Mantle before MobileFrontend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) [15:17:25] ^d: you want to do the rolling restart dance? its all kinds of fun! [15:17:29] ^d: hehe makes sense [15:17:34] physikerwelt: looking [15:17:40] you could even upgrade to 1.3 [15:18:02] but we'll need to do another one after wikimania to pick up the highlighter changes I started on friday morning before we drove 16 horus [15:18:19] <^d> I think I understand how to do the rolling restart dance. [15:18:25] Krenair: incorrect variable fix deployed to wmf15 - please verify [15:18:25] <^d> Upgrading to 1.3 is scarrrryyyyyyy :p [15:18:34] ^d: its pretty much the same:) [15:18:39] I can do the 1.3 upgrade later [15:18:41] that makes more sense [15:19:02] godog: I have also a problem to list instances at https://wikitech.wikimedia.org/wiki/Special:NovaInstance So it might be related to my user-account / browser [15:19:07] I'll have time to update all the other extensions [15:19:08] <^d> (Plus I also haven't tested the plugin with 1.3 yet which shouldn't be a big deal but I need to first) [15:19:16] yeah [15:20:02] manybubbles: Busy SWAT today. :-( [15:20:07] Krenair: ping on verify? [15:20:26] <^d> godog: So, we can get the key in privatesettings and such, but I don't think we'll be ready to test today yet. [15:20:28] James_F: yeah! [15:20:29] manybubbles, looks fine to me [15:20:50] Krenair: cool - starting the merge for wmf14 then [15:21:21] (03CR) 10Florianschmidtwelzow: [C: 031] Load Mantle before MobileFrontend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:21:29] Vogone: I can do your eswiki permissions changes if you are ready to verify them [15:21:54] k [15:22:07] (03CR) 10Manybubbles: [C: 031] Load Mantle before MobileFrontend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:22:20] (03CR) 10Manybubbles: [C: 032] Add 'abusefilter-log-detail' to 'rollbacker' and 'patroller' group at eswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149533 (https://bugzilla.wikimedia.org/68319) (owner: 10Vogone) [15:22:26] (03Merged) 10jenkins-bot: Add 'abusefilter-log-detail' to 'rollbacker' and 'patroller' group at eswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149533 (https://bugzilla.wikimedia.org/68319) (owner: 10Vogone) [15:22:42] (03CR) 10Chad: [C: 031] Load Mantle before MobileFrontend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:22:43] godog: I have logged out and logged in again the problem listing the instances disappeared but the ganglia problem still remains [15:23:17] !log manybubbles Synchronized wmf-config/InitialiseSettings.php: SWAT - update some permissions on eswiki (duration: 00m 08s) [15:23:21] Vogone: deployed [15:23:23] Logged the message, Master [15:23:49] physikerwelt: yeah I'm currently looking at that (ganglia) [15:24:09] ^d: ok! let me know when it'd be good to go on your side [15:25:19] manybubbles, am ready to verify for the echo wmf14 change, by the way [15:25:53] Krenair: jenkins hated it - forcing it to retry [15:25:57] (03CR) 10Jforrester: [C: 031] Load Mantle before MobileFrontend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:25:58] :/ [15:26:04] there is a good chance it was spurious [15:26:12] its pretty rare that it does that, but it happens [15:26:19] godog: Thank you. I was testing that change https://gerrit.wikimedia.org/r/#/c/148836/ and I got the message from puppet that the ganglia monitoring was set up correctly. I hope that is unrelated to the global ganglia problem [15:26:38] Krenair: you can watch it some: https://integration.wikimedia.org/zuul/ [15:27:01] Oh it's that damn qunit issue [15:27:01] looks like the qunit tests passed this time - must have been something silly [15:27:21] Yeah this is a PHP fix and shouldn't be messing with qunit [15:27:36] I've seen it do that before [15:27:45] (03CR) 10Greg Grossmeier: [C: 031] "Please unbreak beta cluster." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:27:48] physikerwelt: ah nevermind I misread the url, don't know much about ganglia in labs though :( [15:27:49] (03CR) 10Manybubbles: "Ok - so many +1s so fast.... Should I tack this onto the end of SWAT if there is time?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:27:58] (03CR) 10Andrew Bogott: [C: 031] "Is this a service that's currently running on production but unpuppetized? Or a new service? (The bug seems to suggest the former, but t" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149486 (https://bugzilla.wikimedia.org/68609) (owner: 10Yuvipanda) [15:27:59] manybubbles: Yes. [15:28:19] manybubbles: yes please [15:28:25] James_F: hmmm - in use but not breaking without that change - I guess not in use for mobile frontend? [15:28:29] whatever - looks pretty safe [15:28:30] James_F: greg-g comment on gerirt! [15:28:36] I did [15:28:42] hashar: :-P [15:28:45] andrewbogott: online? [15:28:53] greg-g: I am, what's up? [15:28:57] manybubbles: It's in use and broken in MF too. [15:29:01] manybubbles: works for me :) [15:29:03] andrewbogott: mind merging https://gerrit.wikimedia.org/r/149873 [15:29:18] plenty o' +1s ;) [15:29:20] manybubbles: Also has broken Beta Labs scap, DB update etc. [15:29:21] (03CR) 10Hashar: "I guess one can +2 it, verify beta is fine and then swat deploy it :-D" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:29:37] (03CR) 10Andrew Bogott: [C: 032] Load Mantle before MobileFrontend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:29:45] (03CR) 10Florianschmidtwelzow: "> Question: is mantle in use in production yet?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149873 (https://bugzilla.wikimedia.org/68704) (owner: 10Hashar) [15:29:45] now andrew has to deploy it hehe [15:29:52] Ha. [15:30:05] ah [15:30:11] greg-g: I'll deploy it.... [15:30:19] thanks manybubbles [15:30:24] manybubbles: thanks [15:30:33] operations is a bit misleading since that is both platform+ops duties and the ops team [15:30:41] (03PS2) 10BBlack: Add explicit mmap addrs for varnish persistent storage [operations/puppet] - 10https://gerrit.wikimedia.org/r/149068 [15:30:42] :) [15:31:42] Krenair: here you go [15:31:45] !log manybubbles Synchronized php-1.24wmf14/extensions/Echo/: SWAT fix bad variable name in echo (duration: 00m 08s) [15:31:51]