[00:37:15] !log Gerrit has once again closed the Zuul socket, ssh event stream down [00:41:29] !log wake up morebots [00:41:37] now what [00:41:37] Logged the message, Master [00:41:42] !log Gerrit has once again closed the Zuul socket, ssh event stream down [00:41:49] Logged the message, Master [02:01:29] !log LocalisationUpdate completed (1.22wmf5) at Mon Jun 10 02:01:29 UTC 2013 [02:01:44] Logged the message, Master [02:02:14] !log LocalisationUpdate completed (1.22wmf6) at Mon Jun 10 02:02:14 UTC 2013 [02:02:22] Logged the message, Master [02:06:50] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 10 02:06:49 UTC 2013 [02:07:02] Logged the message, Master [02:25:14] !updated Parsoid to 956117df0e [02:35:09] gwicke: You want "!log" first. :-) [02:35:33] oops, good point [02:35:50] those bots should simply be come a bit more intelligent ;) [02:35:58] !log updated Parsoid to 956117df0e [02:36:06] Logged the message, Master [03:55:24] New patchset: Krinkle; "wgRC2UDPPrefix: Use hostname-".org" instead of lang.site" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47307 [07:33:35] !log updated Board Election translations on cluster [07:33:44] Logged the message, Master [08:09:52] !log nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master' [08:10:02] Logged the message, Master [08:25:59] !log nikerabbit synchronized php-1.22wmf5/extensions/UniversalLanguageSelector/ 'ULS to master' [08:26:05] Logged the message, Master [08:44:45] Nikerabbit: so is gerrit stuck again ? [08:44:59] hashar: yep [08:45:29] New review: Hashar; "ping" [operations/puppet/zookeeper] (master) - https://gerrit.wikimedia.org/r/66906 [08:45:34] indeed :/ [08:45:51] apergos: good morning! would you mind restarting the gerrit service on manganese pleas ? [08:46:01] mrning [08:46:02] ok [08:46:10] apergos: it got stuck over and over for the last 3 or 4 days and restating it is the only way to restore the service unfortunately [08:46:17] I thought demon got it [08:46:29] it crashes again after a few hours :( [08:46:51] ah so it's dead again after last night, wow [08:47:25] we will talk about it with Chad when he connects 'in roughly 3 hours') [08:48:50] !Log restarted gerrit again, poor thing [08:49:00] Logged the message, Master [08:49:04] good bot! [08:49:07] New review: Hashar; "ping" [operations/puppet/zookeeper] (master) - https://gerrit.wikimedia.org/r/66906 [08:49:11] apergos: thanks :) [08:49:15] yw [08:49:17] Nikerabbit: solved for now [08:49:34] hashar: magnificenta [09:32:36] New review: Faidon; "I'm not sure I understand why a low hit rate results in PHP fatal errors. Do you know why this happens?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67551 [11:32:14] New review: Mark Bergsma; "Perhaps it makes sense to keep APC enabled, but clear the cache on every build (if that's possible)?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67551 [12:33:43] Gerrit is down. [12:33:53] "The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later." [12:34:28] <^demon> I had to restart it so zuul would pick up again [12:34:44] back up [12:34:56] ^demon: *nod* :( Twice already today. [12:35:01] <^demon> I know. [12:42:35] New review: Hashar; "ping" [operations/puppet/zookeeper] (master) - https://gerrit.wikimedia.org/r/66906 [12:47:07] New review: Siebrand; "manybubbles: Where are the settings you speak of found?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [12:49:37] New review: Faidon; "(my previous review on PS1 still stands in its entirety)" [operations/debs/kafka] (master) C: -1; - https://gerrit.wikimedia.org/r/67442 [12:52:08] New review: Manybubbles; "I'm not really sure. Sometimes Java properties are burried in the chain of shell scripts used to ex..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [12:53:31] New review: Manybubbles; "To be clear not setting the -Xmx parameter won't be a problem now, it'll be a problem if we move to ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [12:54:20] New review: Faidon; "Hm, debian/patches seem to be iterations over the same (Make)files, it probably would be more readab..." [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/67442 [12:58:41] New patchset: Petrb; "improved sql script" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67826 [13:02:23] New review: Siebrand; "If it's not blocking now, please don't block this patch set..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [13:04:47] New review: Manybubbles; "Sorry, I'm not used to how we work." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/67252 [13:05:13] * siebrand grins at manybubbles [13:05:38] New patchset: Faidon; "Varnish radosgw: only shard certain containers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67827 [13:06:23] siebrand: so if this were my last job I wouldn't have let that go to production because it'll bite us eventually and we'd never have fixed it. Is the right thing to do here to file another bug about _maybe_ not setting max memory? [13:09:31] hashar, does integration-zuul-layoutdiff return non-zero if there's any diff at all? [13:10:47] New review: Faidon; "No, I'd rather not risk it. Please investigate what Manybubbles suggests and come back with an adjus..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/67252 [13:11:22] manybubbles: I don't know what the right thing is, but Id really like to know. I need a problem fixed that we have, and I'd like to get that change in. [13:11:35] siebrand: then find out what the right thing is :) [13:11:49] manybubbles: If there is more, I need to be able to ask someone what the right change is, so who can I ask? [13:12:00] siebrand: fair enough. can we figure out what the command running java is? [13:12:02] andrewbogott: probably :-) [13:12:12] manybubbles: commenting on patchsets explaining why something is a bad idea, is perfectly fine and *is* how we work [13:12:18] indeed [13:12:20] hashar, ok then… how does https://gerrit.wikimedia.org/r/#/c/67462/ look? [13:12:30] manybubbles: I do not know. I have no access to the machines, I do no know ops procedures. [13:12:32] andrewbogott: the build status is irrelevant, that is merely a convenience to easily diff the Zuul interpretation of the config [13:12:35] manybubbles, and people not liking it and complaining, also :) [13:12:51] hashar: It took me surprisingly long to figure that out :) [13:13:24] andrewbogott: sorry :( [13:13:29] siebrand: so we need that. flying blind is silly. It could be just fine but we can't find where the config is set. that has happened to me a few times. [13:13:47] hashar, not your fault [13:13:48] siebrand: and if it is set then we shouldn't set it twice because then we'd just be confusing [13:13:56] manybubbles: okay, I'll add ops as a reviewer and ask who knows. [13:14:20] siebrand: sounds good [13:14:46] I already became a "reviewer" when I added my comment [13:15:05] hashar, anyway… does my regexp look OK? That test should pass now, and I want to make pep8 mandatory before we backslide :) [13:15:29] New review: Siebrand; "Faidon: I really have no idea, because you operate the machines, and they don't do what they need to..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [13:15:32] andrewbogott: commented at https://gerrit.wikimedia.org/r/#/c/67462/13/layout.yaml,unified :D [13:15:38] thanks [13:16:06] andrewbogott: Imeant, the patchset 2 should be fine https://gerrit.wikimedia.org/r/#/c/67462/2/layout.yaml,unified [13:16:15] ok -- /me simplifies [13:16:25] andrewbogott: which is the run at https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/128/console [13:16:25] uhm [13:16:30] so ttmserver runs on... vanadium!? [13:16:46] oh my... [13:16:54] yes, so here's how it works [13:17:08] some random group asks for a server to do whatever on, doesn't need any ops support whatsoever [13:17:14] so after some hesitation we give that [13:17:22] and then all kinds of services start appearing on that machine, since, you know, there are no barriers [13:17:27] and then suddenly we need to support that [13:18:01] mark: like when you added etherpad for testing? :D [13:18:14] different, but sure [13:18:32] of course noone in ops knows much about that solr install [13:18:37] much? [13:18:38] I'm sure LangEng and translators would have liked to be considered worth a dedicated machine [13:18:56] since we had nothing to do with it, and we're just planning to assign someone to do a support solr (re)install [13:18:57] and also about someone else taking care of setting solr up [13:19:18] Nemo_bis: afaik faidon helped with the packages. [13:19:24] what packages? [13:19:32] Nemo_bis: for solr/jetty [13:19:36] don't think so [13:19:50] okay then. Well, someone did... [13:20:09] so anyway [13:20:58] there's a jetty running on vanadium [13:22:07] jsvc.exec -user jetty -cp /usr/share/java/commons-daemon.jar:/usr/share/jetty/start.jar:/usr/share/jetty/start-daemon.jar:/usr/lib/jvm/default-java/lib/tools.jar -outfile /var/log/jetty/out.log -errfile /var/log/jetty/out.log -pidfile /var/run/jetty.pid -XX:+UseConcMarkSweepGC -Djava.io.tmpdir=/var/cache/jetty/data -Djava.library.path=/usr/lib -DSTART=/etc/jetty/start.config -Djetty.home=/usr/share/jetty -Djetty.logs=/var/log/jetty -Djetty. [13:22:21] manybubbles: that's the command-line, although I doubt it helps [13:22:36] PERFECT! [13:23:19] so we _don't_ set the -Xmx or -Xms parameters meaning we default the to something the JVM figures out based on ram size. [13:23:35] probably? [13:23:38] except that there are other services on the machine [13:24:52] paravoid: actually you did something to provide solr-3.6.0 package for us ;) [13:25:02] maybe? [13:25:15] backporting a package is very different than actually knowing what it's used for :) [13:25:47] be careful when you help someone faidon, someone might even mistake that for complete support and approval of everything they're doing ;) [13:26:22] mark: I don't think you're being helpful, mark. Cynicism won't solve a problem, here. [13:26:26] hashar, ok, restored version 2. [13:26:41] indeed it won't [13:26:44] it makes me feel better though [13:26:47] anyways [13:27:01] as for ULS... can we have some engineer working on ULS in next monday's engineering/ops coordination meeting? [13:27:17] paravoid: would you mind telling me how much memory that process is using just now? that'll give me a sane default to recommend for the parameters [13:27:27] mark: Would you mind sending a mail to localisation-team@wikimedia.org with what you need? [13:27:50] mark: We have sprint planning tomorrow, and so we can reply by Wed morning. [13:28:05] what I need for the meeting you mean? [13:28:25] mark: You're asking for a resource for something. I'm asking you do drop us a mail with that request, yes. [13:28:30] manybubbles: virt 3.3G, rss ~1GB [13:28:45] i'm not asking for a resource [13:28:51] paravoid: perfect - it is likely using about a 1G heap [13:28:52] siebrand: I think the point is that you are asking for a resource... :-) [13:29:06] we have a biweekly meeting to discuss things that require coordination between engineering groups and ops [13:29:35] <^demon> mark: When's the next one for that? [13:29:39] monday [13:29:40] next monday [13:29:42] paravoid: All communication I've been having up to now is not about ULS. mark brings up ULS, and I asked him to send a mail about that. [13:30:16] you will have a mail about that, it'll be my reply to erik [13:30:21] i'll make sure to cc language-team [13:30:33] mark: Ah, that… Great. [13:30:44] manybubbles: feel free to be bold and submit patchsets, you're probably more qualified than most of us and people who've previously done solr work [13:30:48] New patchset: Hashar; "jenkins validation of pep8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67830 [13:31:05] * Nikerabbit rolls eyes [13:31:33] New review: Manybubbles; "paravoid was kind enough to post the command line that all the shell scripts collapse into in IRC:" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/67252 [13:31:56] I'll submit the patch [13:32:05] perfect [13:32:08] we have another solr setup [13:32:28] <^demon> Huh? [13:32:54] the geodata one [13:33:05] that's 9.5G virt, 1.8G rss [13:33:11] the jetty settings are likely the same [13:33:33] that's from solr1001, let me do a quick check on the other boxes too [13:33:39] andrewbogott_afk: deployed :D [13:33:54] 10107768 1926384 [13:34:08] 9843704 1841500 [13:34:34] Change abandoned: Hashar; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67830 [13:34:36] manybubbles: so we either need a superset of defaults, or the puppet class should be parameterized to allow for tuning from the role class [13:35:01] or, I don't know, maybe we don't need multiple solr installations all over the place on random boxes *g* [13:35:25] <^demon> We need clouds, obviously ;-) [13:35:40] obviously [13:35:46] we have Labs [13:35:51] in near term we need something that works [13:36:05] be careful what you wish for: http://www.theregister.co.uk/2013/06/08/facebook_cloud_versus_cloud/ [13:36:06] paravoid: 1G is a pretty sane default anyway. [13:36:30] "I got a call, 'Jay, there's a cloud in the data center'," Parikh says. "'What do you mean, outside?'. 'No, inside'." [13:36:33] There was panic. [13:36:36] "It was raining in the datacenter," he explains. [13:37:48] paravoid: to be clear, you don't want me to hard set the default to 1G, you'd prefer I let that be configurable? [13:37:49] manybubbles: so this extra gb rss that those solr instances use is just garbage that needs to be collected? [13:38:19] I'm just saying that we have another set of solr servers that seem to exceed 1G rss [13:39:11] ah ha [13:39:20] let me read go make sure I'm not confused [13:39:22] I pasted numbers above [13:40:17] there's two role classes related to solr, role::solr::geodata and role::solr::ttm [13:40:30] role::solr::ttm is vanadium [13:40:51] the geodata ones are -confusingly enough- boxes named solr1001.eqiad.wmnet etc. [13:41:00] MaxSem: ping. [13:41:07] pong [13:41:24] paravoid: by rss you mean the resident memory, right? [13:41:27] yes [13:42:20] New review: Hashar; "Ohh. Maybe we should just enable instant commons everywhere." [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/67407 [13:42:31] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67407 [13:42:53] New patchset: Siebrand; "Increase ramBufferSizeMB from 32 to 100 and set Xmx/Xms" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [13:43:21] MaxSem: There was talk about the geo search. Thought I'd poke you.... [13:45:01] New review: MaxSem; "Can we make the total memory settings customizable? The solr* boxes have 64 gigs RAM total, it is co..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/67252 [13:45:30] they have 64G of ram of which they use 2GB [13:45:34] apergos: looks like you fixed up wikilove / instant commons :) [13:45:35] awesome capacity planning [13:45:40] paravoid: so the resident memory is how much the JVM has allocated for itself. My guess is the virtual memory has to do with reading and writing so many files. The 1926384 values mean that machine has _about_ 2G of heap usage. Well, more like 1.8G because there is other junk going on in there. [13:45:40] aww [13:46:00] paravoid: they actually do use quite a bit of page cache [13:46:02] not yet, instant commons is working but the fix for wikilove is awaiting someone to +2 it [13:46:23] also I didn't redo the media infrastructure yet, I am still in the documenting phase for that [13:46:36] I'll send you some pics soon [13:46:43] so would it be possible to get dedicated machine for ttmserver solr, perhaps access to it? [13:47:01] would it be possible to merge the two solr installations? [13:47:08] hashar: can't see the commit because gerrit's down, but fyi :p https://www.mediawiki.org/wiki/Talk:InstantCommons#Enable_by_default [13:47:37] i.e. merge ttmserver into solr100Ns [13:47:41] in any case, if all the machines running solr have like 64Gb of ram the JVM is defaulting the maximum number of ram to 16G, it just hasn't need it. [13:47:50] fyi, Mem: 64354 17564 46790 0 1593 8565 [13:47:55] as far as I know it is not straightforward with solr 3.6, but someone correct me [13:48:02] (overprovisioned) [13:48:04] Nemo_bis: no way we are going to make instant commons enabled in core by default :D [13:48:30] hashar: because of what Chris said or something else? but yes, something smarter would be needed [13:48:34] I've seen Solr use up to ~8G in other places, but I have no experience with instances using more. [13:48:40] paravoid, AFAIK these boxes were intentionally purchased in configuration matching the lucene boxes [13:49:02] I'm guessing from the name that noone expected them to be doing just geodata [13:49:08] wow, it isn't even getting close to using that much page cache. [13:49:11] <^demon> Nemo_bis: I don't see the reason to enable it by default either. It's already configurable in the installer. Since it makes external requests, an admin should have to explicitly turn it on. [13:49:14] who knows, maybe this will soon be the case [13:49:15] the difficulty of running multiple cores is one of the reasons ^demon has been proposing the cloud stuff I think [13:49:44] in any case, setting the max memory to something massive won't hurt us then. 4G would be conservative. [13:50:06] <^demon> Nikerabbit: Well, that's part of the reason, but the main reason is the ability to scale out without thinking about it & not having a SPOF in a master -> slave setup. [13:50:13] <^demon> Easier core management is a nice ++ [13:50:15] Nemo_bis: replied there (I will most probably not follow up though) [13:50:17] multiple cores isn't too bad but you have to think about it and write your queries using it. solr cloud stuff all has to by definition though. [13:50:23] hashar: aw, thanks :) [13:50:32] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67827 [13:50:34] ^demon, not nice but a friggin huge ++++++++++++++++++++++++++ [13:50:36] ^demon: ok, I'll check better how the installer looks like [13:50:47] solr cloud also lets us shard so we don't need 64Gb of ram everywhere [13:52:00] proposal: give solr 8 gigs of ram on the machines we have. That is twice what we've seen it use and it won't hurt with speed. [13:52:10] New patchset: Faidon; "Swift: get rid of test setup configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67831 [13:52:16] sorry, 4 gigs of ram is twice what we've seen it use. [13:52:21] manybubbles, and like 1 gig on vanadium [13:52:44] ^demon, how far is solr 4? [13:52:46] MaxSem: is vanadium still a 64 gig ram box? [13:52:57] vanadium has 8GB and runs a ton of other things [13:52:58] MaxSem: not far, really. [13:53:05] no, 8 G and shared with EvenLogging collector [13:53:18] New review: Mark Bergsma; "+2+2+2+2+2+2+2..." [operations/puppet] (production); V: 2 - https://gerrit.wikimedia.org/r/67831 [13:53:22] paravoid: k. Then it should have 1G if it can afford it. [13:53:34] manybubbles: btw, ganglia.wikimedia.org can help you answer questions like that even without having access [13:53:45] heh - I forgot [13:53:58] wasn't sure if you knew already :) [13:54:26] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67831 [13:55:59] making puppet changes now [13:59:36] <^demon> MaxSem: Like manybubbles said, not that far off realistically. In theory, we could possibly start moving you guys over to it now(ish) if we redid the existing Solr boxes with Solr4. [13:59:59] <^demon> Something has to be the guinea pig :) [14:00:12] mmm, should I start working on Solr 4 schema? [14:00:39] ^demon: so long as they understand multicore then solr cloud would work. [14:00:51] <^demon> yup yup [14:01:38] ok. is there a sane package already? [14:02:09] <^demon> Nope, ubuntu only has 3.x :( [14:02:41] then I'll wait [14:04:16] ttmserver works fine with solr4 [14:04:44] <^demon> Yeah, this isn't like a "let's do it next week" thing...but definitely something to keep in mind. [14:05:57] <^demon> I hate freenode. [14:06:28] New patchset: Ottomata; "Installing OpenJDK Java 7 instead of Sun/Oracle Java 6 on newly reinstalled analytics nodes." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67832 [14:07:17] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67832 [14:07:19] ottomata: \o/ [14:07:41] pretty [14:07:46] :) [14:09:24] User<|title == otto|> { groups +> [ "stats" ] } [14:09:33] lovely syntax [14:09:40] hah, yeah [14:09:54] was the best way I could think to do that (that was a while ago, this commit only changed tabs to spaces there) [14:10:47] New patchset: Ottomata; "Prepping analytics1020 for reinstall." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67834 [14:10:56] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67834 [14:13:17] New patchset: Faidon; "Swift rewrite.py: get rid of shard_containers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67835 [14:13:42] siebrand: sorry this is taking so long, because this has to be configurable I having to spin up a new local puppetmaster machine so I can test it [14:14:06] manybubbles: No problem. In all honesty, this is what I consider quick. [14:14:31] hashar: pep8 failed? [14:14:46] paravoid: in ops/puppet ? [14:15:10] paravoid: andrew boggott has worked on linting all the .py files so we are now enforcing pep8 *grin* [14:15:15] I have enabled it a couple hours ago [14:15:38] paravoid: dashboard : https://integration.wikimedia.org/ci/job/operations-puppet-pep8/3640/violations/? [14:15:45] New patchset: Faidon; "Swift rewrite.py: get rid of shard_containers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67835 [14:15:48] paravoid: all the issues are reported on https://integration.wikimedia.org/ci/job/operations-puppet-pep8/3640/violations/file/files/swift/SwiftMedia/wmf/rewrite.py/? [14:15:56] manybubbles: there used to be a labs instance for ttmserver but it must be outdated by now if it still exists [14:16:24] not bad [14:16:26] surprisingly the pep8 dashboards works fine [14:16:41] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67835 [14:20:34] <^demon> Nikerabbit: Is there any chance you could take a look at https://gerrit.wikimedia.org/r/#/c/67531/? I tested this locally and it seems to work. [14:22:17] <^demon> paravoid: https://gerrit.wikimedia.org/r/#/c/67642/ is a one-line fix for gitblit. [14:23:02] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67642 [14:23:33] The page you requested was not found, or you do not have permission to view this page. [14:23:43] ^demon: are you doing antimony or should I? [14:23:50] <^demon> I'm already logged in, can do it. [14:23:59] thanks [14:24:06] <^demon> Nikerabbit: Did the ? maybe get appended by your client? [14:24:09] yes [14:24:32] ^demon: looks good, but I don't have an easy way to test, should I just +2? [14:26:08] <^demon> Nikerabbit: I installed LU locally and ran its maintenance script. [14:26:16] <^demon> Seemed to work ok. [14:26:29] off for shoppin [14:26:30] g [14:45:43] paravoid, i think I can't do openjdk right now [14:45:46] http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Release-Notes/cdh4rn_topic_2_2.html [14:45:58] • MRv2 (YARN) is not supported on JDK 7 at present, because of https://issues.apache.org/jira/browse/MAPREDUCE-2264. This problem is expected to be fixed in an upcoming release. [14:46:03] also from Snaps: [14:46:49] i had thought I'd seen that openjdk 7 is supported, but its not really [14:46:49] 10:46 [14:46:50] ottomata [14:46:51] people try it, but then report problems, and it is not recommended [14:46:53] 10:46 [14:46:54] ottomata [14:46:55] oop [14:47:59] also, we have had the exact same discussion about openJDK vs Oracle back in August and we agreed we would migrate as soon as CDH would support it [14:49:30] New patchset: Ottomata; "Reverting change to install OpenJDK 7 on analytics nodes." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67837 [14:50:00] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67837 [14:57:03] New review: Ottomata; "> CLASSPATH doesn't really belong in default, looks like init script material." [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/67442 [14:58:20] New patchset: Ottomata; "First version debian kafka package" [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/67442 [15:05:12] anyone know why nothing ever PXE boots for me? :/ [15:05:17] is it me? [15:05:27] maybe brewster doesn't like the way I smell? [15:06:02] notpeter you up for helping? :) [15:06:17] ('up' here means awake and willing) [15:10:00] sigh [15:10:12] how about openjdk 6? [15:10:42] sun java 6 is EOLed and with known security bugs [15:15:53] paravoid: [15:15:54] Note*: OpenJDK6 has some open bugs w.r.t handling of generics (https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/611284, https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/716959), so OpenJDK cannot be used to compile hadoop mapreduce code in branch-0.23 and beyond, please use other JDKs. [15:25:42] New patchset: Manybubbles; "Increase ramBufferSizeMB from 32 to 100 and make maximum heap size configurable." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [15:29:21] ottomata: grumble grumbele [15:29:36] New review: MaxSem; "role::solr::ttm shouldn't have more than 1G, looks good otherwise." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/67252 [15:32:21] New patchset: Manybubbles; "Increase ramBufferSizeMB from 32 to 100 and make maximum heap size configurable." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [15:33:36] maxsem: ^ all better? [15:34:15] +1 [15:35:05] MaxSem: now that I'm one of the authors of the patch do I remove myself from the review list of +1 it? [15:38:12] don't remove, no point in voting yourself:) [15:56:34] New review: Akosiaris; "Uploading new patchset solving most of these." [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/67442 [16:03:32] ^demon: hey, over in openstack i think we need to set up a dedicated gitweb/git server to take some load off of gerrit [16:03:48] ^demon: hashar pointed me at https://git.wikimedia.org/ [16:04:13] we have no gitweb! ;) [16:04:30] <^demon> jeblair: Yeah, there's no real secret magic to that. Using standard gerrit replication to keep repos in sync on the second box, then just running gitblit with apache acting as reverse proxy. [16:04:31] Reedy: you have a gitblit though :) [16:04:52] <^demon> Then configured gerrit.config to point to gitblit. [16:05:01] ^demon: i'm not familiar with gitblit -- how's it compare with gitweb or cgit? [16:05:13] ^demon: (it looks nicer, i can see that right away. :) [16:05:33] <^demon> I think it's got some nice features like statistics. The lucene search is kind of disappointing though. [16:05:44] <^demon> Definitely *looks nicer* than gitweb or cgit. [16:05:54] <^demon> And is generally faster than gitweb [16:05:55] and faster as well isn't it ? [16:05:57] New review: Akosiaris; "Comments inline. One extra point is the overloading of JMX_PORT variable. I would rather there was a..." [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/67442 [16:06:16] jeblair: one use case was to download a tar ball of HEAD, which AFAIK was slow in gitweb [16:06:43] <^demon> That's part of it. I didn't want people's requesting of $randomTar to affect gerrit's performance. [16:06:50] <^demon> Hence a 2nd box. [16:07:39] ^demon: do you know whether the Gerrit configuration stored in the git repo is replicated as well ? And is it available via git blit? [16:07:54] oh wow, impressive. people used to do that with github a lot and it failed all the time, so we started auto-building tarballs of each branch and publishing them to known locations [16:08:13] <^demon> hashar: Well, all of refs/* is replicated, so gerrit config like refs/meta/config is there. [16:08:18] (eg http://tarballs.openstack.org/nova/nova-master.tar.gz ) [16:08:20] <^demon> Gitblit doesn't know or care about it much though. [16:08:51] I have been too lazy to generate tarballs :-D [16:09:48] ^demon: and the (gitblit) links in gerrit -- is that just a gerrit config option to specify the url and link name? [16:10:08] <^demon> Indeed, lemme find it in our puppet repo. [16:10:52] <^demon> jeblair: https://git.wikimedia.org/blob/operations%2Fpuppet.git/0f963e11c9342d1486d0de0eec82b92394ab364a/templates%2Fgerrit%2Fgerrit.config.erb#L53 [16:10:57] manybubbles: hmm now only need to find someone who can give +2 [16:11:23] ^demon: awesome thanks! [16:11:57] Nikerabbit: yup. Is Faidon the best for that? [16:12:23] Nikerabbit: should I +1 it or just leave it? What is normal for someone who submitted a patch? [16:12:26] I'm not sure about best [16:12:29] but I can give it a try [16:12:49] manybubbles: peter y. has helped me before, but doesn't seem around [16:13:12] manybubbles: for my own patches or patches I've updated I almost never give +1 [16:13:20] paravoid: please do. I've been running it against solr3-puppet-test.pmtpa.wmflabs to check saneness if that helps. [16:13:35] Nikerabbit: cool. I'll just leave it then. [16:15:09] ugh, the solr manifests are so horrible [16:15:55] manybubbles: tell you what [16:15:56] thanks, I think they were the first puppet code I wrote [16:16:27] I'll merge it for now [16:16:52] but when you start working on search 2.0, we'll rewrite role class + modules [16:17:43] Makes sense to me. I think we'll want to do for technical reasons any way. [16:18:28] depends on autocommit, so I'll merge that too [16:18:38] paravoid, want us programmers to write less horrible manifests? explain what's wrong so we can learn from our mistakes;) [16:19:07] I do when people put me as a reviewer [16:19:25] I can't really go back to all of our merged manifests and comment on them now, can I :) [16:19:35] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67249 [16:20:01] also, while I think it's commendable for developers to prepare puppet manifests and very much welcome [16:20:27] for important changes, such as building a new big piece of infrastructure I think ops people should be heavily involved [16:21:01] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67252 [16:22:24] I didn't get much help for writing those manifests ~ year ago when I was doing this... [16:22:33] yeah, but we don't have enough ops to pay equal and timely attention to all projects, so sometimes we devs have to get shit done sooner rather than later:) [16:22:43] lobby for more ops hires then. [16:23:10] I do constantly :-D [16:23:11] alternatively, you could put "puppet skills" in your job ads, but that's kind of crazy isn't it? [16:23:33] again, there's nothing wrong with leveling up devs on puppet [16:23:46] hashar for example has great puppet skills now [16:23:47] New patchset: Demon; "Utility manifest for building hiphop" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67120 [16:23:50] thx [16:23:51] :°D [16:24:08] Nikerabbit: MaxSem for puppet, feel free to ping me over IRC. [16:24:32] Nikerabbit: MaxSem I can surely give a first level of review, granted that is not an entirely new module for some entirely new software :-D [16:24:41] soo... if you tell me what's wrong I'll definitely refactor it;) [16:24:44] hashar: aren't you already overworked? :o [16:24:45] thanks hashar [16:25:08] MaxSem: no worries, I think manybubbles & ^demon have grand plans about creating a new unified solr architecture [16:25:32] (and as I was saying the other day I think this should be a cross functional team with someone from ops too) [16:25:43] (and I'll lobby for that :-) [16:25:59] I have this pet project of puppetizing translatewiki.net, but meh this conversation makes it feel even less interesting to me [16:26:00] Peter? [16:26:05] maybe? [16:26:13] who knows [16:26:23] New patchset: Demon; "Block another misbehaving spider" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67844 [16:26:30] Nikerabbit: I am overworked. So if i don't have anytime I will say so :-D But if your puppet change is "simple" enough, it is not going to take more than a minute to review it. [16:26:38] Nikerabbit: so just bring more work hehe [16:26:40] I wouldn't turn down another pair of eyes and a healthy brain [16:27:25] meanwhile, I am off for cooking / diner / daughter etc. Be back around 9pm (GMT+2) [16:27:27] <^demon> paravoid: Can you look at https://gerrit.wikimedia.org/r/#/c/67844/ and its parent? I've got a spider that's not listening to robots.txt [16:27:41] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67533 [16:27:47] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67844 [16:27:53] I was already on it :) [16:27:57] ^demon: got a chinese spider browsing Jenkins, I have basically ip route blackholed it :) [16:28:05] uyeah Sogou :) [16:28:18] damn I need to write the same patch for jenkins [16:29:00] <^demon> !log restarting apache on gerrit box [16:29:02] New review: Faidon; "What do you need this for? This feels something like that belongs in a Debian package's Build-Depend..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67120 [16:29:08] Logged the message, Master [16:29:09] |log having dinner [16:29:43] lol [16:29:49] bon appetit hashar [16:30:08] New review: Demon; "Because I'm lazy and wanted to get a manifest for building this in labs. Ideally we'd package it, ye..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67120 [16:30:25] Change abandoned: Demon; "(no reason)" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60860 [16:31:16] manybubbles: jetty on vanadium running with -Xmx1G -Xms1G [16:31:21] New patchset: Akosiaris; "First version debian kafka package" [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/67442 [16:31:44] paravoid: sounds good to me [16:32:31] <^demon> paravoid: To package for 12.04, we'd have to backport 2 other pages (easy), plus forward port 2 others + their patches (pain enough as-is) [16:32:55] <^demon> As of 13.04, everything's in apt except 1 package which still has hacks. [16:33:06] <^demon> s/pages/packages/ [16:34:17] ^demon: context? [16:34:25] <^demon> HHVM [16:34:27] <^demon> https://github.com/facebook/hiphop-php/wiki/Building-and-installing-HHVM-on-Ubuntu-12.04 [16:34:27] ah [16:34:39] they ship some modified libraries iirc [16:34:50] that they embed in the source [16:34:57] <^demon> No, they don't ship with...you have to compile yourself + their patch. [16:35:01] right, libevent [16:35:12] yeah, we wouldn't ship the modified libevent for everything to use [16:35:15] <^demon> libevent and libcurl (libcurl is fixed upstream as of 13.04, which is nice) [16:35:28] we could set up a new section in our apt just for hhvm I guess [16:35:33] but libevent is too commonly used to risk this [16:35:59] <^demon> libcurl, glog and jmalloc3 are all in 13.04, so backporting should be pretty easy. [16:36:13] <^demon> It's just that damn libevent. [16:36:18] they embed other stuff too [16:36:21] but that's okay for now I guess [16:37:03] liblz4, libsqlite3, ... [16:42:12]