[06:34:14] (PS9) Stefan.petrea: Updating Kraken to cope with libdclass 2.2.2 [analytics/kraken] - https://gerrit.wikimedia.org/r/75349 [06:35:28] (CR) Stefan.petrea: "(26 comments)" [analytics/kraken] - https://gerrit.wikimedia.org/r/75349 (owner: Stefan.petrea) [06:39:42] (PS1) Rfaulk: mod. typo. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79032 [06:39:43] (PS1) Rfaulk: add. get_all_items to broker base class. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79033 [06:39:44] (PS1) Rfaulk: add. method get_all_items to broker. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79034 [06:39:45] (PS1) Rfaulk: add. method is_item to detect key in target. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79035 [06:39:46] (PS1) Rfaulk: mod. handle processing/queued jobs. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79036 [06:39:47] (PS1) Rfaulk: fix. bad ref to request items - should be response items. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79037 [06:39:48] (PS1) Rfaulk: mod. handle url parsing to generate unique hashes. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79038 [06:39:49] (PS1) Rfaulk: mod. parsing methods to handle stripped url. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79039 [06:40:21] (CR) Rfaulk: [C: 2 V: 2] mod. typo. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79032 (owner: Rfaulk) [06:40:33] (CR) Rfaulk: [C: 2 V: 2] add. get_all_items to broker base class. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79033 (owner: Rfaulk) [06:40:44] (CR) Rfaulk: [C: 2 V: 2] add. method get_all_items to broker. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79034 (owner: Rfaulk) [06:41:01] (CR) Rfaulk: [C: 2 V: 2] add. method is_item to detect key in target. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79035 (owner: Rfaulk) [06:41:15] (CR) Rfaulk: [C: 2 V: 2] mod. handle processing/queued jobs. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79036 (owner: Rfaulk) [06:41:25] (CR) Rfaulk: [C: 2 V: 2] fix. bad ref to request items - should be response items. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79037 (owner: Rfaulk) [06:41:35] (CR) Rfaulk: [C: 2 V: 2] mod. handle url parsing to generate unique hashes. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79038 (owner: Rfaulk) [06:41:55] (CR) Rfaulk: [C: 2 V: 2] mod. parsing methods to handle stripped url. [analytics/user-metrics] - https://gerrit.wikimedia.org/r/79039 (owner: Rfaulk) [06:52:13] hey average [06:54:00] hi drdee [06:54:31] in airport ? [06:54:56] nope [06:55:01] not yet at least [06:55:07] qchris are you around? [06:58:32] it's real early here, like 09:00 [06:58:44] I'm up from 04:00, ran a couple of miles [06:58:57] they stole my bike from the garden in the interior garden of the block of flats.. [06:59:47] I wanted to tie it to a tree but I saw that the owner's husband was uhm.. doing gardening and didn't want to bother him...uhm yeah [07:00:35] I still can't believe I got my bike stolen, but autumn is coming anyway so I'm prolly not gonna use it anyway [07:01:31] that sucks [07:01:37] i am in holland right now [07:01:41] 9 am as wel :) [07:02:01] once qchris shows up let's go through the patch set; you addressed all the comments thanks for doing that! [07:02:07] so hopefully we can merge it today [07:02:14] ah, cool, same timezone [07:02:22] yup [09:09:22] qchris around? [13:12:00] average: Hi! [13:13:09] average: As I've been told to use the libdclass0 debs before and there is no libdclassjni0 for 2.2.0, does that mean I should now switch to libdclass-jni (added dash, and without the trailing 0)? [13:14:53] average: Sorry, only now I realize that the versions without 0 do not provide a -dev variant :-/ [13:15:30] average: Which libdclassjni.so and which libdclass.so.0 should we use for the latest gerrit patch set? [14:13:37] yo average, around? [14:43:09] now yes [14:43:13] I just arrived [14:43:27] drdee , qchris pong [14:43:52] Hi average! [14:43:58] qchris: uninstall all your libdclasses on your system [14:44:03] qchris: and install them from apt.wikimedia.org [14:44:12] I am not on a debian like system. [14:44:31] qchris: what's your OS ? [14:44:38] Gentoo. [14:44:45] Is http://garage-coding.com/ no longer good? [14:45:03] I am doing de-ar-ing and de-tar-ing by hand for your debs. [14:45:07] qchris: it is but you can get debs there, which you can't use with Gentoo right ? [14:45:22] I can install the files I need. [14:45:27] ok, moment [14:45:35] But I am not sure which debs are needed. [14:45:38] I settled for: [14:45:43] qchris: http://apt.wikimedia.org/wikimedia/pool/main/d/dclass/ [14:46:12] oh. Nice. [14:46:13] qchris: all the latest, Faidon-approved, gerrit-reviewed debs are there [14:46:33] Great! [14:46:47] I'll use those then :-D [14:47:02] :) [15:06:31] average: Does testIpad2 still work for you? [15:16:05] qchris: testIpad2 ? [15:16:14] qchris: I'm not sure which test that is [15:16:19] From the pig subproject [15:16:24] ooooh [15:16:35] Try running 'mvn test' in the kraken repo root directory. [15:16:44] Does that go through cleanly? [15:16:59] I only checked for kraken-dclass for patchset #9 [15:17:11] I will check mvn test for the whole repo now [15:17:17] Ok. [15:20:09] yes, it fails [15:20:15] Mhmm. [15:20:23] I'm reading it now [15:20:37] Do you know how hard we depend on the OS output? [15:20:45] Is this used in some scripts etc? [15:21:06] it certainly ends up in some reports that an instance of limn has [15:21:16] Ok :-( [15:23:04] The tests pass for the 1.13 dtree file. [15:23:17] Do you know it forcing that version is an option? [15:23:39] (That version is no longer available on http://apt.wikimedia.org/wikimedia/pool/main/d/dclass/) [15:24:07] s/know it forcing/know if forcing/ [15:25:58] I guess we need input from the consumers of kraken, to see if that would work for them. dr dee will not be here today. So I guess we're stalled here for now :-(( [15:34:13] we can fallback on 1.13 dtree file [15:34:20] but.. would that really benefit us ? [15:35:11] hi everyone :) [15:35:14] back from vacation [15:35:36] hIIIIIII [15:35:43] are you more or less beardy? [15:35:49] more, of course [15:35:49] after being in the alaskan wilderness? [15:35:52] good! [15:36:03] TSA doesn't let me travel with my blade [15:36:22] you are a straight razor-er [15:36:23] ? [15:36:46] yea [15:37:18] Hi millimetric! [15:37:26] Good to see you could escape all the bears [15:37:31] indeed! [15:37:32] milimetric: what blade ? [15:37:37] No need for the little redriding hood tricks :-) [15:37:37] milimetric: shaving one ? [15:38:05] it was a close call once, we must have been following a bear through a glacier trail 'cause I found a bunch of fresh poop [15:38:17] :-)) [15:38:22] maybe it was a sasquatch [15:38:35] milimetric: this is my blade: http://www.flickr.com/photos/ottomatona/2400618806/ [15:39:17] it is very dull, but I shave with it anyway, YARRR [15:39:54] yeah average, this one: http://www.theartofshaving.com/Black-Straight-Razor/BLACKSTRAIGHTRAZOR,default,pd.html [15:40:24] wow, that's a classic [15:40:28] ouch otto [15:40:40] just... ouch man [15:40:50] i would probably not be alive if I shaved with that, lol [15:41:05] milimetric: do you ever get hot shaves? like at a barber shop? [15:41:44] yeah, average, I haven't bought a razor blade since 2007 and I go through 1/10th as much shaving soap, so it's actually been a lot cheaper. [15:41:58] I got one hot shave and it was aamazing [15:42:09] but you need a mirror [15:42:11] it's the reason I went straight razor [15:42:17] yeah, hot shave is amazing [15:42:30] i've been taught to use a straight razor too [15:42:35] buuuuut, i didn't actually like it that much [15:42:36] too scary [15:42:43] i cut myself for like two weeks [15:42:44] really bad [15:42:51] sink would be red [15:43:04] but eventually I learned [15:43:22] gotta keep it sharp, stretch the skin [16:04:48] hey qchris , so the failing testIpad2 is coming from the updated dtree file ? [16:04:56] average: Yes. [16:05:11] average: The test works with 1.13 for me, but fails with 1.19 [16:06:05] average: (However, for 1.13 the nem dclass tests fail, as the returned model is A6380 instead of A6366, but that's due to the mismatch) [16:06:12] s/nem/new/ [16:07:07] qchris: so we can just update testIpad2 to cope with the new dtree file ? [16:07:14] would that be a solution ? [16:07:25] average: I do not think so. [16:07:31] why ? [16:07:48] average: If the OS result is used in reports/graphs, that would change the reports/graphs. [16:07:54] average: But only for the new days. [16:08:05] average: So we'd get strange jumps in graphs. [16:08:23] average: We'll have to check with others /if/ they rely on this field already. [16:08:52] actually this is a problem for geoip dbs as well [16:08:56] they modify across time [16:09:26] Yes, but there the change is wanted. [16:09:27] However, [16:09:38] If i buy a phone, the same phone is reported as [16:09:48] iOS in the old dclass [16:09:57] and as iPhoneOS in the new dclass [16:10:03] although the phone has not changed :-/ [16:10:57] Without auditing the dtree file we do not know what else has changed and might cause problems. [16:11:23] but auditing the dtree file would mean re-doing the work that OpenDDR does [16:11:28] which is not trivial [16:11:58] Yes. This is why we should not deceide by ourselves, but ask the others how they use the data [16:11:59] it's like.. http://i.imgur.com/ejBP4Sx.jpg [16:12:05] and check what they rely on. [16:12:22] ok [16:12:46] qchris: how about re-running all the stuff that relies on device detection ? [16:12:55] how much stuff is that? [16:13:03] How many consumers do we have? [16:13:16] milimetric: can I ask a q ? how many reports do we have in the report that depend on device detection ? [16:18:22] (CR) QChris: "(1 comment)" [analytics/kraken] - https://gerrit.wikimedia.org/r/75349 (owner: Stefan.petrea) [16:25:02] (CR) QChris: "(10 comments)" [analytics/kraken] - https://gerrit.wikimedia.org/r/75349 (owner: Stefan.petrea) [18:09:23] qchris, repos moved [18:09:24] and created [18:13:03] average: sorry I missed your question earlier. I think we covered it during scrum but I only know of the one that I made - the mobile platform report [18:28:44] ottomata: Cool. Thanks. [18:33:54] qchris: I GOT IT! [18:34:02] Yipppiiiiie!!!!! [18:34:13] How did you do it? [18:34:17] just had to read more hadoop docs [18:34:17] http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample [18:34:21] final FSDataOutputStream writer = path.getFileSystem( [18:34:21] context.getConfiguration() [18:34:21] ).create(path); [18:34:36] and then i used the anon class just like they did [18:34:36] Cool :-D [18:34:43] and called writer.write with my data [18:35:22] Awesome. So the stuff gets into HDFS ... automatically sorted into timestamp buckets? [18:37:16] well, i haven't done that bit yet, but yes it hsould work [18:37:24] right now i just got it to consume from kafka and write [18:37:28] using current time [18:37:39] but yeah, all I should have to do is set the timestamp field on each record by reading the json [18:37:53] i'm looking a little more into recommended ways to store json in hdfs files [18:37:58] i think we'll just use a line break [18:38:02] each line is an object [18:40:23] Sounds good for a start. [18:40:51] But we ran into problems with tabs in User agent strings for TSVs [18:41:12] So I am sure we'll get Json with newlines in the fields sooner or later. [18:42:18] On the other hand ... we canno use line based tools anyways for json. So sure: Linebreaks sound good. [18:43:32] gah! what do you want to store in Avro or something!??!?! [18:56:19] hmm [18:56:23] qchris: https://github.com/alexholmes/json-mapreduce [19:02:53] ottomata: Mhmmm. Looks interesting. [19:19:30] average: Are you around? [19:43:43] heya qchris [19:43:43] ok real quick q I think you can help me with [19:43:43] so, here's my timestamp json field [19:43:43] "timestamp":"[14/Aug/2013:19:14:30 +0000]" [19:43:44] I do [19:43:45] i read in a gson JsonObject [19:43:45] and then do [19:43:57] String timestampString = record.get("timestamp").timestamp = new SimpleDateFormat("[dd/MMM/yyyy:HH:mm:ss Z]").parse(timestampString).getTime(); [19:44:39] two assignments in a single line? [19:45:01] Ah. the period should be a semicolon? [19:45:10] but, then I get [19:45:11] java.text.ParseException: Unparseable date: ""[14/Aug/2013:19:23:31 +0000]"" [19:45:33] looks like a paste error [19:45:34] here [19:48:01] Let me try that. [19:48:16] (two double quotes in the error message?) [19:49:13] agh, internet! ok back i think [19:49:24] :-) [19:49:32] Are there really two double quotes in the error message? [19:49:44] https://gist.github.com/ottomata/6234875 [19:49:45] yes [19:50:00] Caused by: java.lang.RuntimeException: java.text.ParseException: Unparseable date: ""[14/Aug/2013:19:23:31 +0000]"" [19:51:16] Your string seems to be: [19:51:17] ""[14/Aug/2013:19:23:31 +0000]" [19:51:20] and not: [19:51:23] [14/Aug/2013:19:23:31 +0000] [19:51:29] Wait copy paste error. [19:51:40] Your timstampString seems to be [19:51:45] "[14/Aug/2013:19:23:31 +0000]" [19:51:48] and not: [19:51:50] [14/Aug/2013:19:23:31 +0000] [19:52:20] You can either use: [19:52:23] new SimpleDateFormat("\"[dd/MMM/yyyy:HH:mm:ss Z]\"").parse(timestampString).getTime() [19:52:44] or get rid of them in another way [19:52:54] Did you check thet the Json is valid? [19:53:29] (Not "syntiactically valid", but valid in the sense that the timestamp does not contain additional double quotes) [19:53:33] ottomata ^ [19:56:05] i mean, [19:56:10] i don't think it does [19:56:13] i put the json record here [19:56:13] https://gist.github.com/ottomata/6234875 [19:56:55] I can recreate locally too [19:57:00] just using that record as a string [19:57:03] i added my test class there too [19:57:06] same error [19:57:16] i just looked into Jackson a bit [19:57:22] but it seems way more complicated [19:57:54] To reproduce the error you get, I have to use the double quoted string. Otherwise it works. [19:58:58] Can you try logging the timestampString and see what it is? [19:59:43] yeah that's what I was just doing [19:59:53] JsonObject record = new JsonParser().parse(data).getAsJsonObject(); [19:59:53] System.out.println("ts: " + record.get("timestamp").toString()); [19:59:55] right? [20:00:00] prints [20:00:00] ts: "[14/Aug/2013:19:14:30 +0000]" [20:00:02] with quotes [20:00:33] So it has extra quotes. [20:01:06] ha, not the only one with this problem :0 [20:01:07] http://stackoverflow.com/questions/16639482/json-string-from-gson-remove-double-quotes [20:01:38] ohhhh [20:01:43] i should use getAsString() instead of toString() [20:01:57] that should do it [20:01:58] ok thank you! [20:02:00] :) [20:02:26] Cool :-D [20:22:30] average: Are you around? [20:58:02] (CR) QChris: "(2 comments)" [analytics/kraken] - https://gerrit.wikimedia.org/r/75349 (owner: Stefan.petrea)