[05:32:51] Analytics-Tech-community-metrics, DevRel-September-2015: Provide open changeset snapshot data on Sep 22 and Sep 24 (for Gerrit Cleanup Day) - https://phabricator.wikimedia.org/T110947#1657384 (Qgil) Good point. Then maybe we can forget about "median time" for the last three months (since it is a too volat... [10:59:20] Analytics-Kanban, hardware-requests, operations, Patch-For-Review: Request three servers for Pageview API - https://phabricator.wikimedia.org/T111053#1657727 (akosiaris) > Alex, proceed with VLAN changes! Then we can reinstall. Done. All 3 boxes changed in VLAN and interface descriptions [13:06:34] Analytics-Kanban: Investigate sample cube pageview_count vs unsampled log pageview count - https://phabricator.wikimedia.org/T108925#1657944 (JAllemandou) [13:27:12] Ironholds, yt? [13:28:17] hey ottomata :) [13:28:41] hey joal, have you ever modified a mediawiki template? [13:28:54] btw, hi! good afternoon :] [13:29:11] ottomata I need a hand: launch a manual aggregation for both projectview and projectcounts and push the result [13:29:17] mforns: nope, sorry :( [13:29:25] joal, ok np! [13:29:32] How are you doing mforns ? [13:29:40] good, and you? [13:30:15] Good as well :) [13:31:09] btw(2), thanks for reviewing the slides! After your review they changed a lot, because of other people's comments, and also my own thoughts. So if you want to have a look, they are here: https://docs.google.com/presentation/d/1vkfDow6hrEGXI10zX0Ev5UKB8-9ejdKuc_OO3pxiRWM/edit#slide=id.ge086cb56d_2_71 [13:31:44] hiyaaa! [13:31:47] joal: eh? [13:32:06] like run the workflow? i forget, is pageview & projectview in the same workflow? [13:32:08] it is, ja? [13:34:09] hey ottomata [13:34:27] aggregator, not hadoop :) [13:34:38] The reason why I ask because of the git permissions [13:35:23] I can actually run the process with the stats user, but can't push because of ssh key [13:36:09] ah! [13:36:11] yes, ok [13:36:12] sure [13:36:41] the --force option is needed, to ensure recomputation for missing data [13:36:45] But that is all [13:36:49] ottomata: --^ [13:38:30] Analytics-EventLogging, Beta-Cluster, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Fundraising Sprint Tom Waits: Beta Cluster EventLogging data is disappearing? - https://phabricator.wikimedia.org/T112926#1657984 (Ottomata) Just deleted deployment-eventlogging02 [13:39:36] addshore: ready for merge? :) [13:39:44] yup! :) [13:46:30] Analytics-EventLogging, Beta-Cluster, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Fundraising Sprint Tom Waits: Beta Cluster EventLogging data is disappearing? - https://phabricator.wikimedia.org/T112926#1657993 (AndyRussG) Thanks so much! Closing... For MySQL querying of the bann... [13:46:42] Analytics-EventLogging, Beta-Cluster, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Fundraising Sprint Tom Waits: Beta Cluster EventLogging data is disappearing? - https://phabricator.wikimedia.org/T112926#1657995 (AndyRussG) Open>Resolved [13:47:20] oh, addshore, did you know there are already API webrequest logs on stat1002? [13:47:29] they aren't straight from mediawiki [13:47:29] yup! :) [13:47:34] ok, just checkin [13:47:45] my advice about naming this job 'api' is conflicting because it! i will fix... [13:59:55] hey ottomata, regarding aqs, seems that we have machines ready, do we ? [14:01:43] holaaa [14:02:03] Hi nuria [14:02:33] joal: the vlan change has been made, i am going to reinstall them today [14:02:39] awesome :) [14:02:44] Thanks ottomata ! [14:10:09] hey addshore, q for you [14:10:19] yup! [14:10:27] is this api.log a new thing? [14:10:31] nope [14:10:35] its been around for along time? [14:10:45] afaik its been around for a long while :) [14:10:59] it only goes back to aug 21 on fluorine [14:11:25] which is 30 days ago, buuut [14:11:32] logrotate there says maxage 90 [14:11:59] hmmmmmmmm, thats oddd... im sure its older than that... [14:12:06] and, i just realized that retention days on stat1002 shorter than whatever logrotate keeps on fluorine will be weird [14:12:16] if there were actually 90 days of logs on fluorine [14:12:22] and we had retention_days => 30 on stat1002 [14:12:32] each time it will rsync all 90 days [14:12:46] and then a later cron will come around and delete anything older than 30 days [14:12:54] and then the next time the older days will be rsynced again [14:13:21] so, if something really is only keeping 30 days of api.log files on fluoroine [14:13:30] then no problem with retention_days => 30 on stat1002. [14:13:47] but, i don't see why there are only 30 days of api.log files on fluorine...seems like it should be 90 [14:14:03] hmm *looks in puppet* [14:15:02] ottomata: puppet\files\misc\scripts\mw-log-cleanup [14:16:00] ottomata: https://github.com/wikimedia/operations-puppet/blob/production/files/misc/scripts/mw-log-cleanup#L13 [14:16:04] ah! [14:16:13] perfect. then 30 days is no problem :p [14:16:18] hi nuria! [14:16:30] have you ever edited a mediawiki template? [14:24:56] Analytics-Wikistats: Adding Odia (Oriya) Wikisource to Stats Wiki - https://phabricator.wikimedia.org/T108012#1658089 (Odisha1) p:Triage>High [14:27:05] mforns: sorry, i missed your ping [14:27:11] nuria, np! [14:27:13] mforns: mediawiki template [14:27:15] for /// [14:27:27] dashiki? [14:27:34] I'm editing the schemadoc template [14:27:41] for eventlogging [14:28:07] and I wonder, when I save it, it just works? or do I need any more step? [14:28:10] mforns: i remember looking into how that worked long ago and it was so convoluted i cannot rememeber one bit [14:28:19] ok [14:28:30] np at all, I'll test it and see :] [14:29:01] mforns: some where in mediawiki there is html that gets created according to the page content type when it was created [14:29:45] nuria, you mean it is not dynamic? [14:29:54] that I will have to recompile the templatE? [14:30:10] ottomata: what are your thoughts on if I wanted to add the api logging to hadoop / hive? [14:30:15] mforns: the thing i remember is that some of it is , some of it isn't [14:30:29] mforns: as part of the html depends on the page type [14:30:39] oook, will take that into account [14:30:42] addshore: if you just want to process these exact logs in hadoop [14:30:50] you can hdfs dfs -put them into your hdfs home dir [14:30:58] and then make a hive external table on top of them [14:31:04] thanks nuria :] [14:31:18] if you want them to be productionized into hadoop, you should produce them to kafka instead of to udp2log :) [14:31:38] nuria, btw, thanks for your comments on the presentation, I changed the slides a lot, so if you want to see it again: https://docs.google.com/presentation/d/1vkfDow6hrEGXI10zX0Ev5UKB8-9ejdKuc_OO3pxiRWM/edit#slide=id.ge086cb56d_2_71 [14:31:40] ebernhardson is working on making that easier [14:31:41] he's doing it for search logs [14:31:44] okay :) [14:31:53] mforns: will take another look today [14:31:53] https://gerrit.wikimedia.org/r/#/c/229172/ [14:32:01] yeh, currently the stats I am pulling out of them are just done using a collection of greps etc.... [14:32:49] hey yall, i'm restarting eventlogging with the replace=True setting [14:32:55] am watching logs, etc. [14:32:59] as these logs have the post data the request logs miss for the api [14:33:35] ahh [14:33:35] cool [14:33:43] cat /a/mw-log/api.log | grep action=wbgetclaims | egrep -o 'property=P[0-9]+' | sort | uniq -c | sort -nr [14:33:44] xD [14:33:45] yeah, i'd say sync up with erik and see how he's doing it for search [14:33:51] and make it happen for api logs too [14:33:54] that would be awesooome [14:34:07] cool! got any tickets to link me to? ;) [14:34:43] here's his change [14:34:44] https://gerrit.wikimedia.org/r/#/c/229172/ [14:34:48] epic! [14:34:54] this bug [14:34:54] https://phabricator.wikimedia.org/T106256 [14:34:59] ticket8 [14:35:00] * [14:35:29] you do'nt have to do it in avro like he's doing, json is ok (for now :p) but i think avro would be cooler! [14:35:45] i think talk to them about their experience and see what they have to say [14:36:19] fyi, eventlogging looking fine to me! [14:44:52] mforns: real nice job [14:44:56] mforns: on slides [14:45:01] nuria, thx [14:45:43] mforns: what did you used for the "hand drawn feeling?" [14:47:40] nuria, hehehe, I drew them and scanned [14:48:09] mforns: looks good, you should export as pdf and add it to wikitech eventlogging documentation [14:48:17] aha [14:48:27] ok, will do [14:55:25] yay, it is rsyncing :D [14:56:44] exit [14:56:47] oops :) [14:56:50] xD [14:56:51] xD [15:00:31] milimetric: question for you :) [15:00:53] I have fixed the data issue on aggregator, but vital-signs doesn't update :( [15:00:59] Any idea ? [15:01:22] I thought it would be done automagicaly trhough puppet because of git, but doesn't seem so [15:02:14] joal: hi [15:02:24] checking [15:02:35] Thanks :) [15:05:06] joal: looks good to me, but caching is in place for static files now [15:05:15] Ahhhh, must be it :) [15:05:17] so if you looked at it before and after the fix within 24 hours [15:05:20] refresh [15:05:51] (I hadn't looked since Friday and the hole looks goone). I checked the files on the wikimetrics machine and they're updated. So I'll remove the annotation [15:06:02] (which btw is here https://meta.wikimedia.org/wiki/Dashiki:PageviewsAnnotations) [15:06:04] Thanks milimetric [15:06:32] I think there is local caching or something, cause I still have the hole even with refreshing [15:06:55] But knowing that it is fixed globally is good :) [15:07:07] joal: the local browser cache is the one the headers are controlling. So what browser are you using? [15:07:16] milimetric: chrome [15:07:46] joal: how'd you clear the cache? Chrome is very stubborn [15:08:09] milimetric: I nexcer do that :S [15:08:12] huhuhu [15:08:23] I am not a browser person [15:08:41] :) it's ok, so the easy way to clear the cache in Chrome: [15:09:17] open up dev tools (F12 on linux), make sure the option to not use cache when dev tools are open is checked in settings (little settings tooth wheel thing), reload the page (with the dev tools open) [15:10:28] awesome milimetric :) [15:10:30] Thanks ! [15:11:07] * joal is happy to have successfully backfilled :) [15:12:16] thx for that. potholes in data break our smooth flow [15:14:18] Analytics-Backlog: Update passport-mediawiki module URLs and documentation - https://phabricator.wikimedia.org/T113234#1658198 (Milimetric) NEW [15:18:23] milimetric: joal, aqs nodes installed and partitioned. ready for puppetization. [15:18:32] i'm reviewing the partitioning with alex, so i might have to poke at them some more [15:18:36] Yay ottomata ! [15:18:38] but if he says its cool, then we can go ahead and apply [15:19:02] /dev/mapper/aqs1001--vg-cassandra 10T 37M 10T 1% /var/lib/cassandra [15:19:58] Analytics-Backlog: Update passport-mediawiki module URLs and documentation - https://phabricator.wikimedia.org/T113234#1658211 (Milimetric) FYI, here's the source link: https://github.com/wikimedia/passport-mediawiki/blob/bd230b5c92bc0ded5a12bdac3a52a21683eecc06/lib/passport-mediawiki-oauth/oauth.js#L56 [15:20:25] ottomata: saw that in the task, it's awesome. I'm still talking to marko about the code we wrote now [15:20:34] there's another small misunderstanding [15:21:01] joal: btw, if you want to catch up on the PR that madhu and I sent on Friday, we can chat before standup [15:21:15] milimetric: great, let me join batcave [15:22:16] ottomata: the rsync of api.log may actually cause /a to fill on stat1002 [15:22:55] :/ [15:23:17] ? [15:23:28] /dev/mapper/tank-data 7.9T 7.4T 494G 94% /a [15:23:57] oh ha, i think i was looking at /mnt/data when i looked at this the other day. [15:24:06] why so much in /a! ? looking. [15:24:24] Analytics-EventLogging, Analytics-Kanban: Regularly purge EventLogging data in Hadoop {stag} [8 pts] - https://phabricator.wikimedia.org/T106253#1658220 (mforns) a:mforns [15:24:29] 424gb already copied, roughly 375 left, so /a should have roughly 100gb left after [15:24:53] just thought I would let you know! as that's obviously not what either of us were expecting! [15:25:42] yeah, ok. am looking at /a stuff, seeing if there is anythign we don't need there anymore [15:31:25] (PS1) Christopher Johnson (WMDE): changes per prototype review adds preliminary owl for metric definitions adds dates and percentage to infoBox reorganizes tabs adds items as a separate graph [wikidata/analytics/dashboard] - https://gerrit.wikimedia.org/r/239855 (https://phabricator.wikimedia.org/T108404) [15:35:31] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Handle duplicate insert error {stag} [5 pts] - https://phabricator.wikimedia.org/T112265#1658242 (kevinator) Open>Resolved [15:40:43] Analytics-Kanban, Patch-For-Review: Puppetize a server with a role that sets up Cassandra on Analytics machines [13 pts] {slug} - https://phabricator.wikimedia.org/T107056#1658359 (kevinator) [15:40:44] Analytics-Kanban, hardware-requests, operations, Patch-For-Review: Request three servers for Pageview API - https://phabricator.wikimedia.org/T111053#1658357 (kevinator) Open>Resolved [15:47:16] hey yurik, yt? [15:47:27] ottomata, hi [15:47:42] hey, /a/zerologs on stat1002 is 1.3T [15:47:47] do you still need it? [15:48:05] ottomata, beh, need to look [15:48:17] nothing new there in almost a year [15:48:24] 1.3 Gigawats!!! Great Scot! [15:48:26] it hasn't updated in over a year? [15:48:46] milimetric, that's dzigawats! [15:49:08] find /a/zerologs -mtime -325 [15:49:11] ./.git [15:49:11] ./.git/index [15:49:26] sec, lemesee [15:57:15] ottomata, deleted [15:57:26] Analytics-Kanban: Mark schemas as "containing user-inputed textual data" and add publish section to the docs {tick} - https://phabricator.wikimedia.org/T112271#1659190 (kevinator) Open>Resolved Done! for example: https://meta.wikimedia.org/wiki/Schema_talk:PageCreation [15:57:57] Analytics-Kanban: Prepare lightning talk on EL audit {tick} [5 pts] - https://phabricator.wikimedia.org/T112126#1659192 (kevinator) Open>Resolved [16:04:28] yurik: thank you! [16:09:28] Analytics-Backlog, Analytics-Cluster: Test Impala operationally {hawk} - https://phabricator.wikimedia.org/T96331#1659545 (kevinator) [16:11:48] Analytics-Cluster, Analytics-Kanban, operations: Fix llama user id {hawk} - https://phabricator.wikimedia.org/T100678#1659546 (kevinator) [16:12:05] Analytics-Backlog, Analytics-Cluster: Productionize Impala {hawk} - https://phabricator.wikimedia.org/T96331#1659549 (Ottomata) [16:14:38] Analytics-Cluster, Analytics-Kanban, operations: Fix llama user id {hawk} [5 pts] - https://phabricator.wikimedia.org/T100678#1659563 (Ottomata) [16:14:40] Analytics-Cluster, Analytics-Kanban, operations: Fix llama user id {hawk} [5 pts] - https://phabricator.wikimedia.org/T100678#1659564 (kevinator) p:Triage>Normal [16:17:49] Analytics-Cluster, Analytics-Kanban: Create Kafka deployment checklist on wikitech {hawk} [5 pts] - https://phabricator.wikimedia.org/T111408#1659572 (kevinator) [16:19:37] Analytics-Kanban: enforce group-writeable in stat1002:/a/aggregate-datasets/ and stat1003:/a/public-datasets/ {hawk} [3 pts] - https://phabricator.wikimedia.org/T111956#1659593 (kevinator) [16:20:44] Analytics-Kanban: enforce group-writeable in stat1002:/a/aggregate-datasets/ and stat1003:/a/public-datasets/ {hawk} [3 pts] - https://phabricator.wikimedia.org/T111956#1659595 (Milimetric) [16:21:03] Analytics-Kanban: enforce group-writeable in stat1002:/a/aggregate-datasets/, stat1003:/a/public-datasets/, and stat1003:/a/limn-public-data {hawk} [3 pts] - https://phabricator.wikimedia.org/T111956#1659602 (Milimetric) [16:37:54] Analytics-Backlog, Analytics-Cluster: Implement better Webrequest load monitoring {hawk} - https://phabricator.wikimedia.org/T109192#1659795 (kevinator) [16:49:52] Analytics-Backlog, Analytics-Cluster: Add percent_change to webrequest_sequence_stats_hourly {hawk} [5 pts] - https://phabricator.wikimedia.org/T113251#1659868 (kevinator) NEW a:Ottomata [16:55:39] Analytics-Backlog, Analytics-Cluster: webrequest_sequence_stats_hourly fails/continues Load job {hawk} [13 pts] - https://phabricator.wikimedia.org/T113252#1659883 (kevinator) NEW a:Ottomata [16:56:01] Analytics-Backlog, Analytics-Cluster: {epic} Implement better Webrequest load monitoring {hawk} - https://phabricator.wikimedia.org/T109192#1659891 (kevinator) [16:58:00] ottomata: why is the load distribution of stat1001/2/3 so unevenly distributed? ie, basically everything is on stat1002? Just curious [16:58:17] stat1002 has access to hadoop, and to private webrequest log data [16:58:23] so there's more there to work with [16:58:26] so people use it more [16:58:42] stat1001 is just a webserver [16:59:28] Analytics-Wikistats: Adding Odia (Oriya) Wikisource to Stats Wiki - https://phabricator.wikimedia.org/T108012#1659908 (ezachte) Oriya Wikisource has been added a while ago: https://stats.wikimedia.org/wikisource/EN/TablesWikipediaOR.htm and is listed on the Wikisource Sitemap: https://stats.wikimedia.org/wik... [17:00:56] Analytics-Backlog, Analytics-Cluster: Send email when load jobs fails {hawk] [8 pts] - https://phabricator.wikimedia.org/T113253#1659915 (kevinator) NEW [17:01:15] Analytics-Wikistats: Adding Odia (Oriya) Wikisource to Stats Wiki - https://phabricator.wikimedia.org/T108012#1659921 (ezachte) Open>Resolved [17:01:45] Analytics-Backlog, Analytics-Cluster: webrequest_sequence_stats_hourly fails/continues Load job {hawk} [13 pts] - https://phabricator.wikimedia.org/T113252#1659923 (kevinator) a:Ottomata>None [17:08:28] Analytics-Backlog, Analytics-Cluster: Improve daily webrequest partition report {hawk} [5 pts] - https://phabricator.wikimedia.org/T113255#1659949 (kevinator) NEW [17:14:02] ottomata, so is there a hard-and-fast answer to "how far back do varnish logs go"? [17:14:04] (in Hadoop) [17:15:18] Ironholds: webrequest you mean ? [17:15:28] joal, indeedy-doody [17:15:42] raw: 1 month, refined: 2 month [17:16:40] that is the answer. [17:17:06] hard-and-fast-est as I could Ironholds :) [17:17:16] Ironholds: https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/analytics/refinery.pp#L88 [17:17:21] joal, great, thanks! [17:17:28] ottomata, poifect :D [17:17:42] and by refined we mean the wmf.webrequest table? [17:17:48] to be distinguished from the un-deduped data [17:19:44] hi milimetric. Thank you for the changes. :-) [17:20:06] Ironholds: you are correct [17:20:07] the recommendations are working much more smoothly and more intuitive. [17:20:11] danke! [17:36:18] Ironholds, hi! do you know of any on-wiki documentation that explains the differences between the old pageview definition and the new one? [17:36:31] mforns, the old one stunk [17:36:41] :] [17:37:01] but, not to my knowledge. [17:37:14] I can tell you the primary differences off the top of my head buuut [17:37:23] ok, we're tasking and wanted to know if there was some [17:37:26] thanks! [17:37:47] gotcha [17:41:10] Analytics, Analytics-Cluster, Fundraising Tech Backlog, Fundraising-Backlog, operations: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1660099 (Jgreen) > What is the highest sampling rate we can afford? Significantly unknown. Load is pretty low... [17:55:41] Analytics, Analytics-Cluster, Fundraising Tech Backlog, Fundraising-Backlog, operations: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1660173 (ellery) + 1 [18:01:44] Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on tin.eqiad.wmnet - https://phabricator.wikimedia.org/T112660#1660194 (ori) a:ori [18:05:45] Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on tin.eqiad.wmnet - https://phabricator.wikimedia.org/T112660#1660224 (Ottomata) @Ori, btw, I think @Krinkle can do what he needs now from stat1002 instead of tin. [18:06:14] Analytics-Backlog, Analytics-EventLogging, Performance-Team, Patch-For-Review: Make webperf eventlogging consumers use eventlogging on Kafka {stag} - https://phabricator.wikimedia.org/T110903#1660225 (Ottomata) @ori this might be worth putting into next up for performance team. [18:11:34] Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on terbium - https://phabricator.wikimedia.org/T112660#1660239 (Krinkle) [18:15:46] nuria aaaaaaH ! [18:15:49] cave ? [18:17:05] Analytics, Analytics-Cluster, Fundraising Tech Backlog, Fundraising-Backlog, operations: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1660255 (Jgreen) Ok, this is switched for all americium collectors. [18:25:53] madhuvishy: I'm about to talk to Marko and them about the endpoints [18:26:07] do you wanna be looped in / work on them or is it ok with you if I take it over now? [18:26:21] sounds like just last minute cleanup, I'm looking at jscs syntax problems now [18:26:52] joal: sure! [18:27:00] joal:yt? [18:27:16] yup [18:27:18] :) [18:28:33] milimetric: sure go ahead, i just started reading the comments [19:23:50] joal, yt? [19:24:09] yup [19:24:51] so the select column where in (Blah) . the blah-clause cannot select two things seems to me [19:25:18] https://www.irccloud.com/pastebin/kqhIuRJl/ [19:26:37] right nuria [19:26:40] hm [19:26:42] sorry, let me explain the select in [19:26:46] ah , sorry, i see [19:26:47] I have it [19:27:10] You need the count to be in the selection clause in order to be able to use AHVING [19:27:27] And you can't have two clauses in the IN subquery [19:27:34] So, solution: use a join :) [19:28:14] ah , so joins work in the subquery -> ooohhh [19:28:19] will do [19:28:44] Less easy to read than 'blah IN (subquery)', but works fine [19:29:30] not join inside the subquery. join the subquery with the parent on the filed you are interested in keeping [19:30:05] joal; ok [19:32:49] *waves at all* [19:33:20] If I am going to have a repo full of scripts and things that essentially make tsv files (which should eventually be puppetized) is there a convention for where that repo should be? [19:33:35] currently we have a limn one but, well, we wont be using limn...... [19:35:39] addshore: thus far (i know, sad) we have been using those limn depots [19:35:49] okay :) [19:53:14] addshore: limn-<>-data is the correct naming convention. At some point we'll migrate all of them to a different name, and update all the puppet that uses that convention [19:53:20] but until then it's useful to have consistency [19:53:33] cool ;) [19:54:37] ottomata: got some time to talk about gobblin? [19:55:54] Analytics-Kanban: Investigate sample cube pageview_count vs unsampled log pageview count - https://phabricator.wikimedia.org/T108925#1660700 (JAllemandou) @JKatzWMF, @Tbayer : Please let me know if this explains enough :) Some data on differences from hive vs pageview05 (I call it cube) data on April month (... [19:57:05] madhuvishy: 1:1 with kevin in 3, then sure! [19:57:15] Bye a-team ! [19:57:21] See you guys tomorrow ! [19:57:22] ottomata: okay cool, ping me when you're free [19:57:28] good night joal :) [19:59:06] k [19:59:09] nighters! [20:15:43] milimetric: yt ? to help me with a hive query? [20:18:58] nuria: hi, sure [20:19:17] milimetric: let me paste it on an etherpad [20:19:26] we can hangout if you want [20:20:00] milimetric: https://etherpad.wikimedia.org/p/pii_pageviews [20:20:06] milimetric: ok, batcave [20:20:13] milimetric: thank you [20:30:10] i want to hang out with dan too! [20:30:15] and also madhu [20:31:30] ottomata: :) [20:32:18] ottomata: https://etherpad.wikimedia.org/p/gobblin-sprint [20:36:41] ottomata: should we talk in another room? [20:37:23] ja [20:37:39] ottomata: https://plus.google.com/hangouts/_/wikimedia.org/gobblin [20:47:53] nuria, funny, there's a "collect_set" in hive, which removes duplicates but otherwise works like group_concat. I tried it, here's the SO answer: http://stackoverflow.com/questions/22705398/column-to-comma-separated-value-in-hive [20:48:03] and here's an example query that I just ran and worked on the cluster: [20:48:07] https://www.irccloud.com/pastebin/jzezLA9T/ [20:48:34] one sec, re-writing your query in this way [21:05:32] Analytics-Cluster, Analytics-Kanban: Spike replacing Camus with Gobblin {hawk} [13 pts] - https://phabricator.wikimedia.org/T111409#1661048 (madhuvishy) https://etherpad.wikimedia.org/p/gobblin-sprint [21:08:10] getting pretty close to having mediawiki produce events to kafka. Is there anything i need to do (or tickets i need to create for someone else?) to get that data moving from kafka into hdfs and hive? [21:08:49] oh! yes, welllllLlll [21:08:55] once it is going into hdfs [21:08:58] soryr [21:08:59] sorry [21:09:11] hm, stepping back a sec. [21:09:17] 1. topic needs to be created in kafka [21:09:21] auto_create_topics is enabled [21:09:26] but by default each topic will have only 1partitoins [21:09:34] ebernhardson: how much are you going to throw at this topic? [21:09:51] ottomata: well, probably first a test topic with junk data to later be deleted, just to make sure its not going to error out [21:10:00] ottomata: then, about 150-250M/day [21:10:09] milimetric: if you need help fixing the things they pointed out in endpoint comments/just chat about it let me know! [21:10:47] ottomata: not entirely sure on record size yet, the schema we are considering is https://phabricator.wikimedia.org/T112295#1653078 so its not tiny [21:10:47] ebernhardson: can't delete topics in kafka :) [21:10:53] heh [21:10:59] milimetric: ahajam [21:11:14] ok so yea we should create it with more partitions. [21:11:19] will do 12 like we do for others. [21:11:34] ebernhardson: we should think of a good name for htis topic [21:11:35] ottomata: oh but then not 150-250M/day, i forgot. the 150-250M is total requests from mediawiki->es, but this new schema will actually aggregate together requests from the same process, so maybe more like 50-100M [21:11:39] nuria: it's at 100% right now [21:11:47] holy crap there are a lot of results [21:11:49] something that will make sense if more people use this [21:11:54] maybe mediawiki_ prefix? dunno [21:11:55] 32 thousand cities :) [21:12:12] nuria: Time taken: 1246.689 seconds, Fetched: 37277 row(s) [21:12:19] mediawiki_CirrusSearch ? [21:12:29] does your avro schema have a name yet? [21:12:29] mediawiki_CirrusSearch_requests [21:12:42] ottomata: we were thinking either CirrusSearchAggRequests, or cirrusrequests [21:12:45] madhuvishy: no worries, I have to run out for a bit now but I think I'm ok. I'll poke you later if I get this done to submit a new PR [21:12:55] oliver likes cirrusrequests because it's similar to webrequest in hive already [21:13:00] milimetric: okay :) [21:13:17] milimetric: all right, will mine so it looks at ips and cities, will be back in less than an hour [21:13:24] hm, i think we should use the caps because avro and it seems more standard, especially if/when it gets classes generated for it [21:13:31] don't really like Agg, but iunno [21:13:34] nuria: I pasted you the right query privately [21:13:45] mediawiki_CirrusSearchRequest seems fine [21:13:46] ? [21:13:52] ah but it is multiple requests? [21:13:53] hm [21:13:56] milimetric: thankyuuu [21:13:59] ottomata: we already have CirrusSearchReqests, which is a log line per request from cirrus to elastic, so wanted to call it Agg to capture that its not the same [21:13:59] is it kinda like a session? [21:14:06] ottomata: its a single php proces [21:14:06] hm, k [21:14:41] ottomata: so a process might do both a full text search and a title search, this groups them together to simplify analysis understanding what is a separate web request [21:14:44] yeah but Agg (to me) sounds like you are aggregating analysis for requests based on something, like time [21:14:49] hourly # of requests kidna [21:14:54] yea i can see that [21:15:23] naming things is hard :P [21:15:33] i think there should be a name that indicates it is from a single php process [21:15:40] it is from one webrequest, right [21:15:51] 1 webrequest amking Cirrus do multiple ES requests? [21:16:00] it can, depending on what happens [21:16:18] but this doesn't aggregate ES requests from multiple webrequest, right? [21:16:20] for example if your search gets no search results but we have a suggestion, we run a second search with the suggestion to try and give results [21:16:26] ottomata: nope, just a single web request [21:16:32] (or job in the job queue) [21:16:34] client -> varnish -> appserver -> ES1, ES2,... [21:16:34] ? [21:16:40] oo or job. [21:16:41] hm [21:16:49] i see [21:17:01] and yes, except an LVS between appserver and ES* [21:17:15] Set? [21:17:23] CirrusSearchRequestSet [21:17:24] ? [21:17:32] hmm, that works [21:17:58] ok, ebernhardson since you are treading very new water here, i betcha whatever we come up with will be regretted later :p [21:18:06] but, i thikn mediawiki_CirrusSearchRequestSet sounds good [21:18:16] lol, pretty much :) prepending works for me [21:18:21] so, i can create topic, and I will need to make a camus job to import the data from Kafka [21:18:41] ebernhardson: your Avro schema can be named just CirrusSearchRequestSet [21:18:42] I thikn [21:18:43] ... [21:19:04] do you need the schema? its not finalized yet but i'm doing that today/tomorrow now that we have a decision on exactly what to collect [21:19:09]