[07:55:34] [[Tech]]; ArchiverBot; Bot: Archiving 1 thread (older than 30 days) to [[Tech/Archives/2015]].; https://meta.wikimedia.org/w/index.php?diff=15178836&oldid=15163116&rcid=7184918 [10:35:44] Hi, I have some questions about the enwiki database dumps. I'm using this link -- http://dumps.wikimedia.org/enwiki/20151201/ [10:36:48] I'm getting what I need from enwiki-20151201-stub-meta-history.xml.gz , which is the edit-history metadata , but it is ~45gb and takes a couple hours per processing-run [10:37:56] there are also several smaller files , like enwiki-20151201-stub-meta-history3.xml.gz , up thru history27.xml.gz -- can someone tell me what those contain? [10:47:47] another option is the daily-changes dumpfiles here , https://wikitech.wikimedia.org/wiki/Dumps/Adds-changes_dumps , experimental incremental dumps by User:ArielGlenn ... are these daily dumps public, yet? [10:53:59] ah, here it is: https://dumps.wikimedia.org/other/incr/enwiki/ , which has daily-incremental-dumps for the past five weeks (from 18th Nov 2015 through 23rd Dec 2015). but not older stuff? [13:41:57] hi [13:42:46] I have a project in github, how I could update the source in wmflabs automatically [13:42:48] ? [13:48:01] You're probably better of going into #wikimedia-labs [13:48:24] But in a nutshell: The easiest is probably to use a cron job that just does git pull from time to time [13:50:07] thanks [13:52:43] hoo: could you show me how set cron in toolserver? [13:52:47] sorry wmflabs [13:53:43] hoo: I don't know if is a good idea do http request each minute [13:54:08] You probably want it to be more like daily [13:55:42] hoo: i need test it in the server, I can´t wait [13:55:48] See https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid [13:56:40] well, then do it by hand initially [13:57:01] hoo: I could change the autologout time in toolserver? [13:57:19] autologout time? [13:57:28] Logout from what? [13:58:02] hoo: ssh login [13:58:16] There is no autologout [13:58:25] at least I'm not aware of one [13:58:42] mm [13:58:48] ask your distribution. [15:17:31] Can someone look at T115812 on Phab. and its corresponding patch on Gerrit and see to get it done? Requested in on Oct 18, and it seems stalled since Nov 24. [19:08:21] does anybody know something about that page view api? [19:08:37] the main question I have so far is why is it in some weird place [19:09:05] rather than just extending api.php and having it's "interface" in the APISandbox special page [19:09:12] *its [19:15:23] Isn't it not MW? Essentially [19:19:06] Reedy: well it probably isn't but why can't it be? [19:19:31] Because the data isn't stored in MW [19:19:49] In theory, the mw api could be used [19:19:53] Why it isn't, I can't say [19:40:57] Base-w: it'd be helpful if you left that comment here: https://phabricator.wikimedia.org/T112956 [19:42:17] both for us as we think about our level of ongoing investment in Pageview API, and for you getting an answer to your question [19:43:39] Reedy's main point is one of the bigger reasons by my understanding: because the data isn't stored in MW [20:07:45] Base-w: here are docs for pageview API: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageview_API [20:08:22] Base-w: it is completely decouple from mw software and as such lives of a different stack [20:08:25] *decoupled [20:09:26] Base-w: query-able using a REST client, there are several clients alredy js, python and R [20:09:30] *already [20:26:52] nuria: well but isn't it possible to device some extension to MW so that then it is in MW? I am not that technical person to know the exact way, I just know that many extensions have their own database tables in MW so it looks like it should be possible to attach those data to MW too [20:27:33] It's just that when data are scattered throughout different places, people need to remember where is what. and it rather makes life harder :) [20:29:44] As I said, it would be possible. It's whether there's enough benefit/desire to do so [20:29:57] And as such, whether the WMF would put development time to doing so [20:31:20] Base-w: Can you leave a comment/request per robla on T112956? [20:32:28] I'll try though in that task there are already so many comments left, I would probably be too lazy to read them all [20:32:46] I didn't say you needed to read it all :) [20:33:10] in the meanwhile, does that API provide XML output? [20:34:07] Doesn't look like it [20:34:33] :/ [20:34:41] Most people would suggest to not use XML in the normal MW API either ;) [20:35:26] well I use Wiki.java which is based on XML output and I do not really know how to parse JSON [20:35:58] well some libraries probably exist but it takes me a long time even to attach a library to a project so I tend to avoid them [20:36:03] :) [20:49:00] added a comment in the task [20:50:10] * robla would love to hear what anomie thinks about https://phabricator.wikimedia.org/T112956#1901909 [20:52:25] Base-w: the answer to that is technical but basically there is no benefit to share storage technology with mediawiki backend which is optimized for a completely different type of data. [20:52:47] Base-w: no xml, it is json, you might want to read the docs I just sent [20:53:20] Base-w: I do not see us launching anything that produces xml anytime soon [22:04:41] fyi, I decided to go ahead and have office hour on #wikimedia-office right now [22:05:19] https://phabricator.wikimedia.org/E130 for details [22:10:17] hello [22:10:25] nuria: I see that it's json [22:10:40] the problem is that if it were xml I could use it right away [22:10:55] now I have to spend time on learning how to parse json [22:11:52] is this the right place to ask about turkish wikiquote wiki? [22:12:03] HakanIST: depends on the question :) [22:12:31] rollback feature is not enabled on editors anyone in general [22:12:42] I was wondering if the wiki was lacking or if it was a configuration issue [22:13:19] @link [[tr:q:Special:ListGroupRights]] [22:13:19] https://tr.wikipedia.org/wiki/q:Special:ListGroupRights [22:14:45] HakanIST: well now sysops have the right, editors do not [22:14:50] which is default [22:15:41] if you want editors to have the right, or perhaps to have a separate Rollbacker flag like many other wikis have, you've got to make a discussion on the wiki asking community to vote about this proposal [22:16:33] after at least one week if it's successful you've got to go to phabricator [[phab:]] and file the task for making the change in Turkish Wikiquote config [22:16:35] @link [22:16:35] https://wikitech.wikimedia.org/wiki/phab: [22:17:06] thanks Base-w thats most helpful [22:17:11] https://phabricator.wikimedia.org/maniphest/task/create/ better link [22:17:28] np [22:30:04] fork: Resource temporarily unavailable [22:30:12] anyone know why I can't commit over SSH? [22:31:13] "This is commonly caused by running out of file descriptors." [22:31:54] hmmm, ok, anything I can do? https seems to work ok [22:32:24] What does this tell yoy? [22:32:25] sysctl fs.file-nr [22:33:18] command not found [22:34:22] What OS? [22:35:15] Win 10 [22:35:34] Are you commiting from a vagrant vm or similar? [22:36:21] don't think so - I have it installed on my PC, but I don't think it's running [22:36:30] no, it's dead atm [22:37:21] cygwin? [22:38:18] Nope [22:39:00] Ahh, think I've got it [22:39:23] It's because I've overwrote it in the same place so many times, I moved it elsewhere and it works :) [22:55:19] ori, idk perhaps because of the yesterday's change but now adminstats do not seem to work for *.wikimedia.org [22:55:19] could you look into this? [23:04:32] Anyone familiar with mwclient? [23:04:39] (The python module) [23:04:50] Base-w: OK, looking [23:07:53] Base-w: could you open an issue on github? [23:08:02] ok [23:09:35] https://github.com/x-tools/xtools/issues/84 [23:17:13] Base-w: should be fixed [23:17:20] e.g. https://tools.wmflabs.org/xtools/adminstats/?project=meta.wikimedia.org&begin=2015-10-14&end= [23:19:44] yep seem to work, though does not work for chapters' wikis [23:20:03] ori: [23:20:50] thanks for this change though, it's still better than nothing :) [23:21:59] which wiki does it not work for atm? [23:22:27] works for meta and outreach at least [23:22:37] does not for uawikimedia and ruwikimedia at least [23:23:20] doesn't for wikimania2016 too [23:29:04] nuria: robla: oh that pageview api doesn't work for ukwiki at all? [23:29:07] useless thing [23:31:10] * robla doesn't know [23:33:37] Base-w, https://wikimedia.org/api/rest_v1/metrics/pageviews/top/uk.wikipedia/all-access/2015/12/22 ? [23:35:49] Krenair: I was trying https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/uk.wikipedia/all-access/all-agents/monthly/2015110100/2015113000 [23:36:28] so no monthly data for november, basically? [23:36:35] it seems [23:36:44] nor for october [23:37:40] nor https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/uk.wikipedia/all-access/all-agents/monthly/2015120100/2015121500 [23:39:24] well probably a bot/tool can just request data for each day and make the sums but it's rather silly when aggregate data is expected to exist [23:44:25] robla: https://phabricator.wikimedia.org/T112956#1902709 [23:52:58] * robla looks [23:54:44] anomie: excellent response, thanks! [23:59:59] how hard would it be for someone to make meetbot automatically post office hour notes to Phab pastes, e.g. https://phabricator.wikimedia.org/P2453