[12:51:36] hi [12:52:09] hi [12:57:28] hello everyone. [12:57:36] hi [12:58:30] @zhuyifei I had proposed a new consumer. [12:58:35] k [12:58:46] I am not sure if this time I made any mistake. [12:59:11] I chose commons.wikimedia and also requested upload files grant [12:59:22] version also changed to 1.1 [12:59:34] k [12:59:44] sounds good [13:00:21] pywikibot.data.api.APIError: permissiondenied: The action you have requested is limited to users in one of the groups: *, [[Commons:Users|Users]]. [13:06:33] I think the login was successful. [13:06:50] So this error is not expected. [13:07:04] yeah [13:07:17] so does it work with the new consumer? [13:07:55] No with the new consmer token replaced in the config files I got this error. [13:08:32] can I see your code and grants? [13:08:42] yes [13:09:00] how can I enable you to see them? [13:09:35] actually, if it's publically accessable, you don't have to do anything [13:09:38] let me check [13:10:38] so which line do you get the error? [13:10:53] just a sec [13:13:34] it's from inside pywikibot [13:13:37] File "/data/project/sibutest/www/python/venv/lib/python3.4/site-packages/pywikibot/data/api.py", line 2196, in submit raise APIError(**result['error']) [13:13:55] can you paste the full traceback [13:14:23] Uploading file to commons:commons... Reading file http://afbeeldingen.gahetna.nl/naa/thumb/10000x10000/9a2d41b4-3d2e-4796-b1a3-8bbee887f2ba.jpg Sleeping for 8.5 seconds, 2017-06-04 01:46:56 WARNING: API error permissiondenied: The action you have requested is limited to users in one of the groups: *, [[Commons:Users|Users]]. ERROR: Upload error: Traceback (most recent call last): File "/data/project/sibutest/www/python/venv [13:14:49] File "/data/project/sibutest/www/python/venv/lib/python3.4/site-packages/pywikibot/specialbots.py", line 423, in upload_file comment=self.summary) File "/data/project/sibutest/www/python/venv/lib/python3.4/site-packages/pywikibot/page.py", line 2635, in upload **kwargs) [13:15:03] ok [13:15:08] File "/data/project/sibutest/www/python/venv/lib/python3.4/site-packages/pywikibot/tools/__init__.py", line 1458, in wrapper return obj(*__args, **__kw) File "/data/project/sibutest/www/python/venv/lib/python3.4/site-packages/pywikibot/site.py", line 6286, in upload raise error [13:15:15] what oauth grant did you give? [13:15:47] Upload new files +Upload, replace, and move files [13:16:02] and Basic Rights by default. [13:16:39] try add "Create, edit, and move pages" [13:16:57] ok sure. [13:17:15] I remember upload upload is dependent on having create permissions and sometimes edit permissions [13:17:35] and that sometimes break a lot of stuffs [13:17:46] regarding protection and uploading [13:18:06] alright. [13:19:05] how about adding "Edit existing pages"? [13:19:37] is that necessary? [13:19:52] shouldn't matter, but you can add that if you want [13:19:59] ok [13:20:54] give a minute [13:21:15] I will try and report. [13:26:10] ah thanks a lot [13:26:15] Upload done :) [13:26:21] k [13:26:23] np [13:27:56] well I should be using site.upload() [13:28:03] as you had suggested [13:28:04] yeah [13:28:28] So I will try to modify the code and avoid UploadRobot [13:29:48] About the UI did you use Bootstrap for v2c? [13:30:04] yeah [13:30:27] I went through the basics of Bootstrap. [13:30:53] Should I be using Bootstrap? [13:31:23] hi [13:31:26] sorry, I went afk [13:32:21] hi Basvb did you get a chance to go through yesterday's logs? [13:32:40] https://wm-bot.wmflabs.org/logs/%23wikimedia-sibu/20170603.txt [13:32:42] reading them now [13:32:57] infobliss: up to you whether to use bootstrap [13:33:15] alright. [13:35:14] infobliss that file does not seem to be in their database at all [13:35:23] and it's not CC [13:36:00] ohk [13:36:16] but the user may end up on this image while browing. [13:36:27] browsing [13:36:39] well we should have some error messages when something goes wrong [13:36:45] file not found error [13:36:55] or file already on commons (+ here is the link) [13:37:03] and maybe some other common things [13:37:30] In fact I got this image while browsing. But I did not know that it is not in their DB. [13:37:48] It's there in their website. [13:38:29] yep it's a bit strange [13:38:33] well it's in their database of course [13:38:37] but not in their API [13:39:02] so maybe there is a parameter for that, indicating whether files are public for the website, the API, both or none [13:39:35] hmm [13:40:35] can we find that parameter? [13:40:40] Or may be if the json response is not right then we give a file not found error. [13:41:26] yes [13:41:37] we can not find that parameter I think [13:41:44] but people can also put in a wrong name [13:41:52] we can just give a file not found error [13:42:05] So the GSOC has officially started [13:42:13] yep [13:42:17] we should get around to bringing some structure in the code base I think [13:42:40] Our test tool has uploaded the first image :) [13:42:41] and I should put some more time in reviewing the work and proposing steps to take [13:42:44] aah nice [13:42:45] yes [13:44:09] it's under http://tools.wmflabs.org/sibutest/ currently? but the Oauth issues are blocking I guess? [13:44:52] Actually only I can upload as long as the new consumer I just registered is approved. [13:45:00] https://commons.wikimedia.org/wiki/File:Kerstversiering_-_Nationaal_Archief_-_189-0946.jpg [13:45:06] that's an image you uploaded? [13:45:21] yep this one [13:45:26] ok 2 things [13:45:36] you know what happened with the category/ [13:45:39] you know what happened with the category? [13:46:00] I guess the goal was to have one category: "image category" ? [13:46:30] I thought if the category is not given it becomes the default [13:46:37] just a sec let me see the cod [13:46:40] code [13:47:56] the second thing is, please review the uploads and try to remove such things, we're testing the uploads, but they'll stay there as regular uploads so having the correct info is important [13:48:15] or give me a ping from time to time to do that (if it is concerning dutch stuff) [13:48:49] Category:Images from the Nationaal Archief needing categories [13:49:02] ok I will keep that in mind from next time. [13:50:31] Do you have good plans of what you want to work on in the next week? Or should we discuss that a little bit? [13:51:00] and on what points are you getting stuck/ would you like our support/help [13:51:34] OAuth is one thing I believe? But I think at the same time it's good to work on some other parts of the code [13:52:18] ok sure we can discuss to define the set of tasks for this week [13:52:20] OAuth is working as expected. [13:52:38] ok, you only need to await approval? [13:52:49] You should be able to test that after approval. [13:52:54] ok, that's good [13:53:42] What should have been the image category for this image [13:53:49] https://commons.wikimedia.org/wiki/File:Kerstversiering_-_Nationaal_Archief_-_189-0946.jpg [13:54:02] the user had to find one, or categories needed [13:54:26] I found "Christmas decorations in the Netherlands", but there's not way to do that automatically [13:55:16] So what I think would be good is to discuss the code structure a bit [13:55:53] ok [13:55:55] https://github.com/infobliss/sibutest is that fully up to date? [13:56:13] yeah [13:56:37] ok [13:56:53] except the config files [13:57:13] there I replaced the new consumer tokens [13:57:33] So for the back-end (the nationaalarchief.py file mainly) [13:57:49] ideally we want to split this up into common functions and specific functions [13:58:06] where as much as possible is done in general clean functions [13:58:24] and for a specific glam only mapping the metadata fields to the commons template fields is what has to be done [13:59:00] so we have a folder with some glam-specific codes, which make use of a library of some general purpose function [13:59:16] the main ones will be the mapping to the commons template [13:59:24] for example here the template photograph is used [13:59:46] ideally we have a function where all the parameters in the template are given to as parameters [13:59:51] this fills the template and returns it [14:00:31] in the current code we do articletext+= and add all the text (that wasin my hacky version) [14:00:48] but that makes the code a bit complicated and makes it hard to reuse parts of it [14:00:56] ok got it. [14:01:18] so ideally we just collect the correct parameters from the commons template [14:01:41] {{Photograph |photographer = Willem van de Poll |title = {{nl|Kerstversiering}} |description = {{nl|Kerstversiering}} [14:02:01] so we collect things like: photographer = Willem van de Poll [14:02:05] title = ... [14:02:29] hmm [14:02:31] and then call: commons_template_filler.photograph(photographer, title, description...) [14:02:38] and it returns the filled template [14:03:12] then we can also make a library function for some of the substeps in their [14:03:18] date parsing, title cropping [14:04:10] for example library.title_cropper(long description, title=None, collection, identifier) which returns a short clean title [14:04:32] this ensures that we can make several uploads easily, which reuse the same elements [14:04:54] maybe I should also help a bit on that code, especially the commons-specific things [14:05:08] ok [14:05:08] on the other hand we should think a bit about user experience [14:05:14] and the UI design/steps in uploading [14:05:27] regarding date [14:05:43] we are putting up whatever is returned by Inhoudsdatering [14:06:00] this time it's easy [14:06:01] do you want some more formatting? [14:06:08] they use the perfect form for commons [14:06:23] but sometimes it will say: 1 september 2017 on their end [14:06:27] or 1 mei 2017 [14:06:37] then we have to change that to 2017-05-01 [14:06:42] ok [14:06:58] or they will say start date: 1915 end date 1920 [14:07:17] then we have to make that into: {{other date|between|1915-1920}} [14:07:37] so those functions don't have to be build now [14:07:46] but when we come across a glam having start and end dates [14:07:54] we'd want to make that into a library function [14:08:05] because then for the next glam we don't have to write that funciton again [14:08:21] ok [14:08:45] maybe a good plan is to see if we can find a suitable 2nd glam [14:08:59] then we can work a bit more on making things general solutions [14:09:04] instead of just working for 1 glam here [14:09:14] hmm yes. [14:11:53] Is the europeana collection in collaboration with wikimedia? [14:12:43] I don't think so [14:12:49] but the goals align really well [14:14:44] infobliss: could you go over https://phabricator.wikimedia.org/T164668 and the subtasks [14:14:58] and mark your progress, where you think more needs to be done, and where we can help with [14:15:57] ok sure. [14:16:05] I will do that. [14:16:35] I think [14:17:33] regarding the compiling a list GLAMs I did not make much progress except listing them. [14:18:20] yep, well maybe we added a bit too much to the list since it was just the getting to know phase [14:18:27] but it's important to keep track of where we are [14:18:32] and with what you need help [14:19:08] to ensure that things keep moving forward [14:19:26] well I have a few things in mind. [14:19:52] 1. The UI is to be built. [14:20:19] images are getting uploaded from NA [14:20:33] yep I think it is good to work a bit on the UI [14:20:34] 2. restructuring the code [14:20:40] is remaining [14:21:15] 3. Handling duplicate uploads [14:21:16] yes good one as well for this week, I think that it is good to look into a 2nd archive because this makes restructing into using more library-type functions easier [14:21:35] but I get it's difficult to find a suitable one [14:22:09] well is the list from where I compiled the GLAMs are all in collaboration with wikimedia? [14:22:39] Is that an exhaustive list? [14:22:50] you will never get an exhaustive list ;) [14:23:09] the collaborations are mainly between a group of wikipedians and a glam [14:23:16] never with wikimedia foundation as a whole [14:23:26] often with chapters (country wikimedias) [14:23:36] oh alright. [14:23:38] but often the glam also just releases their images [14:23:51] not specifically with Wikipedia/media in mind [14:25:19] You would definitely be having correspondence with some more GLAMs then. [14:25:40] me specifically or Wikimedia as a whole? [14:25:49] you [14:25:59] you mean about the button or something else? [14:26:23] yeah like knowing them. [14:26:31] I know 10 dutch glams I could add to the tool [14:26:38] but I think we should look to a non-dutch one [14:27:05] the dutch ones I'll be adding somewhere along the project [14:27:15] as far as India is concerned [14:27:22] there's one tool BUB [14:27:30] made by one GSoC intern [14:27:38] do you have a link? [14:27:53] there he uses stuff from West Bengal State Library [14:27:57] yeah [14:28:30] http://tools.wmflabs.org/bub/ [14:28:48] It's for uploading books [14:29:38] aah that looks a bit like what we're doing but then for books [14:30:23] yeah [14:33:18] I should be able give you more info shortly about glams in india. [14:35:32] https://en.wikipedia.org/wiki/National_Digital_Library_of_India [14:35:36] but that's also mainly books [14:35:42] https://openglam.org/ [14:37:58] I will contact some wikimedians who are involved in various wiki projects in India soon and let you know. [14:38:19] I saw on the web that some people are involved. [14:38:27] openglams seems to be a bit switzerland heavy [14:38:44] but for now just picking a glam somewhere in a english speaking country will do fine I think [14:39:11] if we find a glam there's still some questions: is the collection mappable, is it not already uploadid, is it large enough [14:39:58] ok [14:41:12] Is there anything I could do in this regard? [14:42:11] yep, just look around and think what is interesting and discuss those [14:42:25] but it's not easy to do I think, it's also not that easy for me [14:42:47] alright. [14:46:56] maybe the easiest is if for now you focus on getting the steps (UI, code restructure, handling duplicate uploads, the steps a user takes in the process) done [14:47:01] and I could add a new GLAM [14:47:10] you already know a bit how the nationaal archive works [14:47:30] sure that sounds good. [14:47:58] I'll try to work on that a bit tomorrow, and then I can also directly make some suggestions for code restructures [14:48:18] for the UI/ user steps, do you have an idea on that? [14:48:36] would it be a good idea if you sketch that (describe it in words) so that I can comment on those steps [14:48:36] Not much. [14:48:43] ok so I'd have to sketch it [14:48:57] well somebody arrives at the tool [14:49:12] I liked the different tabs from BUB with some general info [14:49:20] we show a list of glams to select from [14:49:34] we can provide a small story for each glam [14:49:39] (link to their website [14:49:44] what the collection contains [14:50:02] and what they have to copy into the next field: the identifier/url [14:50:14] then they press continue [14:50:30] and we can ask for example for categories for the images [14:50:40] they provide those and continue [14:51:02] well continue then is a upload button [14:51:11] and the images get uploaded [14:51:21] after that they get a page which shows either failure (and reason) [14:51:26] or link to the image(s) [14:52:12] how about doing the next upload? [14:53:19] user clicks on the home button? [14:53:37] to go back to the home page and do the next uplaod. [14:53:50] yes, but when we want to support multiple uploads maybe we should allow them to save settings [14:53:55] naah that's complicated [14:54:27] ehm yes just home [14:54:39] well for multiple uploads using search (later on in the project) [14:54:57] we should allow in these steps simply that we can have a list of some images where actions are done [14:55:39] like having checkboxes. [14:56:51] Ok I will have a UI built for single image upload by this week. [14:57:21] nice, one of the important things to keep in mind is that we should be able to communicate some things with the uploader [14:57:55] the: you have to add categories, or in another case: you have to check whether the file is taken before 1945 [14:58:39] Will add categories be a must filed? [14:58:45] *field [14:59:24] not in this case [14:59:35] but that's something we should be able to have some flexibility with [15:00:30] ok [15:02:44] Well I suppose that's it for today. [15:02:50] Is there anything else? [15:05:15] naah that's it, I'll try to be here a bit more [15:05:31] ok great. [15:05:37] but if you're stuck on anything please post it on the relevant phabricator tasks [15:05:50] alright.