[04:41:23] hi zhuyifei1999_ good morning [04:42:21] hi [04:42:47] I happen to handle the duplicate file exists situation here https://github.com/infobliss/sibutest2/blob/master/app.py#L81 [04:42:50] * zhuyifei1999_ will be away in a while, will be back around 4/5 pm [04:42:56] ok [04:43:10] yeah? [04:43:13] Is that the correct way to do? [04:43:43] it could be [04:43:52] at least it's better than the previous one [04:44:31] I wonder how I can do the same here for multi-upload https://github.com/infobliss/sibutest2/blob/master/app.py#L118 [04:44:56] here I cannot return or raise since there other files to be uploaded in the loop [04:45:17] well, one way is to keep a list of fails and report to user once the upload is done [04:45:44] or you can get more sophisticated and wait for user input before continuing [04:46:06] user input is at one go [04:46:38] that is he chooses all the images at once [04:46:46] well, you can return early and store the state in session [04:47:17] 4/5 pm your time? [04:47:24] yeah [04:47:29] ok [04:47:47] I do not understand this returning thing properly [04:47:53] let me be specific [04:48:08] Suppose I return when there is a duplicate file found [04:48:31] Even after that I wish that the loop continues executing [04:48:43] how is that done? [04:48:59] yeah, you can store whatever is needed to reconstruct the loop in session [04:49:32] and resume the loop? [04:49:48] kind of [04:50:25] how to resume execution after an exception is raised? [04:50:36] except: [04:51:32] except block is inside the loop [04:51:49] and the same loop has to be continued [04:51:59] sort of like goto statements in C [04:51:59] you reconstruct a new loop [04:52:20] goto is evil. don't use it [04:52:42] makes all sorts of unexpected behaviors [04:52:48] ok [04:53:21] about reconstructing the loop: there will be except block in each loop [04:53:33] aren't there going to be too many repetitions [04:53:34] ? [04:54:00] then make a shared function [04:54:22] oh right ok [04:54:47] one more thing before you go [04:55:03] you wanted me to build fullglamlist.py dynamically [04:55:17] yes [04:55:29] https://github.com/infobliss/sibutest2/blob/master/glamFullList.py [04:55:32] this one [04:55:51] yes? [04:55:57] oh one more thing [04:56:15] basvb was suggesting if this one can be made a config file instead of .py? [04:56:33] but this needs to have import statements [04:56:47] well sure if it must be static. but imo it should be dynamic [04:57:21] yeah got you. Please give me some pointers on how I can make it dynamic [04:57:34] youtube-dl [04:58:27] https://github.com/toolforge/embeddeddata/tree/master/detection/by_magic zalso use somewhat dynamic reregistration [04:59:03] anyways, what you want is a list of classes. everything else can be done by fetching attributes from the classes [04:59:35] getting name/urlPrefix can be as glamclass.name [04:59:50] agreed [05:00:21] so I have to be able to check the glam folder and read the filenames from there [05:00:25] Is that right? [05:00:37] that's one way [05:01:31] Is ok to do? [05:01:41] https://stackoverflow.com/questions/3365740/how-to-import-all-submodules [05:01:42] *Is it ok to do? [05:02:14] google is your friend ^ see my link [05:02:33] looking [05:04:18] looks good [05:04:27] k [05:06:14] ok nothing else for now [05:06:16] thanks a lot [09:22:34] back [09:44:16] alright [09:44:40] I am re-writing app.py [09:45:06] will inform you once done [11:47:54] hi zhuyifei1999_ [11:48:16] I made app.py a lot cleaner https://github.com/infobliss/sibutest2/blob/master/app.py [11:48:17] yeah? [11:48:23] looking [11:49:18] I still don't see cleanups for https://github.com/infobliss/sibutest2/blob/master/app.py#L50 [11:49:43] oh yes that part I wanted to sak you [11:49:44] and they aren't applied for multiuploads [11:49:50] *ask [11:50:32] https://github.com/infobliss/sibutest2/blob/master/app.py#L73 I like this :) [11:50:46] https://github.com/infobliss/sibutest2/blob/master/app.py#L71 indent [11:51:23] https://github.com/infobliss/sibutest2/blob/master/app.py#L63 if searchstring elif identifier else [11:52:42] https://github.com/infobliss/sibutest2/blob/master/libraries/utils.py#L119 none will break [11:52:50] just raise an exception [11:55:33] sure [11:56:49] https://github.com/toolforge/video2commons/blob/master/video2commons/backend/worker.py#L172 [11:56:58] will this suffice? [11:57:24] maybe [11:57:38] depends on how you code it [11:59:14] Is it necessary to configure Pywikibot separately for multi-upload as well? [11:59:30] why not? [11:59:48] remember: keep ststes only in sessions [11:59:52] *states [11:59:58] login is a state [12:00:14] I thought that once the user logs in we have the pywikibot configured [12:00:19] and that's it [12:00:22] you don't [12:00:37] configure immediately before upload [12:00:48] and unconfigure immediately after upload [12:01:03] whether success or not (finally:) [12:01:15] for each upload? [12:02:08] well, I don't think flask/uwsgi on toolforge uses multithreading, so for each request is okay [12:02:42] if with multithreading you have to use locks + for each upload [12:03:18] i.e., configure pywikibot once for each multi-upload step? [12:03:20] iirc flask/uwsgi on toolforge default configuration use multiple processes [12:03:24] ? [12:03:36] what steps? [12:03:40] before each multi-upload [12:03:54] ??? [12:04:13] https://github.com/infobliss/sibutest2/blob/master/app.py#L106 configure here? [12:04:37] sure [12:04:46] ok [12:04:51] and remember to unconfigure [12:05:00] yes [12:05:29] well https://stackoverflow.com/questions/3365740/how-to-import-all-submodules I gave this some thought [12:05:41] but this looks very much complicated to me [12:05:44] ... [12:05:57] you can do this :P [12:06:16] I can do what? [12:06:37] it's not that complicated [12:06:53] oh and don't exec('%s = module' % module_name) [12:07:00] just skip that line [12:07:03] I don't seem to understand where will it get called [12:07:09] oh ok [12:07:16] and use a callback-style registration [12:07:29] whereever you want to load that module from [12:07:57] what is callback-style registration ? [12:08:05] skip __all__.append(module_name) & __all__ = [] [12:08:21] https://github.com/toolforge/embeddeddata/tree/master/detection/by_magic [12:08:32] or decorator style [12:09:03] https://github.com/toolforge/embeddeddata/blob/master/detection/by_magic/__init__.py#L59 [12:09:23] you can remove that outer wrapper if you want [12:10:18] what is a decorator here? [12:10:33] something like: [12:10:33] def register_glam(c): [12:10:33] glamlist.append(c) [12:10:33] return c [12:10:35] ? [12:10:42] https://github.com/toolforge/embeddeddata/blob/master/detection/by_magic/rar.py#L26 [12:11:03] that's how that repo registers [12:11:39] basically it register on-import [12:11:56] and all you have to do is import all the classes [12:12:19] uh [12:12:30] def register_glam(c): [12:12:31] glamlist.append(c) [12:12:31] return c [12:12:51] my copy pasting keep messing with the indents :/ [12:13:15] np that was understood [12:14:29] so module = loader.find_module(module_name).load_module(module_name) will give the classes [12:14:46] yeah [12:15:11] and following that I will call register_glam(c) for each class? [12:15:28] it actually also sets the module variable [12:15:29] ? [12:15:38] but I will be already having a list of the classes [12:15:50] see https://github.com/toolforge/embeddeddata/blob/master/detection/by_magic/rar.py#L26 [12:16:12] the list population should happen on import [12:18:12] after doing this how will I get a .py file like https://github.com/infobliss/sibutest2/blob/master/glamFullList.py? [12:18:25] ??? [12:18:34] why would you want a file? [12:19:05] sorry I am not sure if I have internalised this fully [12:19:41] I was thinking that we will have a file built dynamically [12:19:55] no [12:20:02] you build a variable [12:20:09] not a file [12:21:07] do we have a separate util function for this? [12:21:42] outside of app.py and call that from app.py [12:25:03] sure [12:25:14] ok [12:25:26] I do think it should be outside of utils.py though [12:25:45] because all the glams has to import that loader [12:26:16] ok [12:26:29] https://github.com/infobliss/sibutest2/blob/master/app.py#L8 L9 will be redundant [12:26:52] yes [12:27:49] I will try to write it [12:28:12] one more thing you had said to use css-flexbox for image gallery [12:28:37] yes [12:28:39] I have used plain css and doesn't look bad to me [12:28:45] k [12:28:47] you may take a look at it [12:28:49] a sec [12:28:58] let me send a screenshot [12:31:39] https://ibb.co/f6TpxF [12:32:11] ... [12:32:39] I mean, my point of having a flexbox is to have a fixed width/height and not screw up with the aspect ratio [12:33:13] here we have fixed height and no deformation of image [12:33:22] width is adjusted with height [12:33:43] * zhuyifei1999_ thinks variable width impairs usability [12:34:13] if you just want to use fixed height variable width you don't need flexbox [12:34:18] how about variable height? [12:34:24] it looks even poorer [12:34:35] variable height is even worse [12:34:41] so just set them both fixed [12:35:02] the point of flexbox is not mess with aspect ratio when both are fixed [12:35:24] Ok I got you. [12:36:27] ok I will be offline for a few hours [12:36:35] will resume after dinner [12:36:47] * zhuyifei1999_ will be offline in an hour [12:37:11] alright [12:44:44] zhuyifei1999_: I do think it should be outside of utils.py though [12:45:01] you said "do think" right? [12:55:12] yeah [12:55:23] in a seperate file [18:00:13] hi [18:01:53] hi basvb [18:05:04] how is it going? [18:06:03] https://codeshare.io/aYW14n [18:06:15] you may test in tool labs [18:09:55] on what toollabs domain is it? [18:10:08] https://tools.wmflabs.org/sibutest ? [18:11:03] yes [18:12:00] did you face any issues? [18:12:10] I think it's uploading now [18:12:31] I found one after uploading [18:12:38] then the end page I get: "The selected images are now available in Wikimedia Commons! Duplicates are ignored! Find the images in your Uploads Page in Wikimedia Commons here Duplicate files found in Commons (if any) are listed below: User must login in this site User must login in this site User must login in this site" [18:12:56] the last 3 are an issue [18:13:05] that is strange [18:13:17] could you send me a screetshot? [18:13:21] also I do not see them in my uploads [18:14:20] I think pywikibot login related thing [18:14:47] you must have selected 3 files [18:14:49] right? [18:15:24] https://ibb.co/mTzMCF [18:15:33] I selected 3 files yes [18:16:27] let me try once more [18:16:37] btw, maybe you could show the logout button at every page? [18:17:06] also I think check-boxes below the images is nicer [18:17:35] https://github.com/infobliss/sibutest2/blob/master/glams/NationaalArchiefGLAM.py#L203 [18:17:36] two comments [18:17:50] '&count=' should be in the string itself [18:18:00] and yesterday I asked whether you could make the number a parameter [18:18:05] for both glams [18:18:09] that's like 3 minutes of work [18:18:30] def search_to_identifiers(cls, searchterm, number_of_files=100): [18:19:07] sorry [18:19:15] no problems [18:19:22] I will do it now [18:19:23] this time the upload works [18:19:35] I think the issue was the previous time I used the back button in the browser [18:19:55] this is because I have have to get pywikibot login to commons per upload [18:20:11] oh is it? [18:20:36] the category didn't work [18:20:52] I'm pretty sure I added "Cees Varkevisser" as category for my 3 most recent uploads [18:20:55] https://commons.wikimedia.org/wiki/File:Persoonlijk_damkampioenschap_te_Apeldoorn,_partij_Van_Dijk_(links)_en_Varkevisser_-_NA_-_917-6679.jpg [18:21:34] likely NA specific [18:22:46] category should have worked [18:22:50] let me check [18:22:59] I added it by hand now [18:23:01] so check the history [18:23:40] "Uploaded from Nationaal Archief with g2c tool" [18:23:45] can you write out glam2commons [18:23:49] and remove tool [18:24:05] oh sorry [18:24:18] the category was not sent to the function for multi upload [18:24:24] I will fix that now [18:24:32] good [18:25:50] are you testing? [18:25:57] I want to restart the server [18:27:14] which will make the server unreachable for a minute [18:32:32] sorry disconnected [18:32:39] I tested with one multi-omage upload [18:32:52] you can restart [18:32:54] ok I made changes to add category in the tool labs server [18:33:00] Already restarted [18:33:58] is it ready for user testing? [18:34:06] or do you want to make improvements 1-2 more days? [18:34:39] It will work all fine [18:34:47] for the user [18:35:11] I requested them to approve OAuth consumer for the new tool glam2commons [18:35:20] ok [18:35:28] so we wait on that before we let users test [18:35:31] what we can do is let users test glam2commons once it is approved [18:35:37] maybe you can discuss that tomorrow with zhuyifei [18:35:48] maybe he can move it along a bit by pinging the persons [18:35:59] yeah [18:36:45] Today I had a discussion with zhuyifei [18:37:08] regarding dynamically loading the GLAM class imports wherever they are statically imported [18:37:46] which will be making adding new GLAMs even easier [18:38:18] Also regarding https://github.com/infobliss/sibutest2/blob/master/glamFullList.py [18:38:42] he said it could be made a json only if it is to be made static [18:38:58] so this is optimal? [18:38:59] but then he said dynamic is prefereed [18:39:10] I read he wants everything in the class (names etc.) [18:39:46] not only that those parameters are already shifted to the class [18:40:18] here https://github.com/infobliss/sibutest2/blob/master/glams/NationaalArchiefGLAM.py#L25 [18:40:26] yep [18:40:33] please also add these to genericglam [18:40:38] and https://github.com/infobliss/sibutest2/blob/master/glams/AmsterdamMuseumGLAM.py#L20 [18:40:46] yeah right I will [18:41:04] what are you using url_prefix for? [18:41:22] you mean where it is used [18:41:43] it will be used as a href to the images in the image gallery [18:41:58] I mean it's already thay way [18:42:17] *that [18:42:43] aah ok [18:42:52] and where are the help-texts from the html files generated? [18:43:09] "The national archive of the Netherlands, located in The Hague" [18:43:20] right now they are static [18:43:29] I could not get to that yet [18:43:46] gor the other works I was doing regarding dynamically loading imports [18:43:49] that has to be dynamic as well to be really scalable [18:43:58] *for [18:44:18] yeah I will work on it [18:45:41] and the image gallery looks good to me [18:45:58] because I think it has the minimum white space wasted [18:46:45] If I fix the box size then there will be more loss of space [18:47:26] because most images are either wide or high, i.e., not square [18:47:27] we'll see what the users say during the testing [18:47:39] can you formulate some questions you are interested in as well for them? [18:47:57] of course [18:48:09] I will think and let you know [18:51:05] hi I'm disconnecting all the time [18:51:27] do you have enough on your plate for tomorrow? [18:52:03] there are not many [18:52:33] improving scalability is what I strive for [18:53:14] making imports dynamic looks like a challenge to me even though zhuyifei says it's going to be easy [18:53:55] secondly, to remove GLAM-specific texts from the form [18:54:02] yep [18:54:08] so that's not enough? [18:54:17] well making the questions and documentation has to happen [18:54:48] Is there any specific place where I have to put up the questions? [18:55:55] no, we'll discuss them [18:56:06] and I'll write in the village pump [18:56:23] so I will just put here for you [18:56:39] yep [18:56:48] I'm going to do some other things [18:56:52] has been a long day again [18:56:54] alright [18:57:10] for me too [18:57:29] we will catch up tommorrow [18:57:47] yep see you tomorrow [20:09:27] Questions for the users who will test: [20:10:41] 1. Are the instruction messages throughout the app (i.e., the home page, the two forms) easy to understand? Do they need to be clearer? [20:14:15] 2. Do they face any issues during multiple image upload? Especially I feeln that for uploading more than 3 images the waiting time is quite high. [20:14:46] 3. In general what do they think about the look and feel of the app? [20:55:21] 4. Do they have any suggestion about the wikitext we are generating for NA and AM GLAMs?