[03:27:38] CKoerner_WMF: I'm trying to use space.wmflabs.org, but I get stuck in an endless loop when I try to sign up / log in. It keeps flashing a name/username modal and then kicking me back to Phabricator to authenticate again. [03:28:14] oh, I see... https://phabricator.wikimedia.org/T226545 [03:28:20] charming. [09:12:13] ragesoss: i had the same.. and then yesterday suddenly it worked.. [09:12:16] no clue.. [18:55:53] !log tools.quickcategories deployed 833372331d (preferences – not linked anywhere yet – with experimental OOUI version) [18:55:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [21:14:53] Guys, why I still struggling in running SQL queries in toolforge?. It's been now 2 weeks, and all my SQL queries have stopped, or in the best cases said something like "Lost connection" [21:15:33] Some quries takes at most 2 or 4 seconds to run. But now it take like 1 hour, before the connection lost!. [21:16:18] Someone said before that's the problem is because of huge load in the toolforge. But that was 2 weeks from now. [21:35:05] !help [21:35:06] ASammour: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [21:37:34] ASammour: we have certainly been having some slow query/overloaded database server issues. Sadly no magic fixes have been found yet [21:42:47] bd808: Thanks for reply.The sad thing that mostly all my bot tasks have stopped because of this problem. Since it depends on SQL queries to retrieve the lists. [21:44:26] To your knowledge, is there a temporary alternative to get lists?. Without using SQL?. [21:44:57] Depends what the lists are [21:45:03] You can get some lists from the API [21:47:22] Reedy: Like this dummy list. It just retrieve the articles that not approved, and in the same time not contains {{not-approve}} tag. [21:47:34] approved? [21:47:52] Flagged I mean :) [21:49:04] In some wikis who use FlaggedRevs extension, any page created by non-confirmed user it will become not approved, until some editor approve it. [21:50:02] seems likely that the FlaggedRevs extension provides an Action API endpoint for that... [21:52:11] bd808: I don't think so. Because you will need to check each article to see if it's contains {{not-approved}} tag. This is a very consuming task. [21:53:10] This query for example is used in many small wikis like ar, ur, zh, hi [21:55:33] FR does provide an api to list reviewed/unreviewed pages [21:56:10] using action=query&prop=templates you can get a list of all templates used on a page [21:56:57] looks like you can use action=query&generator=allpages&prop=flagged to get the status of every page too -- https://ar.wikipedia.org/w/api.php?action=help&modules=query%2Bflagged -- if the wiki is small enough this might not be too horrible to try [22:01:24] Thanks for the suggestions. I've been done this before. But you know what was the problem?. When you want to do the opposite operation (Approved article & and contains {{not-approve}} tag). This operation will take very long time to finish. While running an SQL can get the whole data in less than minute. [22:02:18] but only if the db servers are not overloaded :) [22:02:48] bd808: Exactly :) [22:03:45] The whole idea is to retrieve the list directly from the server, and start working only in that list. Instead of looping through all articles, and check the status of each one. [22:05:57] ASammour: is action=query&list=oldreviewedpages part of what you are looking for? -- https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D9%85%D9%84%D8%B9%D8%A8_%D8%A5%D9%8A%D9%87_%D8%A8%D9%8A_%D8%A2%D9%8A#action=query&format=json&list=oldreviewedpages&ornamespace=0 [22:06:25] "Lists pages that have at some point been reviewed, but for which the most recent revision is not reviewed." [22:06:52] related is list=unreviewedpages "Lists pages which have never been reviewed." [22:06:55] you may prefer to first extract the list of all pages embedding {{not-approve}} [22:07:04] hopefully a short list [22:09:43] !log tools.quickcategories deployed 20a47a955b (preferences linked in navbar) [22:09:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [22:10:14] bd808: This API get the pages that been reviewed before. But some anonymous edit the page. So it's edits will become flagged, until some editor approve the edits [22:11:25] right... I guess I'm not following the workflow you are trying to enable with the list. I thought you were trying to find pages that need to be reviewed? [22:14:22] Let me clear it for you. When any anonymous use create a **new** page, it will become not approved. So what we do is tagging these kind of pages with {{not-approve}} tag, to notify the editors to these pages. The idea is to get all pages that created by anonymous user, and not approved yet. So we can run a bot to tag them. [22:15:56] bd808: This can be retrieved by the API, but what if I want to remove {{not-approve}} tag from an approved page?. In this scenario I have to get all lists that approved, then loop over it, and check if it contains {{not-approve}} tag or not. [22:16:58] ASammour: can you give me a concrete wiki that you would do this on? I think I have some ideas of how to use Action API queries to find these pages, but I need to do some testing to be sure [22:18:19] Sure thing. Take for example arwiki: https://ar.wikipedia.org/wiki/%D9%85%D8%AE%D8%B7%D8%B7_%D8%A3%D8%B5%D9%88%D8%A7%D8%AA_%D8%A7%D9%84%D9%82%D9%84%D8%A8 [22:18:59] This article for example is not approved, and as you can see from the history, my bot find it using the SQL query, and tag it [22:22:25] ASammour: did that article have the {{not-approve}} transclusion in it? [22:22:54] Yes. {{not approve}} tag is {{مقالة غير مراجعة}} in arwiki [22:24:35] !log tools.quickcategories deployed b1a75f69dc (remove debug code) [22:25:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [22:29:53] ASammour: this api call will let you iterate over all pages that embed that template -- https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D9%85%D9%84%D8%B9%D8%A8_%D8%A5%D9%8A%D9%87_%D8%A8%D9%8A_%D8%A2%D9%8A#action=query&format=json&prop=info&generator=embeddedin&utf8=1&formatversion=2&geititle=Template%3A%D9%85%D9%82%D8%A7%D9%84%D8%A9%20%D8%BA%D9%8A%D8%B1%20%D9%85%D8%B1%D8%A7%D8%AC%D8%B9%D8%A9 [22:34:03] bd808: Yes. but the next step is that the bot will iterate again over the list. and check if it's approved or not. [22:34:35] So the same time-consuming task come again. [22:34:37] add prop=flagged to get that info [22:35:31] time-consuming is why we write software [22:35:50] so if we can find the right steps then we just need to write the code to do them [22:39:24] bd808: It doesn't show the flagged data when I add it to the API [22:39:34] https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D9%85%D9%84%D8%B9%D8%A8_%D8%A5%D9%8A%D9%87_%D8%A8%D9%8A_%D8%A2%D9%8A#action=query&format=json&prop=info%7Cflagged&continue=geicontinue%7C%7C&generator=embeddedin&utf8=1&formatversion=2&geititle=%D9%82%D8%A7%D9%84%D8%A8%3A%D9%85%D9%82%D8%A7%D9%84%D8%A9%20%D8%BA%D9%8A%D8%B1%20%D9%85%D8%B1%D8%A7%D8%AC%D8%B9%D8%A9&geicontinue=0%7C54533 [22:42:21] hmmm... prop=flagged really doesn't seem to add anything does it. That seems weird. [22:42:27] * bd808 goes looking for source code [22:46:17] should ask protection_level/protection_expiry [22:48:55] yeah, source looks like it should be adding "level". "level_text", and "tags" data to any results which are also found in the flaggedrevs table. [22:51:28] that seems to work on pages that have been reviewed before -- https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D9%85%D9%84%D8%B9%D8%A8_%D8%A5%D9%8A%D9%87_%D8%A8%D9%8A_%D8%A2%D9%8A#action=query&format=json&prop=flagged&continue=geicontinue%7C%7C&generator=reviewedpages&utf8=1&formatversion=2 [22:52:58] and if the page has never been reviewed then it adds no data -- https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D9%85%D9%84%D8%B9%D8%A8_%D8%A5%D9%8A%D9%87_%D8%A8%D9%8A_%D8%A2%D9%8A#action=query&format=json&prop=flagged&continue=geicontinue%7C%7C&generator=unreviewedpages&utf8=1&formatversion=2 [22:54:24] so when iterating using generator=embeddedin (or anything else) if there is no "flagged" key in the response data for a page it has not been reviewed by anyone yet [22:54:31] ASammour: ^ [22:55:30] assuming of course that "prop=flagged" has been added to the action=query options [22:56:48] bd808: I'm here. Just searching for an article that meets the criteria in the output. [23:04:30] bd808: Works like hell!. Thank you very much. I've never know before that API call can JOIN many props in the same time [23:05:11] the Action API is pretty awesome, but it takes a lot of poking around to understand in my experience [23:06:15] Yeah, sure. Specially generator parameter. I didn't hear about it until now :) [23:06:54] So many thanks to you to solve this issue. [23:07:29] * bd808 wonders how many SQL queries against the wiki replicas could be replaced with Action API action=query&generator=... scripts [23:07:43] Numerous [23:09:17] Many many queries :). My quarry profile is full with SQL queries, where I use it in many wikis [23:09:19] https://quarry.wmflabs.org/ASammour [23:10:02] But now I will change only the basic & very important queries [23:11:14] Reedy: Thanks to you also to help solving this problem