[00:26:06] does anyone here use AWB? [00:26:34] try #autowikibrowser [00:26:40] legoktm: I'm in there but not very active [00:27:12] legoktm: do you use it? [00:27:25] no [00:27:40] but you'd probably get a better response if you asked what your question actually is [00:28:04] Well I'm using AWB on a Mac using CrossOver and it's working fine [00:28:06] but the problem is: [00:28:55] I'm fixing typos using regex, I'm using a database dump. When it finds an error it stops processing the list of articles, and I'd like the list of articles to run in the background [00:29:08] so I can go AFK and come back and not have to wait 1 min for it to find a typo [00:29:14] Don't start the list maker via make list [00:29:22] *db scanner [00:29:28] I don't know how to do that [00:29:41] Give me more than 6 seconds to respond? [00:29:49] Oh sorry.. [00:31:08] Pressing stop in the main AWB window (not the scanner) should stop it processing [00:31:22] so, is https://commons.wikimedia.org/wiki/User:Odder/Sandbox a bugable offence, or just me being silly? [00:31:32] Reedy: But I don't want it to stop processing [00:31:40] I want it to keep processing even when it finds an error [00:31:51] parser functions behaving differently on translation pages than they do on normal ones [00:32:07] Options -> Use pre parse mode? [00:32:18] Reedy: I don't see that [00:32:34] oh found it [00:32:54] Reedy: What's that do? [00:32:57] Reedy: okay, when do I set that? [00:33:48] where* [00:33:58] twkozlowski: I guess it has to do with formatnum using a different language so #expr is parsing it wrong [00:34:06] twkozlowski: What? [00:34:22] Newyorkadam: Why not try it? [00:34:27] Reedy: I did [00:34:30] I just wanna know what it does [00:34:55] pre-parses articles [00:34:57] the name is in the title [00:35:14] Reedy: oh, you were talking to Newyorkadam :) [00:35:26] Reedy: I don't get how some people make changes on AWB like boom boom boom 30 changes every minute [00:35:31] and for me it's one a minute [00:35:37] oh they're using different plugins than I.. [00:35:48] Reedy: I shamelessly assumed you found the answer to my problem :) [00:36:51] Auto save [00:36:56] Better skipping [00:37:00] Faster connection/computer [00:37:01] Reedy: A few days ago someone asked me if I could ask someone to run the January database dump against the TypoScan on AWB [00:37:03] How do I do that? [00:37:09] It'll take weeks [00:37:44] Reedy: Really? [00:37:47] Yes [00:37:56] Reedy: I see you're the project leader of TypoScan [00:38:01] "project leader" [00:38:10] Well "WikiProject Lead" [00:38:12] Is what it says.. [00:38:36] It's been inactive for nearly 2 years [00:39:43] Reedy: I'm running a typo fixing contest [00:39:47] starts in 4 days [00:39:56] No chance [00:41:05] Reedy: Even a small dump? [00:41:14] Small being what? [00:41:24] idk something that can be done in less than 4 days [00:41:58] You've then got the additional problem that said dump doesn't exist [00:43:10] Reedy: What? [00:43:14] I just downloaded a dump.. [00:43:26] What file? [00:43:58] enwiki-20140102-pages-meta-current1.xml-p000000010p000010000 [00:45:29] so it contains any pages with a page id of 10 to 10,000? [00:45:49] Reedy: I don't know? [00:46:10] At 48.8M it's not going to contain much [00:46:19] Reedy: I know, I downloaded that in a few minutes [00:46:26] I'm saying something that will take four days to do [00:46:30] Maybe even 10 is ok [00:46:32] Just something [00:46:55] Well, all of those articles aren't going to have typos [00:47:09] Some of those that do, won't actually have typos when someone tries to fix it [00:47:18] How many people are participating? [00:48:25] The reason I never bothered (well, I did, and got annoyed by it) processing another dump [00:48:39] Even on a hex core i7, 24GB ram and on an SSD, most of the machine was sat idle [00:48:53] Somewhat of a waste leaving on the machine on to do very little [00:50:49] Reedy: You can use the rest of that to compile mw now! >.> [00:51:18] I don't think hiphop is that slow [00:51:39] Newyorkadam: For reference, against the last run, only 40% of pages at time of re-processing actually had typos that were fixed [00:52:09] Reedy: 9-10 people are participating so far [00:52:20] more coming probably, we have a signpost article about it [01:06:28] Reedy: When I'm on 'Random pages' and select 'Make list' it still filters my watchlist? [01:06:37] filter? [01:06:48] What do you mean? [01:06:57] What do you mean? [01:07:02] what's filter? [01:07:09] Oh yeah [01:07:13] Filtering out all talk pages [01:07:19] but i'm keeping all content pages [01:07:32] Oh I had to apply the filter [01:07:39] which I thought I did [01:08:04] I don't believe that it's gone through 100 pages without finding a single typo.. [01:08:51] It's quite possible [01:09:09] why is it finding so many pages I've editied.. [01:09:09] Newyorkadam: Open 100 browser tabs and try? [01:09:16] Gloria: What? [01:09:48] It lists so many articles I've edited.. [01:09:51] I thought you were sampling random pages. [01:10:03] Gloria: I am [01:10:08] but it's supposed to be random pages [01:10:10] not articles I've edited [01:12:23] Reedy: They've solved that dump-scanning problem, by the way. [01:12:34] You can do it in seconds now. [01:14:53] Reedy: ^ Does that mean it can be done? [01:18:22] Gloria: Can you help me with some AWB stuff please? [01:19:05] You can ask on [[m:Tech]] if you'd like. [01:19:14] I don't wanna wait :/ [01:19:36] Learn more. :-) [01:22:17] Newyorkadam: Ask in here. [01:22:24] Or #autowikibrowser. [01:22:27] And be patient. [01:22:49] I'm trying to find everything that says 'in January 1' to 'on January 1' [01:22:57] In options I set the find and replace [01:22:59] right? [01:23:07] Okay. [01:23:08] And then? [01:23:15] Then what.. [01:23:20] How do I search for that in articles? [01:23:23] Do you see that in any article? [01:23:30] Gloria: I know they exist around Wikipedia [01:23:31] You can use regular search for this. [01:23:35] Find one. :-) [01:23:37] Gloria: But AWB makes it automated [01:23:46] this guy does it [01:23:48] https://en.wikipedia.org/wiki/User:John_of_Reading/Typo_fixing_with_AutoWikiBrowser#In_MONTH_DAY_.3E_On_MONTH_DAY [01:23:59] He found like 13,000 of 'em [01:24:07] I don't wanna do 13,000 manually.. [01:24:15] What about "in January 1990"? [01:24:22] Do you want to find those? [01:24:24] Gloria: Yeah I don't know how to ignore [01:24:25] No [01:24:29] But John of Reading did it somehow [01:24:40] You should learn about regular expressions. [01:24:40] that's the error that the search keeps getting [01:24:44] I know some regex [01:24:53] Okay. [01:25:00] So you'd probably want to use regular expressions here. [01:25:12] But how do I use regex to -ignore- certain things using AWB? [01:25:31] by writing a proper regex?:) [01:25:35] [^0-9] probably. But really \b. [01:25:39] Maybe. [01:25:46] You need to limit number matching, I guess. [01:25:51] I only know basic regex [01:26:12] This is basic regex. [01:26:58] https://en.wikipedia.org/w/index.php?title=Special:Search&search=%22in+January+1%22 [01:27:01] like I know things like [01:27:05] It looks like there are about 88 instances on Wikipedia. [01:27:09] "hel(lo)?" [01:27:15] would find hel and hello [01:27:16] You're going to scan a whole dump for 88 instances? :-) [01:27:25] Yes. [01:27:26] Gloria: that's JUST January 1st [01:27:30] think about January 2 [01:27:33] and January 3 [01:27:33] and so on [01:27:36] Ah, right. [01:27:37] then February.. [01:27:52] Heh. [01:28:15] Someone else was asking me to scan a dump for them. [01:28:18] I could throw this in there. [01:28:25] Gloria: If I can find out how.. [01:28:32] Gloria: Can you also throw in TypoScan? [01:28:36] I'd just write a small script. [01:28:38] I'm running a typo-fixing contest in a few days [01:28:50] So I heard. [01:28:56] No, I'm not scanning for typos. [01:29:02] wait Gloria I'll find send you some false positives to watch out for in your script [01:29:04] k that's ok [01:29:13] want it in PM? [01:29:15] It's like 10 lines [01:29:20] Try a pastebin. [01:29:25] http://p.defau.lt [01:29:33] http://pastebin.com/bzPQ1f1d [01:29:51] http://pastebin.com/raw.php?i=bzPQ1f1d [01:30:01] If you insist on using pastebin.com. [01:30:25] oh sorry [01:30:30] I did it before you sent the message [01:30:57] Gloria: someone said it takes 4 days to scan. [01:31:42] What is time? [01:32:00] Newyorkadam: Do you think four days is a lot of time to scan a dump? [01:32:05] Gloria: Yes [01:32:14] Will you have scanned in a dump four days from now? :-) [01:32:24] Minus an in. [01:32:32] Typos, always killin' my game. [01:32:47] Gloria: No I'm just asking :p [01:33:03] You want to scan the English Wikipedia? [01:33:12] Do you have a dump? [01:33:29] Gloria: Not one specifically in mind [01:33:32] but yes en [01:33:38] Did you download it? [01:33:48] Gloria: Yes but a small version [01:33:50] only 40 mb [01:33:57] there's ones that're like 5 gbs [01:34:08] I think you downloaded a piece of a dump. [01:34:12] You should download the rest. [01:34:26] It's somewhere around 19G I think. [01:34:30] Gloria: I heard 40 GB [01:34:51] I think downloading alone could take you that long. [01:35:04] that long --> four days [01:35:16] Gloria: Would you mind sending me the regex so I can look at it :) ? [01:35:19] I wanna learn from it [01:35:22] when you're done [01:35:35] I don't think I want to match on the patterns you provided. [01:35:50] Did John of Reading publish his code? [01:36:00] Gloria: I don't see it anywhere [01:36:05] I don't know if he used regex.. [01:36:23] he said he used Google searches [01:36:38] enwiki-20140102-pages-articles-multistream.xml.bz2 is 10.5GB [01:36:46] What's multistream? [01:36:50] I'll see how much it extracts to... [01:37:06] I think it's done concurrently [01:37:09] I usually use the bigger one. [01:37:10] Then concatanated at the end [01:37:11] Because I want all pages. [01:37:50] > enwiki-20140102-pages-meta-current.xml.bz2 19.1 GB [01:37:53] That one. [01:37:57] It takes about 30 minutes to download. [01:38:07] To Labs, which probably already has a copy. [01:38:20] Gloria: If you tell me what to do with the regex I can probably do it [01:38:28] Wait [01:38:31] I think I can do it [01:38:53] Did you download the dump? [01:39:03] You can't scan a dump without a dump. [01:39:41] Seems to be about an hour to the office [01:41:07] Gloria: I DL'd a small dump [01:41:24] Gloria: Why doesn't this detect "on January 1" "on January 2" and "on January 3"? [01:41:27] "on January (1,2,3)?" [01:41:36] It only detects 'on January' [01:42:06] nvm got it [01:42:10] I need to replace the commas with | [01:43:11] Gloria: Here's all of January [01:43:14] on January (1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31)? [01:43:16] on January \d <-- better [01:43:30] on January \d\d? is even better I guess [01:43:44] No [01:43:46] Oh, yes [01:43:55] \d wouldn't work ;) [01:45:23] oh \d\d? is to 99 right? [01:45:24] that's cool [01:46:09] this is the final code I believe: [01:46:10] [1-2]?[0-9]|3[0-1] [01:46:10] on (January|February|March|April|May|June|July|August|September|October|November|December)? \d\d? [01:46:12] or January [123]?\d [01:46:30] still incorrect for some values [01:46:31] 31st Februrary [01:46:37] (e.g. accepts 0, feb 30th etc.) [01:46:59] paravoid: But then we can find articles that have Feb 30 [01:47:05] paravoid: It also includes January 99 ... [01:47:17] no it doesn't [01:47:28] paravoid: Why not? [01:48:45] \d\d? could include 99 [01:49:24] Gloria: So whadoo I do now? [01:52:33] right now I'm scanning for 'The the' (I think) [01:53:35] Reedy: Can you help me to make sure I scanned correctly? [01:55:44] or Gloria? [02:04:16] Newyorkadam: Sounds like you've got it under control. [02:04:28] Gloria: No it isn't working :/ [02:04:30] I don't know why [02:04:39] It's possible that it just found 0 errors but I doubt it [02:05:51] Newyorkadam: How big is the file you downloaded? [02:05:59] Gloria: Like 40 MB [02:06:04] I think a couple thousand articles [02:06:13] Probably the most scanned articles. [02:06:21] Given people like you. :-) [02:06:43] I have a find and replace for 'the the' [02:06:49] How's it going? [02:06:50] but it keeps detecting 'the' and removing it entirely.. [02:06:58] Ouch. [02:07:20] any ideas? [02:07:46] You're doing it wrong? [02:07:51] Well apparently [02:07:53] What are you doing specifically? [02:07:55] But I can't find out the errors [02:07:56] I can't see your screen. [02:08:00] I go to the 'Options' tab [02:08:03] I go to 'Find and replace [02:08:05] ' [02:08:05] You're replacing "the the" with ""? [02:08:11] I'm replacing 'the the' with 'the' [02:08:14] Well I'm supposed to be.. [02:08:20] Try a screenshot? :-) [02:08:56] in this screeny you can see that it deletes 'the' to nothing [02:09:04] and that I have 'the' replacing 'the the' [02:09:05] http://gyazo.com/809bbfb60960ee3cb5279a28f91996aa [02:09:55] Heh. [02:10:05] "the themes" is "the the" ;-) [02:10:06] Any idea? [02:10:10] oh.. [02:10:18] I love screenshots. [02:10:22] so I should add a space after the second 'the' [02:10:30] Or a word boundary. [02:10:31] Like \b. [02:10:35] But that's regex. [02:11:57] We should make this process a whole lot easier. [02:12:11] As I said, you can do this in a few seconds with the appropriate equipment. [02:12:20] Gloria: I don't know how.. [02:12:22] But we're not quite there yet. [02:12:38] afk for 1 min [02:12:54] I don't believe that it's skipped so many articles and hasn't found a single 'the the' [02:12:57] I guess that's good [02:38:01] Newyorkadam: if you want I can run a better scan faster than AWB can [02:39:44] Betacommand: how? [02:40:02] Newyorkadam: I have a python script [02:40:16] Betacommand: Can I use it ^.^? [02:40:21] Newyorkadam: It processes about 5k pages at once [02:40:28] wow [02:40:37] Why don't you publish it? [02:40:38] Newyorkadam: let me know what you want scanned for and I can create a report [02:41:03] "at once" [02:41:08] Newyorkadam: I havent gotten around to making it stand alone [02:41:17] Betacommand: Can you send it to me? [02:41:32] Newyorkadam: its not something that can be sent [02:41:49] Betacommand: oh [02:41:51] Newyorkadam: its tied to a lot of non-relevant code [02:42:07] Ive just been too lazy to make it stand alone [02:42:14] Betacommand: Well can you :D? [02:42:18] If you want [02:42:43] Betacommand: is it fully automatic or do you check each edit? [02:42:46] Newyorkadam: if you tell me what you want to look for I can just run a report for you until then [02:42:53] Newyorkadam: its a dump scanner [02:43:03] Betacommand: "the the" to "the" [02:43:14] Not capsensitive [02:43:17] it just finds the error [02:43:38] ok [02:44:15] Newyorkadam: give me a few minutes [02:44:22] Betacommand: k :) [02:44:29] there's a lot of false positives [02:44:44] Newyorkadam: there shouldnt be that many of them [02:45:02] Betacommand: yeah [02:45:05] but at least some [02:50:49] Newyorkadam: I tweaked the scan to only match \bthe the\b [02:51:42] Betacommand: What does that do? [02:51:44] the /b [02:51:47] \b [02:52:04] \b is a word boundary [02:52:40] Newyorkadam: http://tools.wmflabs.org/betacommand-dev/reports/db_scanner_the_the.log [02:52:48] the list is still updating [02:52:59] those are all what's been scanned, or articles with 'the the'? [02:53:12] Newyorkadam: its not 100% current but it uses the last database dump [02:53:24] Newyorkadam: just the articles with the problem [02:53:30] I'll check one [02:53:56] wow it's right :D [02:53:58] Newyorkadam: for example http://en.wikipedia.org/w/index.php?title=Animal_%28disambiguation%29&diff=592883869&oldid=582620286 was already fixed [02:54:02] Is there a way to make it automated? [02:54:10] Newyorkadam: define automated? [02:54:17] make it auto-correct all of those [02:54:23] HELL NO [02:54:41] :( [02:54:44] What dump is that? [02:55:14] Newyorkadam: http://en.wikipedia.org/wiki/Wikipedia:Bots/Frequently_denied_bots#Fully_automatic_spell-checking_bots [02:55:38] Newyorkadam: its using the 20140102 dump [02:55:39] Betacommand: I don't mean a bot [02:55:42] it'd be like AWB [02:56:03] Newyorkadam: take that list save it to a file, and load it with AWB [02:56:31] I'm confused? [02:56:38] Do I save it as .xml [02:56:45] No [02:56:56] save the .log as a txt file [02:57:04] open AWB [02:57:44] tell me when it's done downloading [02:57:48] See the area where it says "Make list" ? [02:57:58] 1 sec [02:58:00] Newyorkadam: it will be several hours [02:58:00] is it down running? [02:58:02] oh [02:58:06] I'll just do what we have [02:58:32] yeah I see make list [02:58:42] Newyorkadam: its parsing 4.4 million pages [02:58:47] Betacommand: oh [02:58:53] Betacommand: Where do I select log file? [02:59:07] text file? [02:59:10] source > text file [02:59:30] then make list [02:59:46] select the file you saved, and have at it [03:00:26] oh it worked :O [03:00:27] thanks [03:00:54] Newyorkadam: if you go to http://tools.wmflabs.org/?status and look for 2349066 you can see if the bot is still parsing [03:03:11] Newyorkadam: the file that the script is using is 19.1GB [03:03:47] so that should give you an idea of how much data its parsing [03:03:52] and thats compressed [03:05:07] Newyorkadam: Ive got tools for just about anything, If you need something feel free to drop me a line [03:05:40] Betacommand: This is amazing thanks :D :D [03:06:04] Newyorkadam: the script is far far faster than any searching that AWB can do [03:06:21] yeah it's really nice :) [03:06:25] and its zero stress on the servers ton find those [03:06:28] check out my contribs you'll see it's working well [03:07:00] Newyorkadam: the next dump should be sunday or so [03:07:17] we get about one a month [03:07:28] Betacommand: :) [03:08:06] Newyorkadam: if you want me to search for something else feel free to drop me an email or PM [03:08:16] Betacommand: k :) [03:09:05] I wish AWB were faster [03:09:06] but whatever [03:09:13] I could get done like 10 per minute then [03:09:18] instead I get three tops per minute [03:10:08] Betacommand: This is doing EVERY article? [03:10:17] Newyorkadam: yes [03:10:19] damn [03:10:33] Newyorkadam: its no big deal [03:10:48] Betacommand: It's AMAZING :) [03:11:23] Newyorkadam: this is just one of my many toys Ive got sitting around [03:11:32]