[00:27:39] gry: I'm not exactly sure what your question is. Are you looking for advice about installing MediaWiki on a VPS or something more than that? [00:41:07] hello all [00:41:33] o/ tomthirteen. pretty quiet around here right now [00:41:36] can i ask a question, does anyone have experience with ERROR 2013 (HY000) [00:41:58] losing connection [00:42:11] are you getting that after a query runs for a long time? [00:42:25] i've run a query and join on 270 language databases, but it fails with the english database [00:42:40] I guess so. That's how the error is explained online [00:43:21] our database servers kill long running queries. I don't remember if the limit is 10 minutes or 15, but it is something like that [00:43:36] I was querying for a year's time, and I get that error. I've tried to query as little as two days [00:43:46] ' worth but I keep getting that error [00:44:00] It seems to run for nearly and error, and then I get that error [00:44:05] Is there a work around? [00:44:21] your only fix is to figure out a query that completes faster. [00:44:45] there's a tool for checking the explain of how the server will handler the query ... let me find it [00:44:48] Ugh, but it's the English Wikipedia. It's huge [00:47:20] tomthirteen: here's the tool that can tell you what the execution plan for the query is -- https://tools.wmflabs.org/tools-info/optimizer.py [00:48:32] So, it will just tell you how long each part will be, yes? [00:48:35] often the trick to running something on the enwiki db is to find an indexed value that you can use to partition the query. Then run it N times on those smaller chunks [00:48:58] it will show you if indexes are being used which is important [00:49:11] I was hoping to run it on just a smaller time period, but even that is killing it [00:49:22] if you end up with a table scan in there then you need to find a new way usually [00:49:51] if you want to share your sql in a paste I can take a look and see if there is anything obvious [00:50:04] I'm not a wizard with our schemas though [00:50:23] Thanks! It's: [00:50:25] sql enwiki_p -e "select rev_user_text, rev_timestamp, page_id, page_title from page left join revision on page_id = rev_page where rev_timestamp > 20160101000000 and rev_timestamp < 20160103000000 and rev_user_text > 0 and page_namespace = 0" > en_namesake0.txt; [00:50:51] As I've said, making the time period shorter is not helping at all [00:51:12] Perhaps should I loop through the page numbers? [00:51:55] I've used the sql for a year period on all other wikis and it's worked perfectly [00:53:10] hmmm.. the optimizer.py thing seems to be busted :/ [00:53:38] enwiki is massively larger than the other wikis so there is that [00:54:05] yes, i've discovered :) [00:54:42] so you are trying to get a list of all edits with this? [00:54:50] yes sir [00:55:33] https://dumps.wikimedia.org/backup-index.html [00:55:45] ugh that's worse [00:56:03] i've tried working with the datadumps and they are even worse [00:56:06] it seems like somebody else was looking at this a few days ago ... [00:56:18] what do you mean? [00:56:48] * bd808 looks in chat logs [00:58:12] Do you think if I iterate through page id numbers i can do this piecemeal? [00:59:51] yeah, that's probably the right way with a db query. codeofdusk was asking about a similar thing for deleted pages. j.ynus pointed him to the logging dumps and then I think h.alfak updated his python dump reading script to work with that dup format [01:00:15] *dump format [01:00:21] conversation was on 2017-05-01 [01:00:21] how do people actually work with the datadumps [01:00:44] once the dumps were for wikis with more than 40K pages it just didn't work for me [01:00:53] I was working with xml parsers [01:00:57] with tools like https://github.com/mediawiki-utilities/python-mwxml [01:01:47] ok let me try the page ids [01:01:55] any more words of wisdom? [01:02:06] that's h.alfak's library for reading the dumps without uncompressing them. its pretty neat [01:03:32] * bd808 looks for documentation on other ways to do the explain plan trick [01:04:36] I know j.ynus has documented it somewhere... [01:06:16] sweet! I think I got it. Just using page ids in batches is working [01:06:20] woohoo! thanks! [01:06:30] awesome [01:08:01] here's an example from j.ynus of partitioning a similar giant query -- https://lists.wikimedia.org/pipermail/labs-l/2016-March/004350.html [01:11:14] you guys are always very helpful. i appreciate it. have a good night [01:14:29] 06Labs, 10Tool-Labs, 07Documentation: Make a page of database usage tips - https://phabricator.wikimedia.org/T165002#3253906 (10bd808) [01:14:45] Hello, is there a way to see the ammount of memory currently used by my jobs? I see a large ammount of memory in qstat [01:15:26] Amitie_10g: qstat is pretty much what you have to work with I think. [01:17:16] I ask that because I worried about the memory usage during idle and archiving process wityh WebArchiveBOT (at least, I know the initalization process consumes a large ammount of resources) [01:29:39] Amitie_10g: the way that grid engine works, you get allocated some amount of ram on the exec node your job is tied to and nobody else gets to touch it even if you aren't using it. [01:30:10] you can lower your allocation if you want to [01:30:27] when your process tries to go above the limit grid engine will kill it [01:31:01] but unless you are taking crazy amounts of ram, like more than 4-5G, you should be fine [01:31:18] obviously smaller is nicer for sharing with everyone [01:32:10] hhvm is going to have a larger memory footprint than php, but typically lower than java [01:33:17] 06Labs, 10Tool-Labs, 07Documentation: Make a page of database usage tips - https://phabricator.wikimedia.org/T165002#3253976 (10bd808) I'm thinking of advice like @jcrespo gives in this email https://lists.wikimedia.org/pipermail/labs-l/2016-March/004350.html [01:42:18] 06Labs, 10Tool-Labs, 07Documentation: Make a page of database usage tips - https://phabricator.wikimedia.org/T165002#3253985 (10bd808) More links: * https://wikitech.wikimedia.org/wiki/Help:MySQL_queries * https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database [02:53:05] 06Labs, 10Tool-Labs: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3253995 (10bd808) I have a workaround for this now via PHP changes, but something must have changed to cause this encoding error. I know the PHP application code has not changed for som... [03:33:25] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:58:22] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [04:02:16] 06Labs, 10Striker, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Implement Tool Labs membership application and processing in Striker - https://phabricator.wikimedia.org/T162508#3254013 (10bd808) Here are screenshots of how things look from the approval side. the alerts queue:{F8026208} a pending reques... [05:54:21] 06Labs, 10Tool-Labs: Create or join a tool Tools Error - https://phabricator.wikimedia.org/T165009#3254084 (10FriyMan) [06:43:15] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/353230 (owner: 10L10n-bot) [07:50:26] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:30:27] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [10:44:02] 06Labs, 10Tool-Labs, 07Documentation: Make a page of database usage tips - https://phabricator.wikimedia.org/T165002#3254593 (10Multichill) https://tools.wmflabs.org/tools-info/optimizer.py might be useful to include too. [15:11:00] 10Tool-Labs-tools-Xtools: RfX Analysis for oversight and checkuser elections on dewiki - https://phabricator.wikimedia.org/T163444#3255359 (10Matthewrbowker) 05Open>03Resolved p:05Triage>03Low Closing, I haven't heard anything in a couple days and it appears to be working. [16:03:40] 10Labs-project-Phabricator, 05Security: Change incorrect informations on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165059#3255598 (10Framawiki) [16:15:57] 10Labs-project-Phabricator, 05Security: Change incorrect informations on the login form of phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165059#3255682 (10Framawiki) [16:22:59] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3255753 (10Framawiki) [16:23:19] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3255767 (10Framawiki) [16:37:45] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3255846 (10Framawiki) [16:46:48] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3255753 (10Paladox) Done. https://phab-01.wmflabs.org/p/Github-notif-bot/ [16:48:41] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3255896 (10Paladox) I am unsure how to add you so that you can change the settings of the bot. @mmodell is it possible to add users as maintainers as bot users accounts without needing to be an admin? [17:12:08] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3255975 (10Aklapper) @Paladox: What "settings" of a bot account on Phabricator do you exactly refer to? General documentation is at https://www.mediawiki.org/wiki/Phabricator/Bots [17:14:53] 06Labs, 06Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static and develop procedures to keep it maintained - https://phabricator.wikimedia.org/T163721#3255983 (10Andrew) 05Open>03Resolved a:03Andrew [17:18:47] 10Labs-project-Phabricator, 05Security: Change incorrect informations on the login form of phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165059#3256011 (10Aklapper) p:05High>03Triage [[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task_priorities| Resetting task priority... [17:19:57] 10Labs-project-Phabricator, 05Security: Change incorrect informations on the login form of phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165059#3255598 (10Paladox) >>! In T165059#3256011, @Aklapper wrote: > [[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task_priorities| Res... [17:20:46] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3256022 (10Paladox) >>! In T165065#3255975, @Aklapper wrote: > @Paladox: What "settings" of a bot account on Phabricator do you exactly refer to? > > General documentation is at https://www.mediaw... [17:23:31] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3256039 (10Paladox) 05Open>03Resolved a:03Paladox Done. @Framawiki hi, this is your conduit token P5425. Closing as resolved. [17:25:47] 10Labs-project-Phabricator, 05Security: Change incorrect informations on the login form of phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165059#3256044 (10Framawiki) But are you agree that user can try to login with his LDAP credentials ? Is there any way to force the message after puppet has worked... [17:28:31] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3256053 (10Framawiki) Thanks @Paladox. [17:29:08] 10Labs-project-Phabricator: Create a bot account on phab-01.wmflabs.org - https://phabricator.wikimedia.org/T165065#3256054 (10Paladox) Your welcome :) [19:23:15] 06Labs, 10Tool-Labs, 10Wikimedia-Hackathon-2017: Intro to Cloud Services (Labs, Tool Labs, database replicas, oh my) - https://phabricator.wikimedia.org/T164814#3246328 (10Rfarrand) Hi @bd808 ! Do you think this will be a newcomer suitable talk? Just wondering if you would like us to recommend it to newcomer... [19:29:31] 06Labs, 10Tool-Labs, 10Wikimedia-Hackathon-2017: Intro to Cloud Services (Labs, Tool Labs, database replicas, oh my) - https://phabricator.wikimedia.org/T164814#3256477 (10bd808) >>! In T164814#3256468, @Rfarrand wrote: > Do you think this will be a newcomer suitable talk? Yeah I think it should be applicab... [20:06:34] 06Labs, 15User-bd808: Consult with technical community on Cloud Services rebranding plan - https://phabricator.wikimedia.org/T165094#3256673 (10bd808) [20:07:47] 06Labs, 15User-bd808: Consult with technical community on Cloud Services rebranding plan - https://phabricator.wikimedia.org/T165094#3256689 (10bd808) Plan: https://wikitech.wikimedia.org/wiki/User:BryanDavis/Rebranding_Cloud_Services_products Call for participation: * https://lists.wikimedia.org/pipermail/la... [20:09:36] 06Labs, 10Tool-Labs, 10Wikimedia-Hackathon-2017: Intro to Cloud Services (Labs, Tool Labs, database replicas, oh my) - https://phabricator.wikimedia.org/T164814#3256691 (10bd808) >>! In T164814#3256477, @bd808 wrote: > We will also be talking about some upcoming changes that I haven't sent the announcement e... [20:45:54] hey, is there any particular reason why labwiki, aka wikitech.wikimedia.org, isn't included in the dumps on https://wikitech.wikimedia.org ? [20:46:46] probably not and it should be (making wikitech more like any other cluster wiki is a goal) [20:47:31] there is https://wikitech-static.wikimedia.org/wiki/Main_Page btw [20:47:39] in case that wikitech would be down [20:47:55] you mean dumps on dumps.wikimedia.org ? [21:24:04] cscott: I think mostly because wikitech is a special flower when it comes to database hosting. The db is not on the main cluster anywhere and that leads to it not being in the replicas or dumps. [21:24:46] we are working on making it un-special though -- T161859 [21:24:46] T161859: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859 [21:28:06] 06Labs, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3145305 (10bd808) >>! In T161859#3236750, @Jdforrester-WMF wrote: > Does this encompass {T98813} too? Yes, but indirectly I think. Once wikitech is a SUL wiki it should just live in the main wiki... [21:35:10] 06Labs, 10Striker, 10wikitech.wikimedia.org: Allow viewing/searching LDAP account creations including date - https://phabricator.wikimedia.org/T163478#3256950 (10bd808) There is a `createTimestamp` "operational attribute" on each LDAP object that tracks this per [[https://tools.ietf.org/html/rfc4512|RFC 4512... [22:02:28] 06Labs, 15User-bd808: Consult with technical community on Cloud Services rebranding plan - https://phabricator.wikimedia.org/T165094#3257034 (10bd808) p:05Triage>03Normal