[15:44:06] how would I install mysql-python? sudo apt-get install python-mysqldb needs to be done, and I'm not sure if that works in venv? [15:45:12] py2.7, so not sure if https://wikitech.wikimedia.org/wiki/User:Legoktm/toolforge_library is compatible [16:12:29] DatGuy: use pip inside venv, if you are on toolforge [16:12:51] I'm pretty sure that pip isn't enough? at least by what I looked up [16:14:09] The mysql client c library is already installed globally on toolforge servers [16:14:24] pip installs the python binding [16:15:11] or if you want you can install other implementations, like oursql or pymysql [16:22:25] alright [17:25:38] DatGuy: https://pypi.python.org/pypi/MySQL-python [17:25:57] that should work in a venv [17:58:17] valhallasw`cloud: found a pure python package (which I didn't find in the other 6 hours?), but https://stackoverflow.com/questions/372885/how-do-i-connect-to-a-mysql-database-in-python tells me it wouldn't work [17:58:48] "You must install a MySQL driver before doing anything." ... "For Linux, this is a casual package (python-mysqldb). (You can use sudo apt-get install python-mysqldb" [18:02:05] DatGuy: just run pip install mysql-python? [18:02:28] would it work? doesn't it need something special? [18:02:38] that's what i understood from reading the thread and other ones [18:02:59] alternatively pip install PyMySQL should also work (https://github.com/PyMySQL/PyMySQL, that's what legoktm's library uses) [18:03:02] try it? [18:47:36] DatGuy: look, the c library are already installed, what you have read is concerned with does not matter [18:47:50] I know. I'm just replyting to valhall [18:47:59] That's what I installed [18:59:26] DatGuy: I would recommend py3 unless there's a specific reason you still need 2.7 support [18:59:43] just used to it. would you recommend converting? [18:59:46] pymysql is pure python so it doesn't requre any C libraries to be installed [18:59:47] yes [19:00:03] I installed pymysql a few hours ago [19:00:04] it should be rather straightforward to migrate, and practically all libraries support py3.4+ these days [19:00:09] seems like it's working with 2.7 [19:00:50] https://pythonclock.org/ - 2.7 death date [19:06:43] :( [19:06:45] I'll check out https://docs.python.org/3/howto/pyporting.html [19:18:55] DatGuy: also there's a `2to3` auto porting tool which can be pretty useful to get started with [19:19:25] http://python-future.org/automatic_conversion.html ? [19:20:54] legoktm: centos7 will push eol of python 2 even further out :P [19:21:14] DatGuy: that's something different [19:21:52] zhuyifei1999_: It'll probably be decades before 2 is actually dead because of embedded systems, etc. But I am definitely going to help it die faster! [19:22:05] lol [19:22:17] DatGuy: 2to3 should be installed in your virtualenv [19:24:28] https://docs.python.org/2/library/2to3.html then? [19:24:45] DatGuy: yeah [19:32:56] (03PS1) 10Jean-Frédéric: Replace database connection library MysqlDB with PyMySQL [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390895 [19:33:14] so it wants me to change import urllib to import urllib.request, urllib.error, urllib.parse and the same with urllib2 ? [19:36:46] yes, urllib was split into submodules [19:37:14] also you might be interested in http://docs.python-requests.org/en/master/ which has a fanastic API for just making HTTP requests [21:00:11] (03CR) 10Lokal Profil: [C: 032] "works when I test it and we don't seem to use any of the edge cases where the two differ." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390895 (owner: 10Jean-Frédéric) [21:01:20] (03Merged) 10jenkins-bot: Replace database connection library MysqlDB with PyMySQL [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390895 (owner: 10Jean-Frédéric) [21:02:24] (03CR) 10jenkins-bot: Replace database connection library MysqlDB with PyMySQL [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390895 (owner: 10Jean-Frédéric) [21:56:43] legoktm: we should do a "why python3 is a thing you should care about" blog post for the cloud blog at some point :) [21:57:06] and we should figure out how to get python2 out of our prod repos [22:40:26] (03PS12) 10Lokal Profil: Restructure missing_commonscat_links [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/380060 (https://phabricator.wikimedia.org/T176528) [22:55:41] (03PS4) 10Lokal Profil: Make statistics table for unknown fields [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390169 [22:57:14] (03CR) 10Lokal Profil: "rebased on the other statistics patch to make use of new shared functions in common.py" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390169 (owner: 10Lokal Profil) [23:02:50] (03CR) 10Lokal Profil: [C: 04-1] "found one more place where new function can be used" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390169 (owner: 10Lokal Profil) [23:08:52] (03CR) 10Lokal Profil: [C: 04-1] "spotted a left over print statement. otherwise good to go" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/380060 (https://phabricator.wikimedia.org/T176528) (owner: 10Lokal Profil) [23:10:29] When are temporary tables going to be enabled on the new servers? [23:11:09] (03PS2) 10Lokal Profil: [BUGFIX]Allow non-ascii template names [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/390064 (https://phabricator.wikimedia.org/T180068) [23:12:46] Dispenser: I'll try to talk with the DBAs about it this week. [23:12:46] (03PS2) 10Lokal Profil: [DON'T MERGE]Investigate harvest of large wikidata dataset [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375529 (https://phabricator.wikimedia.org/T172691) [23:13:11] I don't know yet if its something that they will be in favori of or not [23:13:46] I keep looking at my missing_entries.sql. I can't use sub-queries because of GROUP BY removing singleton. I can't use tools.labsdb because it needs to be JOINed. [23:14:09] I guess I need a SQL to manual-SQL converter [23:15:51] I guess with my roundtrip latancy that only converts a 30 minute report to 7 hours (250,000 * 0.1 sec) [23:20:00] hmmm anyone else having SQL connection issues on tool labs? I'm getting reports on the hashtag tool: https://tools.wmflabs.org/hashtags [23:20:15] Also, do I need to file a phab task to raise the user_max_connection limit from 10 [23:20:20] mhashemirc: similar to T179922 ? [23:20:21] T179922: guc: "Error: Database error: Unable to connect to meta_p" - https://phabricator.wikimedia.org/T179922 [23:21:52] Dispenser: yes, please do open a task about the connection limit for your credentials if you are hitting the 10 connection limit and can explain why you need to run more parallel queries. [23:23:01] Cuz' browsers use 6x pipelines, meaning I can only support 1.6 users [23:23:49] and you have no control over parallelism in your tool? [23:24:53] the pipelining limits are typically about loading images in parallel, not database results, but I don't know what tool you are referencing [23:26:18] Well from my custom query killer: KILL related.suggest_redlinks, exceeded 15 sec allotted time [23:26:30] http://tools.wmflabs.org/dispenser/cgi-bin/related.py?dbname=enwiki_p&title=Mario [23:27:44] limitadj=5.0 [kills at 1/5 at alloted time]; Connection congestion (10 of 10) [23:28:48] It basically async JS which loads suggestions for users, such as red links [23:29:07] IIRC Toolserver had it at 30 [23:30:13] [23:21] < bd808> Dispenser: yes, please do open a task about the connection limit for your credentials if you are hitting the 10 connection limit and can explain why you need to run more parallel queries. [23:31:23] so far I think the limit has been raised for 5-6 tools. Having a low default is intended to help keep a new tool from running away with the global resources [23:52:56] bd808, it's a different one [23:53:15] bd808: Unknown database 's52467__hashtags' [23:53:37] which seems pretty bad [23:54:19] mhashemirc: that's is very likely from the labsdb1001 crash last week. [23:54:26] * bd808 looks for some links [23:55:00] thread at https://lists.wikimedia.org/pipermail/cloud-announce/2017-November/000007.html [23:55:21] ah ok reading [23:55:22] the TL;DR is that labsdb1001 is functionally dead [23:55:38] and all the *.labsdb traffic is on labsdb1003 [23:56:01] And in about 5 weeks this will happen: https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown [23:56:45] at which time the only place a tool can create and maintain its own database will be tools-db (aka tools.db.svc.eqiad.wmflabs) [23:57:23] ahaaa, ok, i'll read some more and see if changing some configs fixes this [23:57:24] thanks! [23:57:34] yw [23:58:03] mhashemirc: if you are not subscribed to https://lists.wikimedia.org/mailman/listinfo/cloud-announce you should probably sign up :)_ [23:59:38] mhashemirc: you should be able to get to your old database on c1.labsdb. It may or may not be corrupted from the crash