[00:03:16] Got it going :-D [00:03:27] Niharika: ^ [00:04:27] bstorm_: Awesome! Thank you!! [11:26:21] !help could anybody with access to WM Labs look at https://tools.wmflabs.org/meta/ ? It could be some trivial coding problem IMHO, [11:26:21] qq[IrcCity]: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [11:27:05] and most of these tools (incl. /meta/stalktoy/ importantly) are knocked out. [11:30:02] https://github.com/Pathoschild/Wikimedia-contrib/issues/103 [11:30:36] wait, all tools are broken? [11:31:00] * arturo checking [11:31:12] arturo: todas las suyas sí [11:31:23] not all from Pathoschild, but many. [11:32:43] ok [11:33:08] they all fail with "Fatal error: Invalid serialization data for DateTime object in /mnt/nfs/labstore-secondary-tools-project/meta/git/wikimedia-contrib/tool-labs/backend/modules/Cacher.php on line 153" [14:03:21] RoanKattouw, mobrovac, you are both project admins in the 'services' project; do you know much about it? I'm noticing that there are seven VMs named 'ceph-something' created by gwicke; wondering if they're still needed. [14:03:35] akosiaris: same question [14:03:59] andrewbogott: in a migration right now, will look at it lather [14:04:01] later* [14:04:06] akosiaris: ok [14:04:29] Pchelolo: you might also know [14:05:31] andrewbogott: you can delete that [14:05:52] Pchelolo: the ceph VMs you mean? [14:06:28] ye, sorry, didn't specify exactly [14:06:34] great! Thank you :) [14:06:59] you can delete the ceph VMs, but for the rest we need to look into what exactly is on them [14:08:28] that's fine, it's only a couple of the ceph things that are in my way right now [14:09:25] !log services deleting ceph-1, ceph-2, ceph-3, ceph-4, ceph2-1, ceph2-2, ceph2-3 as Pchelolo says they are no longer needed [14:09:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Services/SAL [14:29:18] Technical Advice IRC meeting starting in 30 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @Tonina_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:56:25] Technical Advice IRC meeting starting in 5 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @Tonina_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [16:05:28] !replag [16:05:28] https://tools.wmflabs.org/replag/ [16:19:11] there are 2 sets of servers since a few months ago [16:19:20] let me get the documentation [16:20:08] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_to_the_database_replicas [16:20:26] enwiki.analytics.db.svc.eqiad.wmflabs will get you to the "analytics" replicas [16:20:38] enwiki.web.db.svc.eqiad.wmflabs will get you to the "web" replicas [16:20:44] they are technically the same [16:21:05] but, only small queries are allowed on the web (thinking for web requests) [16:21:19] because it has less load, it usually has less lag [16:21:46] so it is a compromise between allowing heavy querying and responsiveness [16:22:06] would that make sense more or less? [16:22:13] ah, I see. [16:22:32] and I remember having read this just a few weeks before. (oops) [16:23:04] seth_unr: we haven't yet seen exactly the cause [16:23:21] but I think at the beginning of the month there is overload on analytics [16:23:38] because people running monthly updates or something [16:24:19] I'm running two bots. I guess one of them may switch to the web db. [16:24:52] note queries longe than 300 seconds will not work there [16:25:06] it is ok if it is not a web, as long as queries are short [16:27:55] "1000 rows in set (0.35 sec)" (@tools-bastion-03) -- I guess that's fast enough :-) [16:28:54] yeah [16:29:17] the limit is around 300 seconds, or 5 minutes (noone will stay longer than that waiting for a web result) [16:30:15] it is also nice if you do not intend for queries to run wild- they will be controlled earlier [16:31:10] seth_unr: by any chance, are you u4802 ? [16:33:34] aehm, I don't know. (how can I know that?) [16:35:01] don't worry, it is not you [16:35:07] I know who it is [16:41:07] hmm, the replag at "web" seems to increase. [17:10:21] Hi guys, I have a question. I am trying to execute script inside toolforge environment but i cannot connect to the host. My credentials are in replica.my.cnf but error appears when i execute it. [17:10:45] !help Hi guys, I have a question. I am trying to execute script inside toolforge environment but i cannot connect to the host. My credentials are in replica.my.cnf but error appears when i execute it. [17:10:45] aalvarocostaa: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [17:11:27] aalvarocostaa: what is your error? [17:11:43] this one: [17:11:45] Traceback (most recent call last): File "namespace.py", line 8, in read_default_file='replica.my.cnf') File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect return Connection(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) _mysql_exceptions.OperationalError: (104 [17:12:12] 104... then? [17:13:08] I cannot access and i have credentials [17:13:16] I do not understand it. [17:14:11] look, could you post the complete error message? [17:14:24] there should be something after '104' [17:15:35] The full command would also be useful, aalvarocostaa [17:15:43] it will help if you post the script as well. (namespace.py) [17:15:51] Traceback (most recent call last): File "namespace.py", line 9, in read_default_file='replica.my.cnf') File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect return Connection(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) _mysql_exceptions.OperationalError: (10 [17:16:26] this is the error [17:16:40] I need the message after '(10' [17:16:50] Also, are you able to connect using the command line like mentioned here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_to_the_database_replicas [17:16:57] 'aalvarocostaa'@'10.64.37.14' (using password: NO) [17:17:06] wrong username [17:17:10] Yeah [17:17:22] but im using replica.my.cnf data [17:17:22] and you are not using password [17:17:50] please post your namespace.py and how you called it [17:18:08] import MySQLdb import csv if __name__ == '__main__': # Connect to data base connection = MySQLdb.connect(host='wikidatawiki.analytics.db.svc.eqiad.wmflabs', db='wikidatawiki_p', read_default_file='./replica.my.cnf') # Cursor in order to execute the queries cursor = connection.cursor() # Execute select query = 'SELECT page [17:18:14] * chicocvenancio is always too slow [17:18:28] there are sites like https://dpaste.de/ https://pastebin.com/ for pasting code [17:18:54] I don't fully understand the data at https://tools.wmflabs.org/replag/; right now the first table says "26:20:43" for s5, but the third table says for dewiki.labsdb (at s5) "04:01:49". what is the reason for this difference? [17:19:10] https://pastebin.com/kLxCw3Sc [17:19:15] thats the script [17:19:40] seth_unr: "26:20:43" is web, dewiki.labsdb is analytics [17:20:12] note there is 3 servers, some of them load-balanced [17:20:21] aalvarocostaa: read_default_file='./replica.my.cnf' <= that file is expected in the current directory [17:20:30] so it can change depending on where it ends up [17:20:45] yes, the script is in the same folder [17:21:08] How did you call it? [17:21:15] python namespace.py [17:21:17] seth_unr: query you servers's heartbeat_p table for results relevant to your server [17:21:36] i am executing this way [17:22:02] zhuyifei1999_: no "26:20:43" has been analytics. this value seems to hav decreased now to ~4h. [17:22:16] Are you executing this as your own user? Or as a tool? [17:22:43] in my own user [17:23:39] fwiw: [17:23:49] https://www.irccloud.com/pastebin/uhFZVUrB/ [17:23:53] works for me [17:24:05] jynus: yes, I can do that via sql in the shell. my question is just about the tables displayed at https://tools.wmflabs.org/replag/ [17:25:04] seth_unr: right, I guess I was being dumb :/ [17:25:07] i do not understand why i cannot execute it. [17:25:50] seth_unr: report that to the tool owner :-) [17:26:02] aalvarocostaa: try using read_default_file='~/replica.my.cnf' and see if that makes a difference [17:26:44] jynus: ok, I just wanted to verify that it might be a bug (and not just misunderstanding) [17:27:00] seth_unr: the canonical place is heartbeat_p [17:27:12] (the table) [17:27:45] seth_unr: There are 2 servers today that provide data for *.analytics.db.svc.eqiad.wmflabs. I think that my replag tool may be seeing data from one in the analytics.db.svc.eqiad.wmflabs table and a different one in the *.labsdb table [17:28:25] as jynus says, the canonical answer is to query the heartbeat_p table on the exact replica you are connected to [17:29:26] the replag tool tries to give people an idea of what is happening, but should not be considered to be authoritative information [17:30:31] ok. [17:31:16] it does not work [17:31:38] aalvarocostaa: do you mind if I sudo into your account and execute the script? [17:32:14] well.. Is it safe? [17:32:40] could you explain what you refer to by 'safe'? [17:32:58] feel free you can sudo [17:33:09] i really need execute this script to get the information [17:34:04] do you have the same prompt and command as: aalvarocostaa@tools-bastion-03:~$ python namespace.py [17:34:20] ^ works for me [17:34:44] i do not understand why it does not work for me [17:35:45] I write python namespace.py and then nothing appears [17:35:51] but it is not working [17:36:39] because that query executes forever [17:36:54] you are reading every single page in ns:0 [17:36:59] +bd808: I'm not sure, if I understood it correctly. Maybe my question at https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Database#Identifying_lag will resolve that. [17:37:49] then I guess it's slow, right? [17:38:59] it is slow, and idk why would you want to fetch all wikidata item titles [17:39:35] I honestly recommend https://wikitech.wikimedia.org/wiki/Help:Toolforge#Dumps [17:39:47] please don't do that [17:39:57] there is a titles.txt you can just download [17:40:13] Because I need this information to continue my project [17:40:19] aalvarocostaa: yeah what jynus said :) [17:40:22] I have already parsed the wikidata dump [17:40:45] but I need to know which Q has namespace 0 [17:40:58] all Q have namespace 0 [17:41:52] zhuyifei1999_: not really [17:42:03] chicocvenancio: ? [17:42:23] Ns is the namespace of the item in the wikis [17:42:28] all the Q items do not have namespace 0 [17:42:34] and I need to filter that [17:42:40] let me give you the pages that start with Q [17:42:44] that are not on namespace 0 [17:42:47] right now [17:42:49] chicocvenancio: wikidata items are stored in ns 0. no? [17:42:57] there are 19015 [17:42:58] only [17:42:59] No [17:43:13] zhuyifei1999_: there is no namespace in wikibase [17:43:42] ns is a property for the namespace corresponding in the wikis [17:44:24] Wikidata does have namespace for its mediawiki, but that does not matter for wikibase [17:44:44] WCDO had the same use case [17:45:33] I believe I(or him) created a task in phabricator about that [17:45:50] chicocvenancio: the query aalvarocostaa asked was asking for page_namespace in the page table [17:45:53] https://phabricator.wikimedia.org/P7108 [17:46:41] that is specifically the namespace of the item page on wikidata [17:47:04] bd808: I'm not sure, if I understood it correctly. Maybe my question at https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Database#Identifying_lag will resolve that. [17:47:29] zhuyifei1999_: sorry for my misunderstanding then [17:47:53] what the item page concerns (the linked pages on other wikis) do not affect page_namespace [17:47:56] np [17:48:00] (sorry for repetition, I accidently thought, the "+" was part of the name) [17:48:44] https://www.mediawiki.org/wiki/Manual:Namespace [17:49:00] I just need Q with namespace 0 [17:49:13] aalvarocostaa: why do you need the namespace of items? [17:49:21] aalvarocostaa: could you give an example of a Q item not in ns 0 [17:49:52] I just gave all of them on the list I sent [17:50:08] jynus example [17:50:19] you can see Q with another namespace [17:50:27] aalvarocostaa: but what is the use case? [17:50:45] Why do you want that? What are you trying to achieve? [17:51:02] i need this info because when I parse wikidata dump i want to filter the items [17:51:12] and i have to save on my local data base these items [17:51:36] I need this info. for my research [17:52:05] jynus: they are mostly talk pages, project pages, and other non-wikibase pages, no not technically Q /items/ [17:52:19] but thanks [17:52:40] hey he asked those to filter them, the list allows that! [17:53:17] yes but first i have to select them [17:54:00] aalvarocostaa: depending on what you are thinking of "namespace" you're asking for the wrong property [17:56:34] I think the exports do the right thing: https://dumps.wikimedia.org/wikidatawiki/20180501/wikidatawiki-20180501-all-titles-in-ns0.gz [17:56:53] there is 49M rows there, which match the total number of items [17:57:39] I guess non-Q 0-namespace pages can be substracted [17:58:12] Does this dump contains the list of Qs qith namespace 0? [17:58:27] yep, as far as I see, yes [17:58:41] if you wait a few hours, you can get the latest one, too [17:59:15] there is also specialized wikidata only dumps [17:59:19] which may be more interesting [17:59:46] It does not contain it [17:59:52] thanks jynus [17:59:53] have a look https://dumps.wikimedia.org/wikidatawiki/20180501/ [18:00:03] aalvarocostaa: are you a student of mmecor? [18:00:09] well i am already parsing the dump that contains all the wikidata info. [18:00:15] chicocvenancio: I don't really care it is not ns0 [18:00:21] I think the dumps have that into account [18:00:22] yes, am I master student [18:00:39] I've been through this with him a few weeks back with him [18:00:52] and instead of the ns0 they just dump the wikidata items [18:01:20] Well, the items, from the api, have a "ns" property [18:01:22] but still call it ns0 [18:01:29] That is what they need [18:01:46] There is a task in phabricator describing this [18:02:33] the wikidata dump contains items with all namespaces. in order to parse only the items with ns0 it is necessary to obtain the items with ns0 from somewhere else. aalvarocostaa you can use the code i gave you to query the mysql wikidatawiki_p database replica [18:03:32] This is a non-trivial query, but it is not a slow one (very big result set) [18:05:07] you have a 16 GB export of wb_terms [18:05:11] you can use that instead [18:05:35] https://phabricator.wikimedia.org/T191639 [18:05:51] jynus: ^ this is the relevant task [18:06:07] Can I parse https://dumps.wikimedia.org/wikidatawiki/20180501/wikidatawiki-20180501-all-titles-in-ns0.gz instead of use the code you provide me? [18:10:37] aalvarocostaa: of course you can, just remember to put that into your database and check from there. you will not be able to put all the items in RAM, and it will be easier to check if it is in the database than to run this other dump. [18:11:00] that's what the second function i gave u was doing [18:12:39] aalvarocostaa: i haven't checked that dump myself. but if it contains the qitem titles for namespace 0 that's exactly the same u r looking for in the mysql replica [18:13:12] mmecor: Now im gonna check this dump to verify that [18:14:54] aalvarocostaa: please check, but I asked the responsible for dumps and says he agrees with me that is what you should be looking for [21:04:07] !log deployment-prep ORES: drafttopic in beta [21:04:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [21:11:56] !log tools Added Tim Starling as member/admin [21:11:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:31:20] !log deployment-prep Bump ORES wheels [21:31:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [22:53:07] !log deployment-prep ORES: wheels fixups [22:53:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [23:21:05] bd808: This is strange, I just reprovisioned deployment-ores01 because the virtual disk was 80GB and the root partition only 20GB, but got the same results after recreating. [23:27:00] awight: that is by design. The other 60GB is available for partitioning and mounting but is not allocated by the base image [23:27:46] There is a role that will provision and mount it at /srv [23:28:02] aha, thanks!