[00:13:10] !help mwclient and mwparserfrom hell are listed in ./www/python/venv/lib/python3.4/site-packages , yet I am getting the error that they do not exist when attempting to import them in python 3.4? This is confusing. [00:13:11] TheSandDoctor: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [00:13:24] mwparserfromhell* [00:13:41] (as a tool of course) [00:27:51] TheSandDoctor: is this a webservice you are running or a bot? I'm wondering if your virtual env is being activated when the script is run [00:33:51] TheSandDoctor: if you are doing interactive things, you will need to activate the venv first. One easy way to do this is to exec $HOME/www/python/venv/bin/python3 instead of just `python3` [00:43:15] A bot [00:43:24] @bd808: ^ [00:43:40] Non interactive. Just runs in background [00:44:16] how are you starting it? Are you explicitly using $HOME/www/python/venv/bin/python3? [00:44:53] the `webservice` command will use that venv location if it exists, but anything else will need to use it explictly [00:49:13] TheSandDoctor: ^^ [00:59:21] venv does exist -> mwclient and mwparserfromhell are listed in ./www/python/venv/lib/python3.4/site-packages [00:59:50] and how am I starting the script or? I am just using "import mwclient" [00:59:59] in the interpreter/script [01:00:14] @bd808 [01:04:13] scratch that [01:04:19] your last command worked [01:04:25] $HOME/www/python/venv/bin/python3 [01:09:40] How do I call that from the grid though @bd808? I attempted to submit the script with it and received the error that there is no such directory [01:09:48] submit a job* [01:10:21] use a full path, don't rely on $HOME [01:10:34] so /data/project/yourtoolanme/www/python/venv/bin/python3 [01:10:55] thanks lego! [01:11:02] sorry, just new to tools [01:15:58] no worries :) we're all here to help [07:14:50] bd808: probably not. I'm restarting it and still getting a KeyboardInterrupt. This is the sh file I use to start it: jstart -mem 2G -N afreporter /data/project/datbot/bot2/bin/python /data/project/datbot/Tasks/afreporter/afreportergood.py [07:18:18] DatGuy: is your source code available on the internet? [07:18:38] (in other words, can I get a link to your code) [07:53:51] legoktm: https://pastebin.com/ntCGPEcF [07:55:13] it's failed at 386 and 285 two times each (out of 4 restarts) [08:00:02] DatGuy: I think you need to add more memory [08:07:28] 2gb is not enough? [08:17:44] grid is terrible at counting memory [08:18:01] DatGuy: yeah just add more. don't worry about it [08:18:30] it counts virtual size instead of resident size [08:30:48] still getting a KeyboardInterrupt, have it at 5gb right now [11:24:26] Ruh roh! [11:24:45] I just lost connectivty to en-wiki [11:24:51] Pages not loaing [11:24:56] loading* [11:25:16] Can't log into Huggle either [12:03:08] hi all :) [12:03:23] can anyone tell me what “add toolinfo” on toolsadmin does? it doesn’t seem to be documented on wikitech [12:03:32] does a new toolinfo correspond to a separate tool account? [12:03:40] (i. e. can I `become my-tool-subtool` or something like that?) [12:08:42] toolinfo is used in a tool index [12:08:55] harej knows about it more [12:34:08] DatGuy: which job is this? I'll look [12:35:07] 9998377 right now [12:40:35] your virtual size is 118796KiB and resident size is 9216KiB [12:40:45] that's far from 5GiB [12:43:24] the memory usage don't seem to be increasing either [12:48:45] DatGuy: your max address space (max allowed virtual size) is 524288000 bytes (500MiB), are you sure it is set to 5GiB? [12:49:16] I had the -mem 5G argument [12:49:16] the job is running /data/project/datbot/Tasks/pendingbacklog/pending.py [12:49:27] oh whoops [12:49:29] wrong id sorry [12:49:34] 2886630 [12:49:43] ... [12:49:47] okay... [12:51:47] this one has virtual size of 192816KiB and resident size of 18020KiB [12:51:50] still far [12:54:24] (will observe it a bit) [13:20:40] fyi, resident size has increased slightly to 18048 KiB, but that is not limited [13:21:13] (unless the machine goes badly out of memory) [15:39:24] DatGuy: the memory usage is now 193072KiB virtual & 18184KiB resident. might be leaking but definitely not fast [15:42:44] Hey! How long does it take for the quota on WMCS to be recalculated? I'm reimaging the deployment-elastic* servers to stretch. Old instance is deleted, but I can't start a new one due to not enough CPU quota... [15:43:10] zhuyifei1999_: so its not an issue with the memory? [15:43:58] DatGuy: I don't see any other limiting factors [15:44:10] gehel: hmmm... It should be a real time thing. [15:44:21] https://www.irccloud.com/pastebin/WThhvf8S/ [15:44:46] bd808: it might be something else, I just assumed that deleting an instance and recreating it should "just work (tm)" [15:45:12] In theory I think I would agree. [15:45:26] you have 5 opened files, 2 processes (not sure of limits counts threads) [15:45:56] normal code don't lock memory [15:46:03] Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why :) [15:46:20] * gehel is joking, our labs works great! [15:46:38] bd808: any idea what I should check? [15:46:48] so I can only guess virtual memory being the limiting factor [15:47:57] gehel: sometimes some quotas get stuck we have to force a recalculation. I haven't seen it in a long time though… I'll see what I can turn up for your project [15:48:09] bd808: Oh, my bad, I created an xlarge instead of large [15:48:19] * gehel seems to not know how to read [15:48:21] ok, nevermind then :) [15:48:49] We should think about a quota bump for deployment-prep. It is very tight right now [15:48:59] that or hunt for vms that are not needed anymore [15:49:39] bd808: I'll release a 4 vcpu in a moment... but that's not going to be much of a change :( [15:51:09] deployment-tin, deployment-mira, deployment-deploy01, deployment-deploy02 -- that looks like redundancy to me [15:51:31] 4 xls where probably only 2 are actually needed [15:52:03] looks redundant to me as well, but what do I know [15:52:12] heh. same here :) [15:52:25] lack of ownership ? [15:52:54] * bd808 will look around a bit and ping some people [15:54:43] it looks like the -deploy* boxes are something that Krenair has been working on. He might just not have gotten to the point of being able to shutdown the older tin/mira hosts [16:19:47] (03PS1) 10Rosalieper: Addition of license and author from mediawiki api but the author info still need some work. [labs/tools/Commons-twitter-bot] - 10https://gerrit.wikimedia.org/r/450055 (https://phabricator.wikimedia.org/T190163) [16:47:41] !log jawiki deleted empty project [16:47:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Jawiki/SAL [16:54:55] !log mediawiki-dev deleted empty project [16:54:55] andrewbogott: Unknown project "mediawiki-dev" [16:59:29] !log montage-beta deleting unused project [16:59:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Montage-beta/SAL [17:07:25] !log oa-signalling deleting unused project [17:07:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Oa-signalling/SAL [17:15:31] !log nodejs deleting unused project [17:15:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Nodejs/SAL [17:17:41] !log patchtest deleting unused project [17:17:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Patchtest/SAL [17:19:56] !log performance deleting unused project [17:19:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Performance/SAL [17:22:06] !log persona deleting unused project [17:22:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Persona/SAL [17:23:09] andrewbogott: I'm going to have to update my "we have about 220 projects in our OpenStack env" interview preamble now. :) [17:23:38] we're at 206 now, still have another couple dozen candidates for deletion [17:29:38] !log propertsuggester deleted unused project [17:29:39] andrewbogott: Unknown project "propertsuggester" [17:29:51] !log propertysuggester deleted unused project [17:29:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Propertysuggester/SAL [17:29:57] (03CR) 10BryanDavis: [C: 032] Temporarily replace requirements.txt with `pip --freeze` from prod [labs/striker] - 10https://gerrit.wikimedia.org/r/443203 (https://phabricator.wikimedia.org/T198076) (owner: 10BryanDavis) [17:31:15] !log puppet-ca-replacement deleted unused project [17:31:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-ca-replacement/SAL [17:32:34] (03Merged) 10jenkins-bot: Temporarily replace requirements.txt with `pip --freeze` from prod [labs/striker] - 10https://gerrit.wikimedia.org/r/443203 (https://phabricator.wikimedia.org/T198076) (owner: 10BryanDavis) [17:32:43] !log puppet-cleanup deleted unused project [17:32:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-cleanup/SAL [17:34:04] (03CR) 10BryanDavis: [C: 032] Use #acl*repository-admins instead of #repository-admins [labs/striker] - 10https://gerrit.wikimedia.org/r/440758 (owner: 10MarcoAurelio) [17:35:15] (03Merged) 10jenkins-bot: Use #acl*repository-admins instead of #repository-admins [labs/striker] - 10https://gerrit.wikimedia.org/r/440758 (owner: 10MarcoAurelio) [17:35:18] !log ragefixing deleted mystery project [17:35:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ragefixing/SAL [17:37:56] (03PS3) 10BryanDavis: Order maintainers by cn [labs/striker] - 10https://gerrit.wikimedia.org/r/421669 [17:38:14] (03PS4) 10BryanDavis: Update UI to use term "Wikimedia developer account" [labs/striker] - 10https://gerrit.wikimedia.org/r/421670 (https://phabricator.wikimedia.org/T190543) [17:38:40] (03CR) 10BryanDavis: [C: 032] Order maintainers by cn [labs/striker] - 10https://gerrit.wikimedia.org/r/421669 (owner: 10BryanDavis) [17:38:47] (03CR) 10BryanDavis: [C: 032] Update UI to use term "Wikimedia developer account" [labs/striker] - 10https://gerrit.wikimedia.org/r/421670 (https://phabricator.wikimedia.org/T190543) (owner: 10BryanDavis) [17:40:12] (03Merged) 10jenkins-bot: Order maintainers by cn [labs/striker] - 10https://gerrit.wikimedia.org/r/421669 (owner: 10BryanDavis) [17:40:14] (03Merged) 10jenkins-bot: Update UI to use term "Wikimedia developer account" [labs/striker] - 10https://gerrit.wikimedia.org/r/421670 (https://phabricator.wikimedia.org/T190543) (owner: 10BryanDavis) [17:55:40] (03PS1) 10BryanDavis: Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/450070 [18:45:20] !help is there a way to change the python default encoding to utf-8 from ascii on tools? on my local systems it appears the default is utf-8. (Please tag me in any response as I have to go AFK for ~5 hours) [18:45:20] This key already exists - remove it, if you want to change it [18:45:35] and that will make it easier to find* [18:46:28] Now that I actually have the bot not generating anything in .err, this has popped up: 'ascii' codec can't encode character '\xdc' in position 18: ordinal not in range(128) [18:46:54] TheSandDoctor: try "export PYTHONIOENCODING=utf8" before running your script [18:47:17] in the interpreter or? (where?) @mutante [18:47:49] on the shell, before running python something.py [18:49:54] @mutante same error [18:50:15] I have tried everything I have found online so far with no luck [18:50:33] https://stackoverflow.com/questions/2276200/changing-default-encoding-of-python [18:50:36] and others [18:51:33] TheSandDoctor: see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#An_error_with_%22ascii%22_codepage,_%22file_not_found%22,_or_UnicodeEncodeError [18:52:58] tldr: use the grid, add `-v PYTHONIOENCODING=UTF-8` to jsub [18:55:23] I could wrap the command for jsub in my own sh file, right? [18:55:31] and that appears to have worked (fingers crossed) [18:55:41] so far no .out or .err has been generated yet [18:56:07] I will look more in to it when I am back later @chicocvenancio @mutante thank you! [18:56:39] it appears to definitely be working atm, barring any actual error during execution [18:56:47] .out now is populated with what I expected [18:56:48] thanks! [18:56:59] 👍 [18:57:33] and just confirmed - it has started editing [18:57:34] :D [18:57:38] thanks everyone! [19:04:24] !log striker Deleted alex-horizon-test.striker.eqiad.wmflabs instance [19:04:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [19:17:46] !log striker Building new deploy server as striker-deploy04.striker.eqiad.wmflabs [19:17:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [20:19:37] Is there an easy way to run a query against every db via SQL, or is that better done via an external script a la PHP etc? [20:19:51] !log servermon deleting unused project [20:19:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Servermon/SAL [20:21:14] !log spam-honeypot deleting empty project [20:21:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Spam-honeypot/SAL [20:24:24] SQL: you'll need some kind of driver script I think. We have a thing that helps with this one the Foundation's production hosts, but nothing that matches it in Toolforge [20:25:02] bd808, kinda what I was thinking, thanks [20:25:15] the production thing is the `foreachwiki` script that reads from the dblist files and runs a given command targeting each wiki in the list [20:27:16] !log testproject deleting unused project [20:27:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testproject/SAL [20:27:18] Yeah, I can build all the queryies easily enough via https://pastebin.com/YVDjnBtS - was hitting a wall getting mysql to eval those tho via union etc. Would be less memory used likely to do the PHP route anyhow. [20:28:30] !log staging deleting empty project [20:28:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Staging/SAL [20:35:32] !log wikidata-metrics deleting empty project [20:35:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-metrics/SAL [20:38:05] !log wikivoyage deleting empty project [20:38:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikivoyage/SAL [21:01:09] !log tor deleted empty project [21:01:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tor/SAL [22:56:08] PAWS/Quarry question at https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Database#Access_to_user_database_from_paws_or_quarry [22:57:15] I think it is unacessible from both at this point [22:57:42] *inaccessible [23:18:04] quiddity, chicocvenancio: I answered it by linking to the 2 feature request tasks that are open [23:18:22] :)