[13:18:33] Hello, I am making some progress with the webservice. I have no more errors by its start (at least I see none): *** uWSGI is running in multiple interpreter mode ***spawned uWSGI master process (pid: 1)spawned uWSGI worker 1 (pid: 9, cores: 1)spawned uWSGI worker 2 (pid: 10, cores: 1)spawned uWSGI worker 3 (pid: 11, cores: 1)spawned uWSGI worker 4 [13:18:33] (pid: 12, cores: 1) [13:18:46] but I have Bad Request (400) [13:18:56] when I go to https://cycling-init-bot.toolforge.org/ [13:54:17] !log admin moving cloudvirt1035 from aggregate 'spare' to 'ceph'. We're going to need all the capacity we can get while converting older cloudvirts to ceph [13:54:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:13:02] Hi, I am unable to install the python-package xgboost into a virtual environment in the instance (instance: addlink-simple). I am following the steps from here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#venv [14:13:02] specifically, running "pip install xgboost" simply stops and returns "killed". surprised by this error since I had installed and used the package on toolforge not long ago in the same way. [14:13:02] Is this a memory-issue? What could I try to mitigate the issue? any ideas? [14:14:18] Psemdel, try running with debug and see what is happening [14:14:36] ok [14:15:27] Psemdel, you might have an error in your config file for instance. I'm not all that familiar with uswgi though. [14:17:33] mgerlach, using the kubernetes backend? What tool? [14:17:56] it is possible [14:18:46] Psemdel, feel free to share your $HOME/www/python/uwsgi.ini or $HOME/uwsgi.log if you are still stuck [14:19:15] how do I activate debug mode? [14:19:45] It is quite easy: [14:19:46] [uwsgi]check-static = /data/project/tool-cycling-init-bot/www/python/src/staticfiles [14:19:56] I haven't put any "home" [14:20:25] ballons: yes. tool: addlink-simple [14:20:25] - I follow steps 1-5 in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#venv [14:20:25] - then I run pip install xgboost (https://pypi.org/project/xgboost/ ), which is killed [14:20:40] balloons ^^^ [14:20:50] balloons: now [14:21:51] Now I see "In the uwsgi.ini file add the following:" [14:21:59] I thought it was write, not add [14:22:42] why do you use xgboost? [14:22:45] what does it do? [14:24:24] Psemdel: it is a tool for machine learning [14:27:45] mgerlach, let me have a look. I'm not sure if it's a resource limitation. [14:28:47] balloons: thanks, any pointers are much appreciated [14:34:56] I tried with [14:34:57] module = oauth_app.wsgiplugins = python3check-static = /data/project/my-first-django-oauth-app/www/pythonchdir = /data/project/my-first-django-oauth-app/www/python/srchome = /data/project/my-first-django-oauth-app/www/python/venv [14:35:04] but not better [14:35:17] module = oauth_app.wsgiplugins = python3check-static = /data/project/my-first-django-oauth-app/www/pythonchdir = /data/project/my-first-django-oauth-app/www/python/srchome = /data/project/my-first-django-oauth-app/www/python/venv [14:36:00] mgerlach, a log of what happened if you can share it would be helpful [14:37:19] module = oauth_app.wsgiplugins = python3check-static = /data/project/my-first-django-oauth-app/www/pythonchdir = /data/project/my-first-django-oauth-app/www/python/srchome = /data/project/my-first-django-oauth-app/www/python/venv [14:37:25] still not [14:37:57] [uwsgi]module = CyclingInitBotSite.wsgiplugins = python3chdir = /data/project/cycling-init-bot/www/python/srchome = /data/project/cycling-init-bot/www/python/venvcheck-static = /data/project/cycling-init-bot/www/python/src/staticfiles [14:38:00] now [14:38:05] !log upgrade facter to version 3 [14:38:06] jbond42: Unknown project "upgrade" [14:38:09] Feel free to use pastebin.com or https://paste.toolforge.org/ to share logs [14:39:21] https://paste.toolforge.org/view/1f11c3d4 [14:39:21] mgerlach, I can't reproduce it, though xgboost is a large package to install so it takes a bit [14:39:25] https://www.irccloud.com/pastebin/Qqx2VCHt/ [14:39:34] balloons: ^^^ [14:39:41] sorry we all speak at same time [14:39:57] mgerlach, oh, it's killed before it even fetches anything [14:41:27] mgerlach, did you try running the command again? [14:41:37] i tried a couple of times [14:43:24] i just tried on a new project and it all went fine: https://paste.toolforge.org/view/17620a2f. Do you have other pods running under your tool account? [14:43:59] I was thinking you were experiencing a cpu kill during install. But during the collection process, that is confusing to me [14:52:36] balloons: ah ok, how do i check the pods? [14:52:49] or how do i start a new pod? [14:55:21] !log admin [jbond42] upgraded facter to v3 across the VM fleet [14:55:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:00:12] Psemdel, I'm concerned with the "unable to load app 0 (mountpoint='') (callable not found or import error)" line in your log [15:00:26] ah ok [15:01:23] my app.py [15:01:36] import osfrom django.core.wsgi import get_wsgi_applicationos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'CyclingInitBotSite.settings')app = get_wsgi_application() [15:12:00] I suppressed the row module=Cycling.. in my uwsgi.ini [15:12:06] https://paste.toolforge.org/view/1c959905#L21 [15:12:36] still not working but this error disapeared (I think) [15:13:18] WSGI app 0 (mountpoint='/cycling-init-bot') ready in 1 seconds on interpreter 0x55febb9f5400 pid: 1 (default app) [15:19:08] mgerlach, I only see 1 interactive pod session running. So that's presumably not the issue [15:20:00] balloons: I am just trying to re-run now (after having started a new shell) [15:20:18] mgerlach, good idea. [15:20:27] I also deleted the previous venv [15:20:45] but still get the same error message [15:32:53] balloons: solved by adding --no-cache-dir; after downloading again not using cached version installation works . thanks for checking and showing that installation should work : ) [15:33:57] mgerlach, glad you got it to work! [15:40:36] Something else: has anyone an example of the django configuration for redis? https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge writes, A Redis instance that can be used by all tools is available via the service name tools-redis.svc.eqiad.wmflabs, on the standard port 6379. but that seems a bit short. [15:41:58] Psemdel: https://github.com/Wikidata/editgroups/blob/master/editgroups/settings/prod.py#L16 [15:42:05] thanks [15:42:07] (make sure you adapt the prefixes to your own tool!) [15:47:23] @op [15:47:40] wm-bot: ? [15:50:13] anyone struggling to log into horizon? Or is that taken down by the Ceph maintenance stuff? [15:50:57] tarrow: i was having trouble but just succeeded on retrying a few seconds ago [15:51:46] cool! [15:55:36] also working for me now; also is there any point in updating the "Status:" of this channel while Ceph stuff is ongoing? since (I guess?) it is actually happening now? [16:01:54] tarrow: there was a brief hardware outage, should be resolved now [16:02:04] (a hardware outage + some failure of redundancy which I don't understand yet) [16:11:40] andrewbogott: cool! Thanks :) [16:31:23] pintoch: what should be REDIS_PASSWORD and REDIS_DB? [17:13:32] I ask once again, as it is still not working. I get "Bad Request (400)" on cycling-init-bot.toolforge.org [17:13:34] https://paste.toolforge.org/view/c4ae52b6 [17:13:36] my log [17:14:26] https://paste.toolforge.org/view/c710f39e [17:14:35] my uwsgi.ini file [17:27:10] Psemdel, perhaps https://docs.djangoproject.com/en/3.1/ref/settings/#allowed-hosts [17:31:32] I included 'https://cycling-init-bot.toolforge.org/', 'toolforge.org', [17:31:35] but still the same [17:35:22] you could try ['*'] quickly to see if this is the actual issue or not [17:36:05] good idea [17:37:46] ahh very good guess, something happens [17:38:31] ok I stop for today, more issues to solve tommorrow [17:38:35] thank you! [17:40:05] is it a problem if I keep a connection to toolsdb open 24x7 ? [17:40:32] (writes happen 50-200 times a day; read happens once a day) [17:42:42] sidd: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connection_handling_policy -- that applies to ToolsDB as well as the Wiki Replicas [17:43:09] sidd: "Usage of connection pools (maintaining open connections without them being in use), persistent connections, or any kind of connection pattern that maintains several connections open even if they are unused is not permitted on shared MySQL instances (Wiki Replicas and ToolsDB). " [17:54:32] Trouble is I don't know how long the db connection could be idle, would it be better to reopen the connection every time for every write and then close it immediately? (the writes are taking place at random intervals, as and when messages from an eventstream api are received) [17:56:48] sidd: yes, it would be better to open the connection when you need it and close it as soon as you are done with that operation. [17:57:16] the "cost" of opening a new connection is pretty low (a couple of milliseconds usually) [17:58:54] The cost of keeping a connection open on the database server side is higher because we have a finite amount of RAM on the server and each open connection takes some of that for bookkeeping, caches, and related things. [18:00:07] Ok cool, thanks for the info! [19:43:18] hey, i'm writing a tool that is going to use phabricator's conduit api, and it looks like i need to use a token to access it. what are the best practices for storing it? [19:43:28] do i just put it in a text file somewhere and call it a day? [19:43:59] (also, i need to register a separate phab account to create the token, right?) [19:44:38] MatmaRex: yes to the file idea, just chmod it so that only your tool can read the credentials. And no you do not necessarily need a second Phab account [19:45:12] the token has the same access as the account that generated it thought? i don't necessarily want to give it access to security tasks i can view [19:46:28] hmmm... yeah I don't think you can limit the scope of the token, so it might be a good idea to get an bot account or at least a non-privledged account. I guess it all depends on what you are building [19:46:30] (it really is quite annoying that phab requires the token btw, if there's some secret way of avoiding that, i'd love to know. i just need some read-only queries for public things) [19:47:25] THings being "public" in phab was a weird thing we did here really. I don't know of a way to query Conduit with no auth [19:48:34] alright. thanks bd808 [19:48:45] the things I have that talk to Conduit are either using bot accounts (Stashbot, Strikerbot, phabban) or just local scripts I run as me [19:50:13] MatmaRex: if you decide you want a bot account, https://phabricator.wikimedia.org/project/view/981/ is the place to ask [19:50:20] hmm, if i end up making a new account… would it be a problem if i made my token public? e.g. committed it to a git repo [19:50:53] I guess not until our Phab gets DDoS and it is traced to that token :) [19:51:19] oh, huh, i didn't realize there were bot accounts [19:54:25] functionally a bot account in Phab is an API only account [20:27:01] Hi here [20:27:49] MatmaRex bd808 > this kind of questions could also be solved if we implement a secrets store, using a product like Hashicorp Vault [20:28:29] At worst, it could be somewhat a mouse-and-ball problem: you store the token to access Vault instead of the token to access Phabricator Conduit API [20:28:46] dereckson: patches welcome I guess ;) Secret storage in Toolforge is something that the admin team is interested in, but we haven't got that far yet [20:29:19] but yes, a Vault is only as good as the master auth to it. and that makes things harder in a multi-tenant, untrusted environment [20:29:20] even chmodding a file was more effort than i'd like, i definitely don't need any products ;) [20:29:40] at best, we could devise some automatic metadata extraction to authenticate the source (I see how to do that for OpenStack instances, perhaps less easy for accounts) [20:30:18] (btw the tool i needed this for is https://dtcheck.toolforge.org/dtcheck-2020-09-27.html , you can see "Related tasks" on that page there now) [20:30:38] (i just created a phab account for it) [20:31:26] For OpenStack, Barbican is the "magic". A.ndrew has been poking at that a bit, but hit a point where we need to get upgraded to a newer version of OpenStack. That in turn is blocked by an OS upgrade. [20:32:08] yes, it's inspired by AWS KMS, which is for example use for Vault automatic unsealing [20:33:34] and to allow machines to prove "I'm this machine, and I've some rights for those secrets", we can use Keystone [20:33:35] In the "turtles all the way down" spirit, the OS upgrades we need are semi-blocked right now on completing the migration of all VMs to Ceph storage for their main disks. And that is actively happening! [20:34:39] The list of things to do is endless, but we keep chipping away [21:19:17] About how much disk space is needed to become a Wikimedia dump mirror? [21:19:33] bd808: only two things in life are certain, death and migrations [21:20:24] hare: https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps#Space [21:21:09] somewhere between 5Tb and infinity I guess... [21:25:21] So how much disk space is needed? "all of it" [21:31:25] heh [21:31:35] It definitely depends how far back you want to mirror [21:32:20] there is some breakdown and historical data at https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps/estimates [21:32:56] checking the history for that page reminded me that Ariel is awesome and has been for a very long time :)