[12:18:50] I see bstorm, thanks for the context, things make more sense now :D [16:31:40] !log tools Ran `sudo -i apt-get purge nscd` on tools-sgeexec-0932 to try and fix apt state for puppet [16:31:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:36:51] !log tools clush -w @all -b 'sudo -i apt-get purge nscd' [16:36:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:40:43] Hello, I'm trying to create a demo for phan-taint-check-plugin on Toolforge (T257301), see https://taintcheck.toolforge.org/. The problem is, CSP headers disallow WebAssembly, which is vital for the tool. Is there a workaround for this? [16:40:44] T257301: Create a web demo - https://phabricator.wikimedia.org/T257301 [16:42:17] Daimona: we don't have any system of opt-in out of the CSP header, but it is also report -only so I'm not sure how it would block anything [16:42:43] Oh, it's report only, I see... [16:42:53] (Well I didn't really see, huh) [16:43:47] Nevermind then, but thank you for the heads-up [16:44:26] I would love to make the CSP header enforced honestly, but today it would break a very large number of maps related tools [16:45:53] Yeah, I agree with the intent, but I also guess that it would indeed break a lot of stuff... [16:51:59] https://csp-report.toolforge.org/ -- mostly maps things [16:54:35] Well perhaps some domains might be whitelisted? As long as there are no privacy implications of course [16:54:47] In this case that would be openstreetmap.org [16:57:04] there are privacy implications though. We don't control those servers. [16:57:31] and we have our own tile servers that are actually more comprehensive than then osm upstreams [16:59:41] Yeah, that makes sense [16:59:58] how are they more comprehensive? [17:01:17] Lucas_WMDE: I may be mistaken, but I'm pretty sure that the Cloud VPS maps project enables tile sets that OSM's tile servers do not. The hike/bike tiles are the ones I'm remembering [17:01:52] T190451 [17:01:52] T190451: Find out who maintains http://hikebikemap.org/ - https://phabricator.wikimedia.org/T190451 [17:02:26] the main difference I remembered was that our maps servers didn’t update at all for the better part of a year, but apparently that was resolved recently, yay T243609 [17:02:27] T243609: Maps master servers running out of space - https://phabricator.wikimedia.org/T243609 [17:02:36] we also have black and white tiles on wmcloud [17:02:50] (supposedly) [17:02:52] Lucas_WMDE: that's a different set of tile servers :) [17:03:16] it is? [17:04:03] Lucas_WMDE: yes, the maps Cloud VPS project is very different from the "production" tile server [17:04:33] ok, but I thought “our own tile servers” referred to the production ones [17:04:44] (we really shouldn't have anything hitting *.tile.osm.org anyway, they don't exactly have a ton of resources) [17:05:18] no, in this case I meant the maps.wmcloud.org tile servers that the community runs [17:05:23] (I didn’t even know we had another set of tile servers in Cloud VPS, didn’t read the messages carefully enough) [17:05:26] thanks! [17:05:53] the tile servers in Cloud VPS go back to ~2013 and were at toolserver before that [17:06:05] the WMF tile servers are more like 4 years old [17:44:14] !log tools rebased labs/private.git. 2 patches had merge conflicts [17:44:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:47:58] BTW, may I know what the memory/disk quota is for a tool? I have a configure && make still running after a hour, and cannot parallelize it because it deadlocks and fails [17:51:43] Daimona: is that in kubernetes or gridengine? [17:51:51] Kubernetes [17:52:11] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Quotas_and_Resources [17:52:18] You can query them and see what they are [17:52:37] The resource limits are quite possibly what you are hitting, which can be adjusted on the command line [17:52:45] But there are also overall quotas [17:52:54] If you need an increase, you can request them as well [17:52:56] Those are for the webservice though? [17:53:11] This command is running "offline" [17:53:19] Yeah I guess I'm hitting some quotas [17:54:21] That's all in kubernetes webservice or not [17:54:43] To adjust resource limits outside of webservice, you need to include a resources section in your yaml [17:54:58] Ah [17:55:32] Is there a guide for that? So I can experiment a bit [17:55:41] https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ [17:55:50] That's got a nice overview of it [17:56:06] There's defaults for your namespace as well that are listed in the wikitech page above [17:56:23] And your totals for all pods cannot exceed your quota [17:56:33] You can get quota increases if you need them, though [17:57:16] Gotcha, thank you! [17:57:27] I'll experiment a bit once the make command finishes [17:57:32] 👍🏻 [17:58:35] Oh btw, the closer you place your "requests" to your "limits" the more of a guarantee it is that you will have that amount available on the node your container runs on. Generally, requests is the guaranteed RAM and CPU while the limit is the maximum burst (in case that is not clear from the docs) [17:59:03] So if you know it needs at least x, it is actually better for the health of the cluster to "request" x...and set the limit above that a bit. [18:01:46] That's helpful, thank you. I still don't know how much this is going to consume, but at least building from source with emscripten surely seems to be greedy [18:50:21] bstorm: So, I think I managed to change the webservice quota, but I'd also like to increase the namespace quota before compiling the program again. Would it be possible to temporarily increase the quota for the time necessary to build it? [18:51:08] 'kubectl describe resourcequotas' says I have 1 CPU and 4Gi memory, which might effectively be low to rerun the compile script [18:51:52] It would be best to just put in a request. Our meeting to review such things is tomorrow, and part of the idea is to keep a record in case we run into a large failure or something and have to recreate it. [18:52:35] I think there's a link to the form in the doc? [18:53:37] For that matter if I find time to do a temporary increase, it'd be best to note it in the quota increase ticket too [18:53:44] Ah, I understand. Yes, there is a link, I was just wondering whether a temporary increase would be possible. However, I hopefully only have to compile it once, so waiting for it to finish, even if it takes 1 hour, is going to be quicker than waiting for tomorow :-) [18:54:30] I'm trying it as-is for now, but will open a task if I found myself having to compile it again, thank you all the same! [18:54:40] 👍🏻 [18:54:54] np [18:57:21] Daimona: my senior thesis project (decades ago) took 6.5 hours to compile on the fastest computer on campus. ;) [18:58:12] Ohhh that's the kind of things you want to run overnight, I guess. Not like this, constantly checking the terminal and begging it to go faster :D [18:59:03] ooooh, that web interface looks awesome [19:00:08] Daimona: I know b.rion has done a lot of work with emscripten, he might have suggestions on how to speed up compile time [19:00:25] Heheh that's just forked from phan's demo [19:00:42] I really have to note it somewhere, but let's do that once it works :D [19:01:22] Yeah, that might be appreciated. It's the first time I use it, so it might very well be my fault [19:01:34] is compiling the project slow or are you building emscripten/llvm? [19:02:44] I'm just running https://github.com/Daimona/demo/blob/master/build.sh [19:03:06] Both the 'configure' and the 'make' are very slow [19:04:22] oh, I guess you have to build all of PHP for this [19:04:39] Indeed [19:05:14] A small (?) price to pay [19:05:29] even on my laptop PHP builds would usually take like 20-30 minutes [19:05:55] Yeah I would definitely expect it to be slow, that's for granted [19:06:27] but 1+ hour really seemed to much, especially if you forget to include the taint-check path and have to compile it again [19:07:49] oof [19:07:55] if the quotas are a real bottleneck, it might be worth moving to a more powerful cloud VPS instance [19:15:01] Yeah well, for now I just wanted to see if it works. More resources would certainly be a good idea [19:21:31] Daimona, you are also impacted by using NFS storage in toolforge, unfortunately. Until we get that on 10G ethernet, the IO caps are pretty small still. Hoping to make that a bit better soon. [19:21:53] "soon" == this year or early next year [19:23:10] Ah, is there a task for it I can subscribe to? [19:25:27] * bstorm looks for it [19:25:58] It'll end up linked against this when we adjust it T218338 [19:25:59] T218338: labstore: Re-evaluate traffic shaping settings - https://phabricator.wikimedia.org/T218338 [19:31:07] Subscribed, thank you [19:36:31] !help Hello, can you check why instance smw-dev (flavor ID 2a6fca3c-9acd-42c0-8284-929e5345b607) haven't 40GB of disk? [19:36:31] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [19:37:06] Flavor is g2.cores2.ram4.disk40 but `df -h` shows less. /dev/vda2 19G 3.1G 15G 18% / [19:42:28] Kizule: maybe you need to follow https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances, but I’m not sure [19:43:58] ☝🏻 Yes I'm quite sure [19:45:00] thanks Lucas_WMDE [19:45:10] \o/ [19:45:57] Yup, works now. Thank you! [19:46:35] https://pastebin.com/6biFNCFe [19:46:38] * Lucas_WMDE enjoys reading about rack tetris in T266198 [19:48:06] I think strictly speaking it's sokoban and not tetris but I know from experience that no one knows what I'm talking about when I say 'sokoban' [19:52:10] rings a bell but I’d have to look up which one that was [19:52:24] oh, that one :) [19:53:39] I think there was also a penguin-based block moving game? But I think that was about escaping rather than about getting the blocks in the right place [19:54:19] yeah, it's def sokoban and not pengo [21:56:02] !log tools.lexeme-forms deployed 59f2c38fed (the previously-uncommitted JS fix, now committed; some uncommitted debug code is still there) [21:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [21:58:35] !log tools.lexeme-forms undeployed debug code, I don’t remember what it was for anymore [21:58:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [22:25:52] legoktm: I finally managed to get it working as expected :-D [22:26:11] Ooh [22:27:01] this is awesome!! [22:27:13] A bit slowish, but I guess this is just implicit in what it does [22:27:31] I mean, it's running PHP in our browsers [22:27:40] Exactly [22:28:05] how long did compilation end up taking? [22:29:37] 74 minutes according to 'time' [22:29:47] For the full build.sh script, that is [22:30:23] Which is why there's no PHP8 version, and also why I'm not going to bump its version of phan or taint-check very often, hehe [22:30:41] I bet we can get jenkins to auto build this after a new release [22:30:58] Ohhhhhhh that would be super-cool [22:33:16] maybe I'll poke at it this evening [22:33:21] is everything in your demo repo? [22:34:13] Yes, it's totally in sync with the toolforge tool [22:34:26] Except for the result of runnin build.sh obviously [22:34:40] I created something very cool and I can't brag about it to anyone because it's so esoteric [22:36:17] hare: this is the place to brag about esoteric things :) [22:36:50] ^^ [22:37:46] BTW, legoktm, thank you for your help, integrating with jenkins would be crazy good [22:38:41] I haven't done anything yet! [22:39:21] Well, you expressed your interest [22:39:52] haha [22:40:47] I remember the first time bawolff showed it to me, and my mind was blown. and now you've taken it to an entirely different level with it running in my web browser!! [22:40:59] I have made significant progress in a project I started over a year ago to decentralize RDF graph querying [22:42:11] :oo [22:42:45] Hah, a lot of things have changed since the first version. I mean, the backbone is the same, but it's much more feature-complete and less buggy. There are still a couple of weak points, but they're going to be addressed [22:42:55] hare: decentralize the querying or the or the query service? [22:43:16] https://github.com/harej/trp-uploader [22:44:28] hare: if you put enough data into skynet to make it self aware... [22:44:54] I'm causing a spike in their registry usage through my data mass-uploading >:]