[06:02:22] any way to scp files over to toolforge? I tried scp'ing them to login.tools.wmflabs.org but of course that didn't work [06:03:39] ...nevermind [12:56:28] Hi, I created https://phabricator.wikimedia.org/T230284 as per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Requesting_additional_tool_memory instructions. Can an admin please review my request to increase memory limit for webservice tool? [13:04:53] don-vip, those docs say memory increases can only be done for SGE services, but your task says this tool is on k8s [13:06:16] Krenair are the docs up to date? If yes, what would be the way to request more memory on k8s? [13:06:24] I don't know [13:09:14] I believe the SGE restriction is no longer true, since https://phabricator.wikimedia.org/T183436 and https://phabricator.wikimedia.org/T234702 are resolved [13:10:25] did they document how an admin should set a higher limit for a particular tool? [13:13:57] root@tools-k8s-control-1:~# kubectl get -n tool-spacemedia limitrange tool-spacemedia -o json | jq .spec.limits[0].max.memory [13:13:58] "4Gi" [13:18:02] (I didn't change anything here that's just what I see) [13:18:32] Krenair I didn't find better documentation. Why does kubectl describe pods spacemedia-597444c5db-fc6mg | grep memory returns 512Mi if your command return 4Gi ? [13:19:30] root@tools-k8s-control-1:~# kubectl get -n tool-spacemedia pod spacemedia-597444c5db-fc6mg -o json | jq .spec.containers[].resources.limits.memory [13:19:30] "512Mi" [13:20:08] root@tools-k8s-control-1:~# kubectl get -n tool-spacemedia deployment spacemedia -o json | jq .spec.template.spec.containers[].resources [13:20:08] {} [13:20:48] looks like the Deployment object doesn't set anything specific? [13:21:51] I don't know, I'm new to all of this and have very little k8s knowledge. I found this: https://phabricator.wikimedia.org/T246553#5930977 [13:23:49] it's not gonna be the ResourceQuota object: [13:23:50] root@tools-k8s-control-1:~# kubectl get -n tool-spacemedia resourcequota tool-spacemedia -o json | jq '.spec.hard["limits.memory"]' [13:23:50] "8Gi" [13:23:50] root@tools-k8s-control-1:~# kubectl get -n tool-spacemedia resourcequota tool-spacemedia -o json | jq '.spec.hard["requests.memory"]' [13:23:50] "6Gi" [13:24:05] maybe it's up to me to request more memory when I start the webservice? [13:24:46] right now I start it as follows: webservice jdk11 start /data/project/spacemedia/run.sh [13:25:03] it looks that way [13:25:21] I don't really know how tools' webservice tool works, I'm used to dealing with kubernetes directly [13:25:49] `webservice --help` shows a -m MEMORY flag for k8s [13:25:58] ah, interesting [13:26:05] I'm trying it then [13:29:45] all ok? it looks like your deployment vanished [13:29:48] oh there we are [13:30:13] I just relaunched it [13:30:33] kubectl get pods returns nothing yet [13:31:25] root@tools-k8s-control-1:~# kubectl -n tool-spacemedia describe rs spacemedia-5658f69bc4 | tail -n 1 [13:31:26] Warning FailedCreate 14s (x6 over 94s) replicaset-controller (combined from similar events): Error creating: pods "spacemedia-5658f69bc4-7chw9" is forbidden: maximum cpu usage per Container is 1, but limit is 2 [13:33:06] # kubectl get -n tool-spacemedia limitrange tool-spacemedia -o json | jq .spec.limits[].max.cpu [13:33:06] "1" [13:33:14] ah sorry [13:34:27] ok much better with 1 cpu [13:35:23] root@tools-k8s-control-1:~# kubectl get -n tool-spacemedia pod [13:35:23] NAME READY STATUS RESTARTS AGE [13:35:23] spacemedia-7f67f96687-2zzbj 1/1 Running 0 95s [13:35:31] success? [13:36:31] it's starting [13:36:58] with default cpu settings it takes 5 min to start, should be faster with 1 cpu [13:39:05] nice big ascii art in the logs :) [13:39:42] yes :) [13:39:59] real logs are redirected to spacemedia.log file [13:40:07] it seems a lot slower with 1 cpu [13:40:36] what was it before? [13:40:42] no idea, default settings [13:40:45] ok [13:41:22] I thought it was 500m. Maybe not [13:42:08] I'd suggest leaving an update on the task with where we're at, then on Monday someone who knows what we really want to do can talk about how limits are to be raised on a per-tool basis... it might just be a case of an admin editing the LimitRange or ResourceQuota object in the cluster or something [13:42:22] or, tuesday, depending on public holidays etc [13:42:27] sure :) [13:43:00] it's too slow, I'm restarting it with default cpu settings [13:51:59] Krenair it works ! [13:52:08] thanks a lot for helping me! :) [13:52:15] no problem [14:03:44] I've updated the docs [15:17:49] not seeing anything from CVNBot21 in #cvn-mediawiki from the last few days, any idea what's going on there? [15:19:06] did you ask CVN people? [15:20:34] noticed some reports from being missing on other channels, just didn't know if that one specifically was mentioned that you knew of [15:22:33] DSquirrelGM, we don't know of specific CVN bots [15:24:01] CVN stuff may be hosted within Cloud VPS but that doesn't mean wikimedia-cloud people will know details about the CVN service, you still need to ask CVN people [15:24:58] you can find some people who are likely to know more about it here (exclude and.rew I think in this case): https://openstack-browser.toolforge.org/project/cvn [15:27:03] ok, I guess I'll just post in #countervandalism next time if I see it then [17:03:45] hello fam [17:05:21] i am trying to install jupyterhub with helm, but am getting this issue and i couldnt continue "Error: validation failed: unable to recognize "": no matches for kind "Deploymen [22:48:00] !log tools.stashbot Updated to e6cb99a (Update for elasticsearch7) T247715 [22:48:08] ouch [22:59:55] !log tools.stashbot Updated to f2ca4e4 Update !bash handling for es7 (T247715) [22:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [23:11:22] !log tools.stashbot Updated to f2ca4e4 925b463 Update !log handling for es7 (T247715) [23:11:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [23:17:29] !log tools.sal Updated config to point to es7 cluster (T247715) [23:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL