[08:43:56] <matanya>	 the video nova resources are not accessible due to extreme cpu load 
[08:44:03] <matanya>	 I am going to reboot them
[08:44:13] <matanya>	 this will impact video2commons
[08:44:25] <matanya>	 which isn't working now. zhuyifei1999_ FYI
[08:51:00] <matanya>	 !log video rebooting encoding* nodes due to cpu overload 
[08:51:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[10:48:28] <jem>	 Hi, I'm checking my web tools under xxx.toolforge.org and the only problem seems to be that the css files can't be found as before, and the docs doesn't talk about that, any help?
[10:49:00] <jem>	 don't*
[10:51:54] <arturo>	 jem: which tool URL?
[10:52:02] <jem>	 arturo: https://jembot.toolforge.org
[10:52:51] <jem>	 https://jembot.toolforge.org/jembot.css isn't found, but the file is at /data/project/jembot/public_html/jembot.css
[10:54:07] <arturo>	 mmmm
[10:54:16] <jem>	 (and it has permissions 644)
[10:57:08] <arturo>	 the tool is running in kubernetes or in the grid jem ?
[10:57:37] <jem>	 In the grid, I think
[10:57:46] <arturo>	 jem: I think you are affected by this bug: T254640
[10:57:47] <stashbot>	 T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical` - https://phabricator.wikimedia.org/T254640
[10:58:15] <arturo>	 which already has a patch https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/603668 (still not merged though)
[10:58:24] <jem>	 Ah
[10:58:52] <jem>	 Ok, so changing the name to whatever.css should be enough
[10:59:01] <arturo>	 apparently yes
[10:59:07] <arturo>	 try it and let me know
[11:00:17] <jem>	 Yes :)
[11:00:26] <jem>	 Thanks, arturo
[11:01:22] <arturo>	 de nadas!!
[11:01:49] <jem>	 :)
[11:45:55] <arturo>	 !log paws reset wikitech user password for the service account `paws-dns-manager` to what is in labs/private.git/hieradata/common.yaml `profile::acme_chief::cloud::designate_sync_password` (T195217)
[11:45:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[11:45:57] <stashbot>	 T195217: Simplify ingress methods for PAWS - https://phabricator.wikimedia.org/T195217
[12:18:04] <arturo>	 !log paws release floating IP not in use: 185.15.56.43
[12:18:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[12:18:18] <arturo>	 !log paws release floating IP not in use: 185.15.56.42
[12:18:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[12:18:59] <arturo>	 !log paws associate floating IP 185.15.56.57 with VM paws-k8s-haproxy-1 (T195217)
[12:19:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[12:19:01] <stashbot>	 T195217: Simplify ingress methods for PAWS - https://phabricator.wikimedia.org/T195217
[12:20:57] <arturo>	 !log paws created DNS record `paws.wmcloud.org IN A 185.15.56.57` (T195217)
[12:20:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[12:27:59] <arturo>	 !log paws manually created an Ingress object to test routing to the hub (T195217)
[12:28:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[12:28:01] <stashbot>	 T195217: Simplify ingress methods for PAWS - https://phabricator.wikimedia.org/T195217
[15:03:06] <zhuyifei1999_>	 matanya: probably NFS related
[15:47:07] <matanya>	 Yes, I guess so
[15:47:13] <matanya>	 Anyway, working now
[15:59:51] <arturo>	 !log paws created DNS record `deploy-hook.paws.wmcloud.org IN CNAME paws.wmcloud.org` (T195217)
[15:59:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[15:59:54] <stashbot>	 T195217: Simplify ingress methods for PAWS - https://phabricator.wikimedia.org/T195217
[16:00:02] <MacFan4000>	 !log tools.zppixbot-test restart for config/code changes
[16:00:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[16:13:53] <mpeel>	 Hi, is the grid engine particularly busy at the moment? I submitted a job this morning, but it’s stuck in ‘qw’.
[16:15:27] <bd808>	 mpeel: do you have the job number handy?
[16:15:39] <mpeel>	 bd808: 6468169
[16:16:06] <mpeel>	 are there tools for checking server load? I tried ‘showq’ but that feature doesn’t seem to be available.
[16:16:29] <bd808>	 `does not request 'forced' resource "h_vmem"`. Did you submit using qsub rather than jsub?
[16:16:53] <mpeel>	 ah, yes, I used qsub.
[16:16:57] <bd808>	 mpeel: for cluster load, see https://sge-status.toolforge.org/
[16:17:25] <bd808>	 using qsub directly is difficult. We have a lot of queue limit things that you have to get just right
[16:17:29] <mpeel>	 bd808: thanks, that’s a useful link!
[16:18:21] <mpeel>	 ok, resubmitted using jsub, now 6490883
[16:18:30] <mpeel>	 … and now running, thanks!
[16:18:47] <bd808>	 mpeel: at https://admin.toolforge.org/ there is a "see also" drop down. It links to a lot of monitoring/status tools
[16:19:37] <mpeel>	 aah, at the top of the page, that’s well hidden!
[16:21:21] <mpeel>	 bd808: BTW, is there a parameter I can set that will send me an email when the job completes? I think for qsub it is ‘-m ea -M email@address’, is it the same for jsub, and does it work on this machine?
[16:24:56] <bd808>	 mpeel: maybe? :) We pass most qsub flags through, so I would suggest trying it.
[16:25:40] <bd808>	 I stopped thinking hard about grid engine stuff about 2 years ago which may or may not be a good thing
[16:27:11] <MacFan4000>	 !log tools.zppixbot-test remove test channels
[16:27:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[16:28:29] <mpeel>	 OK, I’ll try, thanks. :)
[16:43:47] <MacFan4000>	 !log tools.zppixbot restart for config/code changes
[16:43:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[16:51:29] <bd808>	 mpeel: I tried adding ‘-m ea -M email@address’ to a test job and it does seem to work :)
[16:51:57] <mpeel>	 Thanks, next time I run this script I’ll give it a go. :-)
[16:52:55] <mpeel>	 at some point I should probably set the bot running directly from toolforge, so it can just do the query and make the edits itself without my intervention in the middle, but that seems a few steps away right now…
[17:52:17] <bd808>	 !log toolsbeta Building webservice 0.71
[17:52:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[18:04:59] <bd808>	 !log tools Building webservice 0.71
[18:05:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:06:53] <Zppix>	 new webservice version, bryan?
[18:10:23] <bd808>	 yeah, fixing a couple of bugs. Most importantly T254640
[18:10:24] <stashbot>	 T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical` - https://phabricator.wikimedia.org/T254640
[18:12:14] <bd808>	 !log tools Deploying webservice 0.71 to bastions and grid via clush
[18:12:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:13:26] <Zppix>	 nice GL bryan
[18:14:25] <bd808>	 !log tools Rebuilding all Docker images to pick up webservice 0.71 (T254640, T253412)
[18:14:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:14:28] <stashbot>	 T253412: webservice 0.69+ fills /tmp with k8s ca cert files - https://phabricator.wikimedia.org/T253412
[20:47:46] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> renamed default branch from master to main
[20:47:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[20:54:47] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> renamed default branch from master to main
[20:54:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[21:05:44] <wm-bot>	 !log tools.wd-image-positions <lucaswerkmeister> renamed default branch from master to main
[21:05:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[21:07:27] <wm-bot>	 !log tools.wd-shex-infer <lucaswerkmeister> renamed default branch from master to main
[21:07:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-shex-infer/SAL
[21:09:45] <wm-bot>	 !log tools.wb2rdf <lucaswerkmeister> renamed default branch from master to main
[21:09:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wb2rdf/SAL
[21:11:54] <wm-bot>	 !log tools.speedpatrolling <lucaswerkmeister> renamed default branch from master to main
[21:11:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL
[21:15:08] <wm-bot>	 !log tools.pagepile-visual-filter <lucaswerkmeister> renamed default branch from master to main
[21:15:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile-visual-filter/SAL
[21:28:27] <bstorm_>	 !log tools cleaned up killgridjobs.sh on the tools bastions T157792
[21:28:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:28:31] <stashbot>	 T157792: /usr/local/bin/killgridjobs.sh is outdated - https://phabricator.wikimedia.org/T157792
[21:54:27] <bstorm_>	 !log toolsbeta removed killgridjobs.sh from toolsbeta bastion T157792
[21:54:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[21:54:30] <stashbot>	 T157792: /usr/local/bin/killgridjobs.sh is outdated - https://phabricator.wikimedia.org/T157792