[08:11:20] !log tools restart maintain-kuberusers service in tools-k8s-master-01 [08:11:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:11:26] AntiComposite: try again [08:12:23] This user is now online in #wikimedia-tech. I'll let you know when they show some activity (talk, etc.) [08:12:23] @notify Gopa please open a phabricator task and we will take a look (RE: ffmpeg issue on Toolforge) [08:41:07] !log shinken add myself as projectadmin to investigate shinken issues [08:41:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [08:57:17] !log shinken restart shinken service. Was in failure state, and is working after that [08:57:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [09:46:34] arturo: I opened this issue here https://discourse-mediawiki.wmflabs.org/t/cant-able-to-run-ffmpeg-with-videocuttool-in-wikimedia-toolforge/1451/2 [09:47:28] Gopa: ok I see [09:47:38] Thank you arturo :) [09:47:42] probably ffmpeg is not installed in the nodejs container image [09:48:16] we don't support having your own dockerfile or running your container image in toolforge (might do eventually in the future) [09:48:19] ffmpeg is installed https://ibb.co/FB4jTnj [09:48:40] is installed in the bastion [09:48:56] but you expect it to be installed for your webservice inside your container inside kubernetes [09:49:05] that's where it's missing I believe [09:49:53] so how and where can I install ffmpeg in server now ? [09:51:42] perhaps you can download directly ffmpeg to your tool home directory and access that from the container (i.e, put it next to your source code) [09:53:15] yes I also tried to change the ffmpeg location, you can see this here https://ibb.co/FB4jTnj [09:53:15] inside ~/www/js/routes/ there is ffmpeg. [09:54:48] https://ibb.co/dLmYFvH [09:54:48] I also tried to run this ffmpeg in the tool script, you can see this here https://ibb.co/dLmYFvH [09:56:15] does it have exec permissions? [09:57:53] yes I guess :) [10:02:14] please double check :-) [10:02:31] try ls -la [10:03:12] Yeah let me try again :) [10:10:13] anomie: Any idea one why I'm getting a readapidenied error for InternetArchiveBot on a simply query to retrieve namespaces? [10:11:01] bd808: ^ [10:16:23] Cyberpower678: are you setting a proper custom UA? [10:16:40] I think so? [10:16:40] (forwarded reply from bd808 ) [10:17:10] Is there something new in that regard that's preventing the bot from reading the API now? [10:17:52] you may be getting filtered because the UA. I don't know how new is that [10:18:21] Well it's new here at Wikimania [10:19:03] mw1345 served the requesty [10:19:43] if you are around, probably grab your laptop and come by the small wiki toolkits booth? [10:20:33] arturo: yeah i tried this ls -la here it is : https://ibb.co/gdWqL30 [10:21:34] Gopa: what you have is a link to the host binary [10:22:02] you may need to embed ffmpeg totally into your source code tree [10:22:33] arturo: in a session right now. Sitting next to thedj [10:23:02] so I also tried this type of comands '/usr/bin/ffmpeg .......command' [10:23:35] arturo: the exact same setup works on my localhost [10:23:43] With the same auth credentials and UA [10:23:48] again the same error :( [10:23:48] atleast this has to work ryt ? [10:23:53] Cyberpower678: famous words :-) [10:24:21] Well the credentials and UA config are pulled from the prod DB on my localhost. [10:24:26] So they are the same. :p [10:24:39] Cyberpower678: but not addressing [10:24:40] Not the credentials, but the config. [10:25:09] True. So I'm led to believe someone on Toolforge is causing this or the MW API is denying Toolforge. [10:25:19] Cyberpower678: bryan is probably your right debugging buddy for that [10:26:10] Gopa: two options 1) run in the grid instead 2) embed the static ffmpeg binary into your source code tree [10:27:43] arturo: grid ? [10:28:02] arturo: lol. As if bd808 wasn't being annoyed enough by me. :p [10:29:21] Cyberpower678: he is not very grumpy today LOL [10:29:27] arturo: how can I run in the grid ? [10:29:28] XD [10:30:53] Gopa: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Backends [10:31:49] * Gopa looking into the above link [10:32:19] let me know if you have any additional question [10:32:51] Ooh you ment gridengine, Ahh let me try now itself :) [10:34:18] !log admin downtimed cloudvirt1023 for 2 days [10:34:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:38:06] arturo: does gridengine supports node10 ? [10:38:47] Gopa: we can check that. If the debian packages are installed, then yes. Actually, if they are installed in the bastion, then probably yes [10:39:26] !log admin rebooting cloudvirt1023 for new interface names configuration [10:39:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:40:29] When I' trying to deploy using `webservice --backend=gridengine node10 start` [10:40:29] I'm getting this warning `type must be one of generic,lighttpd,lighttpd-plain,nodejs,tomcat,uwsgi-plain,uwsgi-python` [10:41:04] yes, is a bit different in the kubernetes backend and in the grid backend [10:41:11] in k8s, you specify the container type [10:41:27] in the grid, you specify the web server engine type [10:42:21] here is some documentation specifically about node https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#node.js_web_services [10:42:35] yeah I was following the same [10:47:04] I restarted https://tools.wmflabs.org/video-cut-tool-back-end with grid, Let me try testing by posting reqest from from front-end :) [10:48:23] 👍 [10:49:34] how to see logs now (grid) ? [10:49:51] I couldnt find in docs above. [10:53:42] arturo: [11:02:02] you should have some logs files in your tool home directory [11:02:06] Gopa: ^^^ [11:02:37] * Gopa checking again [11:05:26] arturo: This is again failing :( [11:27:18] arturo: so bryan says he can't help. [11:30:08] arturo, Thanks, the tool exists now and I can become it. [11:48:37] Maybe when anomie wakes up, he can help with the readapidenied error IABot is getting on Toolforge only. [12:09:06] Gopa which error? [12:09:12] Cyberpower678: :-( [12:09:40] how's wikimania arturo? :) [12:10:16] inspiring and vibrant hauskatze [12:10:34] Good to hear [12:20:48] I'm unable to see the error in logs, I guess it's same 'ffmpeg not found' (guess) [12:42:51] Gopa: can you enable additional debugging in your tool? ffmpg should be installed in the grid I think [12:44:15] I'm not able to see logs, the logs directory in home server is empty. [12:52:28] mmm [12:52:35] we should fix that first [13:05:07] arturo: that ? [13:08:21] lack of logging [13:09:17] BTW where are you searching for logs? I mean where in the tools home directory [13:16:44] here: ~/logs [13:17:11] * Gopa arturo: tools.video-cut-tool-back-end@tools-sgebastion-07:~/logs$ [13:25:19] ping! Is there still a conscience on https://phabricator.wikimedia.org/T125589 (Allow each tool to have its own subdomain for browser sandbox/cookie isolation). I have a tool which is missing Google OAuth API verification as we cannot verify the domain (tools.wmflabs.org) :-( [14:02:09] tonythomas: yes, we are working on it [14:04:28] Gopa: check error.log right in the top level dir in your home [14:05:10] it seems you are indeed calling ffmpeg [14:05:13] https://www.irccloud.com/pastebin/acf7b58C/ [14:05:45] and there is even more information in that file! :-) [14:09:19] arturo: Good to know. Waiting patiently (my only option [14:09:25] now it seems [14:26:59] tonythomas: I can give you some more concrete information if you are interested [14:27:09] arturo: please yes! [14:29:26] tonythomas: first, I would point you towards this: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/DNS_domain_usage#Resolution i.e, we plan to use `$tool.toolforge.org`. For implementing that, upgrading our kubernetes deployment is probably the first step, and we are working on that in this phabricator task: T215531 [14:29:27] T215531: Deploy upgraded Kubernetes to toolsbeta - https://phabricator.wikimedia.org/T215531 [14:37:55] arturo: thank you. I will keep an eye on those tasks! Thanks. [14:38:16] cool 👍 [14:45:59] * Gopa checking the error.log file [15:14:28] Gopa: If you think it works locally, will you be able to spawn up a docker container using the nodejs Dockerfile as given in https://phabricator.wikimedia.org/diffusion/ODIT/browse/master/nodejs/base/ and try starting your server in there ? [15:14:50] probably might give more insights (unless there are some additional blocks and checks in place in toolforge). [15:15:45] * Gopa going through the link [15:22:30] Gopa tonythomas as discussed before, containers won't solve your issue, because toolforge doesn't allow running arbitrary containers and our containers doesn't have ffmpeg on them [15:23:44] Ooh arturo okay :) [15:23:53] aha. tough luck (and makes sense for tool labs then). [16:47:04] After 25hrs of trails, finally solved ffmpeg issue with video-cut-tool-back-end in toolforge [16:47:05] Thanks arturo tonythomas :D :) :P [17:31:59] great! :-) [17:32:08] Gopa: what was it finally? [17:33:02] perhaps we can document it [17:42:18] with kubernetes container I'm not able to start npm, as you suggested grid supports ffmpeg and I just refactored my code, now ffmpeg is working in toolforge in videoCutTool :) [18:10:00] Sorry, [18:10:00] with kubernetes container I'm not able to start npm -> with kubernetes container I couldnt run ffmpeg [18:10:14] * Gopa going to sleep