[19:06:49] andrewbogott I am here now if it helps [19:08:46] matanya: I think I wrote all my thoughts on the quota ticket -- [19:09:03] if you can tolerate the risk I'd like you to use the pre-created encoding nodes I made for you, so we can test the hardware they're on [19:13:04] Thanks andrewbogott I am fine with it, I am fairly sure zhuyifei1999_ is as well [19:13:15] great — let me know how it goes [19:13:20] oh hi [19:13:34] you should have enough additional quota to create the other non-encoding nodes you need; just ping that ticket when you've deleted the jessie bits [19:13:37] yeah lgtm [19:14:07] Ok, thanks. So zhuyifei1999_you have two new shiny buster boxes to play with [19:14:58] let me disable the interface [19:16:00] My plan is to create two more encoding boxes, and one redis [19:16:06] + one for dev [19:16:41] I am starting now, you should be good with encoding0[45] [19:17:29] matanya: could you delete encoding01 and create a instance with same size as video-redis? [19:17:41] yes [19:17:42] doing [19:17:55] ok web interface is disabled [19:18:12] also could you apply the /srv puppet role to encoding0[45]? [19:18:34] Sure [19:18:55] the new redis node is called video-redis-buster [19:19:06] thanks, I'm stopping the service on encoding0[23] [19:19:08] ok [19:21:01] I added the role to all nodes of the project [19:21:06] thanks [19:21:19] now deleting gfg and encoding01 [19:21:22] on the services are stopped on encoding0[23]. I'm waiting to be able to ssh into the new redis instance [19:21:25] wait not gfg [19:21:54] ok, not gfg [19:22:58] !log deleting encoding01 from video project as part of buster migration [19:23:00] matanya: Unknown project "deleting" [19:23:28] !log video deleting encoding01 as part of buster migration [19:23:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [19:23:35] I'm checking stuffs on gfg and see what needs migrating [19:24:24] ok, encoding01 is gone after a year of service, so long, and thanks for all the fish, we will miss you [19:25:38] andrewbogott and zhuyifei1999_ I am thinking of writing a blog post about the video project and my experience as a wmf cloud user, any objection? [19:25:58] np for me :) [19:25:59] not at all [19:28:26] video-dev-buster is live now as well [19:29:31] ok I think I removed everything that isn't needed on gfg01, can you delete encoding02 and videodev? I just checked videodev has nothing 'interesting' on it (like, its v2c config) has a copy on toolforge so I'll just copy over [19:30:02] I'll migrate gfg once I finish redis [19:31:18] !log video deleting encoding02 and videodev as part of buster migration [19:31:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [19:32:34] gone [19:35:08] Just a fun fact, this project is almost 5 years old [19:37:33] :) [19:37:52] I just set up replication from video-redis to video-redis-stretch [19:38:33] buster, you mean? [19:39:10] yeah [19:39:13] oops [19:39:36] ok I think redis has been copied over [19:39:47] let me reenable the interface and see if it works [19:41:42] !log video replicated data from video-redis to video-redis-buster and then disabled replication [19:41:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [19:44:34] It doesn't work for me, stuck at loading [19:46:01] yeah, I realized the socketio webservice also needs to change the redis server. that is done [19:46:53] ok I'm gonna shutdown video-redis and hopefully nothing breaks [19:47:57] lgtm [19:48:40] ok could you delete video-redis and recreate a gfg instance? I basically just have to copy over the files [19:50:00] I'll setup encoding04 now [19:54:54] Yes, can you remind me the use of gfg? [19:59:28] I also want to replace encoding03 with encoding06 [20:00:14] gfg was for 'giant file grabber' which I was using to grab files off of google art project. it is still sometimes used this when when people ask me to upload some gigapixels. the other use right now is I'm using gfg to compiled some optimized & more recent ffmpeg [20:00:16] sure [20:00:39] ok, remember now [20:00:55] ok, recreating gfg and building encoding06 [20:00:57] the row-mt wasn't available in jessie, not sure if it's available in buster [20:01:50] can check, where is that needed? [20:02:11] a lot of files on gfg is still 'not automated' so I'd rather copy files over in /srv than to create from scratch [20:02:30] ffmpeg libvpx encode option, for multithreading [20:02:44] gfg is ready [20:04:01] ok [20:04:09] https://www.irccloud.com/pastebin/6zP6ysiR/ [20:04:12] o.O [20:04:39] weird [20:04:56] what is the new hostname of gfg? [20:05:19] gfg [20:05:27] I dropped the01 [20:05:48] I don't think we will have more gfg's in the future [20:06:43] ok [20:12:25] I don't know what is going on with this 'Invalid user/group name or numeric ID' [20:12:52] should I try numeric ID? [20:13:27] !log video copying data over from gfg01:/srv/zhuyifei1999 to gfg:/srv/zhuyifei1999 [20:13:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:14:45] It probably won't work [20:15:07] there seems to be a mismatch in passwd or ldap somewhere [20:18:05] -rw-r--r-- 1 root root 766 Oct 26 19:56 /lib/systemd/system/v2ccelery.service [20:18:13] zhuyifei1999_ can this be related ^ ? [20:18:53] yeah I created that file [20:19:15] that's the systemd service for v2ccelery [20:20:03] but no execute rights [20:20:46] oh, nvm [20:28:08] argh [20:28:30] indeed [20:28:52] does it work with setting the user/group to root? [20:29:26] let me just try numeric. cuz in the end, it's only the numeric id that matters to the kernel [20:29:52] no. root doesn't work because root_squash on scratch [20:38:54] ok I need to install the patched pywikibot from 2016 :( [20:41:17] ah? why? [20:46:42] https://gerrit.wikimedia.org/r/#/c/pywikibot/core/+/277060/ <= I really need to update this patch [20:47:08] and recent pywikibot is barely compatible with python 2 since around may I think [20:47:22] so I really need to migrate the whole thing to python 3 [20:48:07] in December / January I think I'll have time [20:48:35] what is going on... [20:48:45] https://www.irccloud.com/pastebin/qWVWSp7Z/ [20:52:16] I thought nowadays chunked is well supported within pywikibot [20:52:49] async chunked is not [20:53:09] normal chunked you get to upload around 1.5G and after that it'll time out [21:00:08] I think I messed up redis somewhere [21:00:54] https://www.irccloud.com/pastebin/VEsZxt0O/ [21:02:21] Seems so [21:02:34] maybe clean and start from scratch? [21:03:05] oh might be celery version difference [21:03:16] cuz I didn't rebuild the toolforge one [21:04:06] Can I help in anyway? [21:05:50] umm, I just restarted the toolforge interface, you you submit a task once it's done restarting? [21:05:55] *could you [21:06:16] yes [21:06:45] well, no [21:07:05] the interface troughs an error: /venv/local/lib/python2.7/site-packages/redis/connection.py", line 718, in read_response raise response ResponseError: value is not an integer or out of range [21:09:16] argh [21:10:19] updating redis then [21:12:11] matanya: ok restarting [21:12:59] still dead [21:15:21] :( [21:19:35] oh I know why [21:22:48] matanya: ok restarting again, hopefully this works [21:23:48] https://github.com/toolforge/video2commons/commit/7a5109c62adfebe2b587f05fb94997e08f139646 this was the fix hopefully [21:25:16] matanya: ok seems alive now, can you submit a task [21:25:17] ? [21:25:24] Error: An exception occurred: DownloadError: ERROR: 8ayBkYOAGBA: YouTube said: Unable to extract video data [21:25:58] oh yeah that the blacklist thing [21:26:18] kaldari hasn't replied [21:26:36] anyway to overcome this for testing? [21:27:56] not really... use a video from commons or vimeo or some other sites? [21:29:41] yay I see something [21:31:04] yes, me too! great! thank you [21:31:11] move forward to the remains? [21:31:29] I'm setting up 05 [21:31:38] can you delete 03 and create 06? [21:31:38] I still have to get rid of gfg01 and encoding03 [21:31:44] I created 06 [21:31:48] deleting 03 now [21:32:21] !log video deleting encoding03 as part of buster migration [21:32:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [21:33:08] done [21:33:17] oh also can you create the proxies for v2c[4-6]? [21:33:19] thanks [21:34:49] yes [21:35:03] target is v2c.wmflabs.org? [21:36:20] We had port 80, right? [21:37:40] target is every instance [21:37:53] so like encoding04 gets v2c4 [21:37:59] yeah port 80 [21:39:24] done [21:39:50] thanks [21:40:43] I think you can delete gfg01 now [21:41:15] doing [21:41:29] !log video deleting gfg01 as part of buster migration [21:41:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [21:42:18] !log video buster migration done zhuyifei1999 is awesome [21:42:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [21:43:15] heh thanks [21:43:50] On Monday I will try to escalate the youtube issue at wmf, and if that fails, i will try my connections at google [21:44:16] can you stop the task running at 04 so I'll stop celery at 04 so we can test 05? [21:44:21] it05 shoudl be running [21:44:25] *05 should [21:46:04] though, celery global status is completely broken [21:46:06] well it is not my task, so i can only kill it from shell [21:46:14] oh [21:46:22] I thought it's yours [21:46:32] I guess I'll wait for it to finish then [21:46:36] there where several attempts when the service was down [21:46:56] and the minute it was back 2 or three people tried to submit jobs [21:48:48] ok [21:53:23] matanya: fixes that. bug is https://github.com/celery/kombu/issues/1063 :) [21:53:36] https://www.irccloud.com/pastebin/qW1I2LJ2/ [21:53:48] with alias v2ccelery='sudo celery -A video2commons.backend.worker --workdir=/srv/v2c' [21:54:01] now let me setup 06 [21:54:08] great! [21:54:43] and that task you were waiting for is done zhuyifei1999_ [21:55:01] oh yeah thanks [21:55:45] ok can you submit a task to 05 and see if it works? I stopped 04 [21:56:01] yes [21:56:19] I wonder who submitted 'Real ID Traveler' [21:57:51] submitted [21:59:15] ok seems to work [22:03:25] So only 06 is left and the work is done [22:04:29] We should have a plan for a follow up work on migrating to python3. I took the action to stress the whitelist of youtube [22:04:42] yeah I'm applying that backend.pp script [22:04:47] ok [22:05:16] ok 06 should be up, let me disable 05 and test [22:06:37] I think that broken task was Eissink's [22:06:44] I'll leave them a message [22:08:31] I am ready to test 06 [22:08:32] notified https://commons.wikimedia.org/wiki/User_talk:Eissink#video2commons [22:08:55] ok yeah only 6 is online now [22:09:26] seems working [22:12:17] nice [22:13:11] ok I just enabled 05 and let's keep 04 stand by in case we need to rotate [22:16:10] ok, cool [22:16:25] I guess everything is done now? [22:16:32] yes, thanks much! [22:16:39] np :) [23:13:01] matanya: oh when you write the blog post, I wanna see it ;)