Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
Tags

no frameinterpolation at scene change that's right but...

A topic by ChristianundCo created Jul 12, 2020 Views: 1,064 Replies: 6
Viewing posts 1 to 6

Thank you for the scene change detection so that there aren't any mixed frames. That's good.

But there must be a frame because otherwise the video gets out of sync with audio.

Yes I saw you change the fps so that that issue is tried to solved. But that's a problem when I interpolate 25.0 fps to 50.0 fps I want to have there 50.0 fps and nothing else like 49.xxx fps or such stuff.

I know how SVP resolves this problem at scene changes :  It dublicates the first frame of the next scene. So the next scene starts with two times the same frame. Is that option possible to build in? "At scence change, just dublicate the next first frame" and thats' it. No audio sync issues anymore.  :)

Or perhaps instead of the mixed frame a simple blendframe, that is the best compromise for scene change...

Today I made some tests with scene detection. Well, like others tools too it does'nt work very   reliable. Many scene changes aren't recognized and when I increase the sensitivity nearly every frame is a scene change. So it's worthless to me.  As  I sad, other scene detection tools I tried work equal bad.      Seems we need an AI for detect scene changes. ;)

Hey ChristianundCo,

I agree that scene detection would be a really great feature to get working well in DainApp. I've had some problems too with interpolated frames being created that have merged scenes together. Even worse is when you use Split Frame and you get a mind bending mosaic of two scenes!

For the meantime, I've been splitting my footage using Premiere (loads of other video editing apps let you do this) into scenes and then rendering them out as individual videos before putting them into DainApp. Then I put the output back into Premiere to reconstruct the video, add the original video audio back in and render the video as a whole.

Very time intensive depending on how many scenes you have, but this method means you get all the frames possible. Also, adding the video audio back in at the last step means you can use time remapping to stretch/compress the audio to match the interpolated video if you've lost or removed some frames. This becomes hard if you have a video with some long scenes and then short ones together as you loose more frames from the quick scene changes end of the video and then the audio doesn't match up if you only change the overall length. So you have to start using keyframes on your audio time mapping... lets not even go there...

When you try to remap time manually you start to realize how difficult it is to do automatically!!


So as long as your original video doesn't have any fades between scenes, splitting the video up before processing works really well. 

Thinking about it a bit more, if you have a fade between scenes of a few frames in your original video and there is no motion in those frames, you could remove the fade entirely before processing. Then once processed you could use the last and first frames from the two scenes to re-create the fade and fill in those pesky gaps that mess up your audio sync.

I love these types of thought experiments!


Theo

In the meantime I found and created my own  solution. I have a goo d avisynth -script, which find scenes (very reliable) an d list them in a textfile. Then I made a tool which reads the textfile and  creates blendframes (with running an avisynthscript) of last an d first frame at scene change. So I replace all mixed frames with blend frames. 

Such a showoff ;)

Now that's a good interim solution! To be honest, looking at the avisynth Licence it may be an option to bundle it into DainApp with the scripts you created.

Either that or like you suggested before, implement another ML algorithm to pick out and replace those mixed frames with blends. In a way that almost seems harder than the solution you've come up with though.

And sorry if my Hardware explanation was too simple for you, I figured I'd write it as though you didn't know much about PC hardware and then other people could use it too.

Theo

(4 edits)

No problem, I don't know everything, too. I'm learning by doing. The avisynth - scene change recognition wasn't created by me I only got it to work. But my tool  does  some batch calls. At the moment it's very quick and dirty  programmed and some steps must made manually, but it's a  good and easy solution for me.