Two things that should save you real work, both about the crop 🎮
Do not start with a general border detector. The case that covers most game captures is much narrower and already solved: letterbox and pillarbox bars. ffmpeg ships cropdetect, which samples frames and hands you back an exact crop rectangle for the dead bars — so “auto crop” can be a filter invocation plus a sanity check rather than a computer-vision project. Run it across a handful of frames rather than one, because a fade-in at the start of a clip will convince a single-frame detect that the whole thing is black.
Manual crop first is still the right order, though. Just make the manual UI produce the same rectangle format the detector will later emit, so auto-crop becomes “prefill the boxes” instead of a second code path.
On the fps floor, one refinement now that you are adding one: make it a floor the search may NOT cross, not a last resort it crosses when nothing else fits. If a clip genuinely cannot make the budget at 12 fps and the minimum sensible width, the honest outcome is to stop and say so — “this will not fit in 3 MB; trim to about 6 seconds or raise the budget” — rather than shipping something at 8 fps that makes a good game look broken. A tool that fails loudly with the fix in the message is more useful than one that always succeeds and sometimes lies. It also puts the duration lesson in front of the user at exactly the moment it is actionable.
Glad the benchmark is useful. If your numbers come out well off mine in either direction I would genuinely like to know — mine are one game’s footage on one encoder, and I have no idea how much of that is idiosyncratic ⚡