I am not an expert but I had a somewhat similar problem with one old VN, only it didn't want to go fullscreen and was always stuck in native (super small) resolution. Fiddling around with video cars options solved the problem.
