I wonder in cases of VRAM limits, doesn't windows as well as many videocards allow for shared memory use? Seen it done in other areas but not with things like stable diffusion
Basically if shared memory can be used, then core functions can still work even if it will be really slow.
For example, allowing things as far as a worst of all worlds memory setup (cursed image of a GTX970), as soon as a task starts to need more than 4GB it can gradually stop using that last 512MB pool to avoid context switching there, and start using a bunch of system RAM. 