The base game, Coffee Story, has over 100 4096x4096 texture pages, which take up about 6gb of VRAM when fully loaded. And while I tried to cut down on unused assets for this game as much as possible, a lot of them are just too integrated to remove. I believe it goes up to 5GB. I imagine what’s happening is as you play, more resources get loaded in, eventually capping your VRAM and then sprites get loaded into your RAM instead, impacting performance.
tl;dr You’ve just about got the minimum requirements to run this covered.