Haha, it's been like that since the dawn of the PC :)
But I have a pretty clear Idea of how to proceed now, by only keeping metadata of images in RAM and loading the actual pixels from disk on demand. And then use the rest of available memory for caching. Should get me below the 4MB line.