I made a mod where it loads the images on-demand, instead of all at once. I likely need to update it so it works with 0.5.20a and it's tweaks to load them as smaller images. However if you run 0.5.20 and the mod, it'll likely be a ton faster than what you see in 0.5.20 without crazy memory usage unless you spend the time to scroll through the entire list.
The 0.5.20a update fixes the memory issue but it should actually make the load time LONGER, as before it was just loading them into memory. If you had enough memory, it'd finish. Now it loads them and resizes them. 0.5.20a will take longer to load than 0.5.20, but yes it will use less memory.
With large portrait collections, the delayed loading is vastly more useful than converting them to smaller portraits. However the combination would be great too. I can probably update the mod for 0.5.20a tomorrow, maybe later today.