I've got a mod in the works for the portait/body buttons to load images on-demand when you scroll to them, instead of everything at once. Probably release it in it's semi-broken state as that's better than them not working at all on large collections. Basically, after loading, you need to scroll some once so it loads the first set.
It shouldn't have any impact late game. The most intensive point is when you first load a save, or start a new game. At that point, it re-populates all 3 slave traders, so generating 15 or so at once. Unless you mean something else? In general though, none of this behaves really any different late game vs. early, if it works fine with 3,385 now, it'll work fine with it end game too.