So my raving fan follower isn't growing past like 320 size am I missing something?
Shouldn't be... perhaps you're not getting enough exp to level him up? Or you need to reset and adjust his stat points, just don't leave any on the table or it kinda bugs out and you lose them. If you need to you can give your followers the essences and that becomes permanent just like it does to your avatar. I didn't have problems with him in v4.5 or lower except a couple bugs here and there, but he's like level 800 on my current playthrough and had him at like 1000 back in v3.9