I think you're right, I'll just make the growth rate the same (or very minimally different) for all difficulties so the only difference is the speed at which you can collect points.
The second thing is unfortunately just a skill issue, I'm able to do what you said at 100% difficulty pretty consistently. But I guess I can increase the timers for lower difficulties.