Maybe its just me but it kind of feels like getting perfects has more of a tendency to slow the pace down if your a little late on it but not to speed it up if your a little early
That is correct.
There are basically 2 frames of "perfects". One keeps the exact pace and one slows down a bit.
You are not meant to get them every time, it just incentivizes you to maintain a steady rhythm for a bit until it is time to change. If you could speed up while still scoring them, that would seem like the "correct" way to play (it would not be)