That is correct.
There are basically 2 frames of "perfects". One keeps the exact pace and one slows down a bit.
You are not meant to get them every time, it just incentivizes you to maintain a steady rhythm for a bit until it is time to change. If you could speed up while still scoring them, that would seem like the "correct" way to play (it would not be)