Yeah the power bar is scuffed. It's conveyed terribly, but there's an animation delay between you starting your swing and the ball actually getting hit, and the power bar isn't paused during that period. Basically there's input delay. It doesn't help that the power bar's value is raised to an exponent, which increases the difference in effective power that was annoying you. A lower exponent and higher base multiplier would have lead to a more consistent feel without removing the impact.
Getting to see someone play this in a video was really helpful, thank you for playing! The dramatic reading of the intro was also fun