Actually, I think you're completely right.
The cooldown wasn't really meant to discourage the use of quick switching.
I don't know, I was probably overthinking it when I added it and thought it needed some kind of downside.
There was also the issue that holding the button would stop time meaning that you could basically stop time forever, but I will just remove this as well to fix that issue.
So yeah, removing those two things will very likely be for the better, thanks for the feedback! ^^
EDIT: and the time limit is going away as well, since it is pretty pointless without those two things. ;)