Because trashies don't recharge when not in your hotbar, I think it makes sense for you to be able to reselect whenever you want, because you still have to choose what you want to be coming off cooldown as soon as possible for your defense.
That's what I think should be made to be less important as a mechanic, to not turn a tower defense game into an inventory management game. I'm sure some people will find inventory management fun, not me though.