You gain attack Runes over time (It's a cycle, so you always get them), and getting the castle too early can screw you over for the exact reason you said- which is intended.
There's also a Rune you can get that will increase the Charge Speed, but it's not needed, just helps.