You're a lot farther along than I am (I left my 'time machine' back in Trimps) , but at the 'just recently unlocked towers' point I wouldn't advocate waiting that long.
Even a few of a new type of heirloom make a massive difference, and there are some to be found in the towers. Add in the mild bonus of the low hanging attack/health crown-bought boosts, and I think it makes sense to burn a few keys pretty quickly.