I think it was meant to be that way, you have to go up through the dungeon and loose powers due to that
