This is actually really good feedback. I ran into the exact same problem. I ran around for over an hour trying to figure out how I was supposed to get into that tower and had to watch a youtuber to figure out that there were separate ruins in the town. I know they seem obvious now that I know where they are, but the fact that there are other ruins really throws this off.