There's two side quests that get you the pass. One is a "Collection" thing from the male hero and the other is with the purple haired girl. Unfortunately, there's no way to "lose height". Growing too fast can cut some side quests and such out.
