I just went back with high size and trampled the crystal walls blocking the path, not sure if there's any other way atm.
Spoilers: As of the time of writing this the only thing there is a unique sword and a golem you can rematch infinitely
