I was actually just about to post about this until I figured it out. Basically the crumbling statue has two problems: one you need to optimize movement and two (more importantly considering it isn't that hard to find the quickest path) it is unfortunately also just a raw stat check I'm pretty sure. I had 10 AGI when I completed it. That was overkill, but I'm also not entirely sure how to do it otherwise as prior to grinding I had 8 AGI and couldn't make it by one tile but with 10 I had more than 1 or 2 tiles left before it broke (like 5 I think for some reason). I did start with only one part of the hunter's gear set on my first attempt at 8 AGI and then both top (craftable) and bottom (idk where I got it) on my successful 10 AGI attempt.
Things to test for less level grinding would be if hunter's gear as a set actually gives you a better bonus than they say they do (for the task at least) or if 9 AGI would be enough (as in only grinding either one level and no full hunter set or just the full hunter set without intentionally grinding to get more level(s)).