Well there is a NPC under the blue rock straight up from the start it can teach you a skill that will let you remove the rocks faster.
I did aim for it to take about a day or two to finish.
I think you are right about a sense of progress though, it would probably work better in a bigger world where it can ramp up more slowly.