It sounds like you've encountered a remnant of spaghetti code. You're not locked out of it by progressing through the story.
There's a switch that tells the game whether you've found the place and know what it is that needs to be flipped. Once it's flipped, you'll get a bit of dialogue every time you enter the map that'll tell you that you can work on it and roughly what to do. If you're not getting that dialogue and you've already been there, you might've just found the place before you had Phoebe around to tell you what it is. If that's the case, go inside the shack, check the chest (even if it's empty), then leave the grove and come back. That'll flip that switch and let you go about fixing the place up.
If you're getting dialogue that says to come back alone when you can't invite someone yet, that means I need to update the dialogue.