You can't, but once you have the gems, you can go in and the shiny area in the cave will disappear. Before you have all the gems, when you enter the cave next to the badger den, there will be a shiny area that you can click on. Sorry, I should have been a bit clearer about that.
