You can mine it under the tavern, once you've unlocked that area. The easiest way into it is to go into the tavern (so go in at night), but you can also get there via going underground in general if you're that far.
If you're not that far yet, then keep playing more until you unlock it.