Hmm, that's really weird. You should be able to just walk into the cave entrance after giving the pickaxe to him. (It should spawn to the right of where he was and the question mark should disappear)
Would you mind recording what happened if you can replicate it? If you do you can send it to me via Discord (pastruvium) or through e-mail (pastruvium@pastruvium.com)