I tried again just now. It does seem to be a time thing. Must have only gone at night before, I went daytime and he answered the door for once.
Ah.. well glad you can continue now :)
Yes. At daytime (specially after 8ish or 9ish) he's available in his cave. But at nighttime, you can find him in the pond in the village.
