Try several times, you should be able to enter it at a moment (don't know if you have to wait for a certain hour or not)
Edit : I just checked the guide, and they say that you unlock the forgotten temple by accepting the quest "Tame the raptor" from the witch at night, but don't think you can enter in it without exploring some times