Already pointed out, the aabandoned temple isn't fully fleshed out yet, so they are some interactions missing and the teleporter can't bring you to it and will instead take you to the neko village. You will have to wait for the next update for this to be patch