I have never made a horror game.
IMO atmosphere is one of, if not the most important parts of a horror game. There's a lot of tiny elements that can help add to a game's atmosphere, but the two most important things are lighting and sound. Sound especially, a simple droning track can instantly make a place feel far more foreboding.
I recommend that you play through a horror game that you yourself find scary, and see how atmosphere is used. Think about what subtle things make it scarier while you play.
As for making monsters, I personally think that the best monsters make me say "what the hell is that thing." A monster that has an uncanny design, makes weird sounds, acts strangely, and just feels wrong or off is super effective to me.
It is also my first time making one of these, and I'm personally gonna go to the side of making the player feel anxious due to the ambient and player expectation. As in my experience these have been the most upsetting for me maybe for you those are the jumpscares or the chilly story ones, making something that would scare you its the best but consider that if you only playtest it yourself you probs wont get scared knowing all that happens behind the scenes