I saw one of those 'how to fail at' videos that DragonSkies posted before. Forgot they're pretty good.
I'd say easiest way to start is to think about what some of the most effective horror games/movies/experiences have been for you. I'm not saying copy them, but take a while to think and analyse why they were so effective. Mechanically, how did they work? What was going on?
Speaking of which, what were some of the most effective horror games/movies/experiences you've had?
I myself, have played and watched the classics and respect how they can create such a response from you by skillful arrangement of technology. But I can't handle much of it....