I'll chime in and add some of my favourites to those already mentioned. There are pure VNs as well as games with RPG-elements or stat-rising system.
Sorry for the language, I'm not a native English speaker :) So, let's start:
Amnesia: Memories - You play as a girl who lost her memories because of a spirit who somehow crashed into her mind. With his help she has to find out everything about her previous life while hiding her amnesia from everyone.
I'm more into the western VNs than the Japanese ones, but somehow I couldn't pull away from my PC unless I finished all the routes in this one. It was long ago that I played it, but the plot didn't seem typical, especially the so-called true route. The MC might seem quite naive though, but her amnesia explains it I think. There are some really creepy moments, not many though.
Freak-Quency and its sequel Amplitude - I won't separate them cause I'd recommend playing them one after the other, many things will be clearer that way. The games have different plots, but the same world, so don't blow up your brain like I did when I played Amplitude looooong after I played Freak-Quency and forgot almost everything lol. Both games are more story-focused, not romance-focused, and the story is really great. Rather dark in some parts, and hilarious in the others. A nice mix if you ask me :)
Cardinal Cross - a really exciting science-fiction story of a great scale + drama + complex relationships between the characters + LOTS of morality-related choices, where there are no really right or wrong, good or bad answers. Enjoyed this one VERY MUCH, though it is also more of a story-focused VN, not your typical otome at all. Reminded me of Dragon Age RPG series with the 3 types of MC's behaviour: kind, ironic, and evil. That concludes the likeness though, lol :D
Dandelion - Wishes brought to you - - A Korean game, that I played looong ago, but it still has a place in my heart. It's generally lighter than the previous points of my list, but has its fair share of drama closer to the endings. Though for the most part it is still a light comedy about the girl's pets turning into handsome guys. So if you have several free evenings - go for it. The game features a stat-rising system, which I heard many people dislike. As for me, it was a bit tiring, but only in the end.
Girlish Love Revolution - A game about a girl who weights 100 kg. and we have to help her to get slimmer and find her love. Its great and hilarious, features an interesting gameplay. The only drawback - the English patch is made for DS. I played it through emulator though.
Magical Diary: Horse Hall - A magical school simulator that I re-played several times. Some school system-related details are inspired by Harry Potter, but the game has many unique features, fair share of interesting plot twists, and GREAT GAMEPLAY (really, I can never get tired from the puzzles that have looooots of solutions).
Nightshade - A VN about shinobi set shortly after the Sengoku Period. I reeeeally enjoyed it, and it is especially great that each character route is unique and opens up a story from a new angle. Some of the routes are angstier than the others, and when I say ANGSTY I mean it. I cried like a baby during some of the routes.
Rock Robin - One of my latest favourites <3 It is half VN and half a band management simulator. Both parts are really cool though. The gameplay is not boring, and as for the story, it makes you greatly emotionally involved, that I think is great. The MC is...unique. Has a dark moments in her past, a difficult disposition and unhealthy lifestyle, not a typical otome MC, but still loveable, as for me. The drawback is, the game is still in development. The couple of routes are not finished, but the rest is ready and playable.
The Flower Shop - Winter in Fairbrook - A light-mood dating sim with the farming sim elements. A story of the girl who had to look for a part time job in a flower shop, and ended up finding great friends and (naturally) love. Great for gloomy evenings.
These Nights in Cairo - A fantasy otome VN about archaeology and Ancient Egypt. It is not long, will take an evening or two to finish it, but the plot and characters are great. Loveable and strong-willed MC. Though, I played it in Russian. There's an English translation, but I never tried it, so I can't say anything about writing. It is great in Russian though.
C14 Dating - An otome dating sim about an anthropology student participating in a summer archaeological internship. With GREATLY developed details (I mean it, lots of interesting archaeology-related info), nice mini-games and loveable characters. Played it several times, and I never get bored :)
Backstage Pass - An amazing dating sim where you get to play as a girl working as a makeup artist in the entertainment industry. She has to combine her college life with a career, friendship and love. I personally enjoyed it since the MC is not cliched, the routes are not repetitive, plus cool graphics and voice acting.
Unmoor - Another Russian VN (that has an English translation), that I enjoyed very much. Features pleasant visuals, self-determived MC, intriguing story, beautiful soundtrack, and interesting LIs.
Phew, I somehow got tired while writing this :) Hope anyone finds this list helpful to spend a couple of nice evenings :))))