Hey,
From what I understand things other than RPG maker are allowed, but how close should the game be to a RPG Maker-made one? Should I try to immitate for example Ib's mechanics as much as possible while using other engine or can I just go fully off the rails (for example do something like Amnesia Dark Descent?)

I feel like trying to reproduce RPG Maker feel in other engine will be a very fun challenge, but I want to know I'm not breaking any rules.

As long as you follow the ANTs guidelines everything else is fair game! 

•  Atmosphere: pixel-style graphics with mood-setting sounds (or lack of)

•  Narrative: creative story, plot, and/or world that often has multiple or twist endings

•  Tone: surreal, "off", mysterious, or creepy tones are common. Horror takes many forms, use your imagination!

You can make a fan-game, try to copy game mechanics, or make something all your own as long as it follows ANTs :) Go wild

Thanks :)