Oh, definitely not. I think it just depends which style will better fit the vibe you're going for. I personally am a huge fan of a hand drawn look, but I'm not the only audience for your game, and I'm a weirdo; some nice color pixel art would probably be more conventionally pleasing.
Ultimately, I think it oughta be your decision. You know what looks good and what fits best for what you want to make. I wouldn't think either choice would push people away.
Sorry to cause any anxiety! I like to think that there are no mistakes with something that subjective. (Unless you're trying to make money/make a living. Then there might be, haha.)