it doesnt have a negative effect yet, i really hope it never does but it might eventually.
also don't apologize for asking questions, unless they are obvious ones the game states specifically and often. which I don't think this is not in any way shape or form really mentioned in game lol
make a harem of lovers as there no downside to it