Some people are stupid.
If the girls in the harem are playing with each other, it's completely fine.
I think the silent majority agree with this.
Either way, just make what you want to make.
