The big reason this harem story works as well as it does is that the harem is a manageable number and the stories are actually all interconnected. Lucy and Victoria are the two not directly tied to the founding families but are the original ties to the other characters and the MC's rejection of that world. Time and attention is given to all characters fairly and the intertwining of the interactions between the girls keeps the ones not getting direct attention at this moment from being forgotten characters.