I'm nearing the end of Epilogue and I can't imagine how the story could change when you excluded some family members. Especially the one I lately skip all adult scenes.
I think that HaremRoute is the only way to read this novel. Skip the adult scenes of unwanted girls to not ruin the whole story.