Finally found it! The one decision i kept making was accepting Summer's advances after meeting Ise for the first time. Either that or shutting Summer down from ths very beginning. Turns out i had to acceot Summer's advances at the start and then reject her when i come back home after talking to Ise.