That happened to me too, lost like 3 from haato and 2 from Hikaru. I'm pretty bummed out about that, tempted to replay hikarus route just too 100% it again but all the chibi cutscenes and the day transition scene take too long for me to go through with it.