Each type of mission (warehouse, lab, and mansion) has its own set of H-passives. If you lose stats, it's because you obtained them all in that type of mission.
I appreciate the info! Would like to say also I've been enjoying the game quite a lot! It has been a breath of fresh air to me after playing so many RPGM games with medieval fantasy settings. Look forward greatly to future updates and content 😄!