Tbh, considering his backstory, it really shows how much he was latching on his love for you, and hasn't banished you due to that or for being a hero despite everything, but i'd bet his heart aches every time he sees you after the events.
Aside from that, it's by design one cannot save everyone or get everything in one route and has to try them all for the full experience, lore and different rewards.