If he doesn't you missed the flag in Chapter 4 where you visit the mansion and tell Yukinari's older brother that you are dating Yukinari. In order to get this option you must hide from the younger brother first.
Just as a heads-up to anyone else reading this thread later, this isn't wholly accurate, as it's possible to get this flag by talking to the younger brother, too.
Also, I'm working on a comprehensive guide for the game, but it probably won't be done until a little after the holidays... Hang tight!