In Chapter 3 (I think that's where the Asher-in-the-garden scene is), Yren is just in a different spot. That moment isn't really based on your affinity with anyone. You just encounter different people depending where you go.
The garden is Yren's spot usually but it's also a comfort spot for Asher, which was why he was there and Yren was somewhere else.
Chapter 5 is a complex chapter and there are multiple ways to both get onto someone's route and be off-ramped from the romance paths. There are several checks and choices through the route that determine how things end up.
If you have the opportunity to touch Yren's cheek, and you do so (IE you end up initiating a 'soft' moment, even if it's brief), you should get the opportunity to speak to someone that evening in the lounge. And speaking to Yren should keep you on his path. You're just not at the highest possible level with him.
If you are off-ramped before you hit the kiss/cheek touch moment, you actually don't get any options there. It just skips past that moment entirely and in the evening, you won't be able to talk to anyone. It's difficult for me to say for sure what the situation was by the end without seeing screen caps.
If you had the option to talk to someone in the evening in the lounge and you chose to talk to Yren (after being on the planet with him) then you should technically be on his story path and the game should still be tracking your affinity with him though.