If you are not choosing flirt options, you will not increase your romance stats. For Irus, you lock in your romance only if you spend time with him on the balcony in Chapter 3 and if you don't, you're locked out of his romance because the romance lock-in scene which triggers by Chapter 4 when you speak to him will not happen so you're automatically cut off from future romance scenes with him.