Sorry, there is no way to avoid going with Oliver. The game would branch out too much otherwise.
But then why is there an option? :/ Oh well.
Cos if you decide to stay it can make you close to Oliver. And not all the options have to lead to a live changing decisions. Some can lead to nothing important
