If you refuse to participate in a scene it doesn't happen canonically and there is no impact on the story. Characters won't reference it in dialogue. Scenes can be replayed by visiting specific characters at specific times of day if you change your mind later.