Basically you have to follow Agnes' plans and such. If that doesn't work, then I'd guess there's at least a minimum affection number you need for her. I usually go all out and try to get and do everything, so I didn't have any issues with any of the three ending I got. Another spoiler, there's one with Azumi and one with, let's say The Director, to avoid big spoilers for anyone else.