If you let everyone in then you let a doppelganger in so that's why you are not getting the achievement.
Ah thank you! Weird I didn't get it the first time I tried.
