Some of them will only leave if you start a fight with them (or another NPC does). Might also be a bug!
