Usually, you have to kill them by using distance weapons (shurikens and stones). They will avoid your attacks, so you have to chase them a bit.
Yes, they leave the board with paper (= escape with valuable items) but then they don't count in your "killed rats" counter.
Thanks!