The game counts both adopted and non-adopted cats the same, so they’ll just stop entering at all. If you let a kitty leave, another kitty will come to replace them (as long as you have food out), unless you sell toys in the meantime.
I’m changing how it works in the next version, so instead of having to buy more toys, you can buy permits to allow more kitties, directly! and it will show exactly how many kitties can visit at any given time so you won’t be left guessing.