Oh!
Thank you so much. Whenever I've tried to detect both objects and tiles before in one check it would go completely wrong. I was checking or instead of and, which usually resulted in players that can't move at all.
Adding and removing tiles would work in most cases, but I like the destructible objects to have their sprites collision masks, and be able to be freely positioned. With tiles I would need need to stick to the grid(and tile size), and every object that has a unique collision requires me to change my tile sprites, which I can see being tedious.
It still helps performance, since the amount of collision objects is going to be so small.