If you want to get to the bottom of it, the most likely cause is some resource/class/object that you load or instance, maybe something you do every time you switch between scenes. If it's in your own code, you're looking for a logical error like a list you never clear.