It looks like when that happens, the balls lose all their speed and just float around slowly. As far as I can tell the only way to fix it is to by a new ball, at which point all the balls speed up to their correct speed.
After a while of idling, 90% of my balls are just crawling around on the screen and progress stops. It makes it so you can't run the game idle, you have to always be playing.
I also noticed that when I buy speed upgrades, they aren't actually applied until I buy a new ball (doesn't matter what type).