Interesting idea. Although I would have like a different approach more.
Maybe instead of keeping the number the same after every bounce. Change it. Maybe the first ball always starts with 1 ( or a random numner) and after every bounce the number increases. This way you have to think about the next balls number. Is 42 a prime? Is 43 a prime?... This way the player is forced to continue to engage with the prime numbers. This way you prime balls would also not be absorbed but be played back like a normal ball.
This would also solve the issue with the player not having enough time when a new random number appears. If the ball goes to my side. I have less than a second sometimes. Feels a bit to random for me.
Great execution!