Alright, I added visual feedback, and even I noticed the difference in the game!
Added a super simple note system where you can circle the numbers you have heard, nothing too special.