You will know you caught a fish by the crunching sounds they make. After a few captures, you hear some bubbling and you level up to a new place with even more confusing sounds XD
Among all the chaos of underwater ploosh and blips, as you turn to the sides, you will hear some percussive sounds that repeat. The louder you hear them, the more centered they are. Move towards them and their beat will become faster and faster. Get close enough and you will hear the crunch of having captured them.