It seems that happens when you are fishing too eagerly and the fish get frightened out of their screen space.
