Please, clarify, are you accidently clicking on the blue waypoint?
Clicking on another piece/free cell must cancel the current selection.
Sometimes, yes, when using the laptop + trackpad. No such issue with a real mouse.