Oh, now I got the root cause of the issue: I am playing on a laptop's touchpad and sometimes tap the right bottom corner of the it which is causing a right mouse button click. I guess the easiest fix is just to add the same behavior on the rmb.
Ahhhhh. An easy enough fix. Thank you for identifying that!!