This is an intentional mechanic, although I agree it can feel a bit unintuitive, I've added a visual indicator for this interaction now to hopefully fix that.
