Yeah, it's Unity's Canvas bug, sometimes the button's hitbox is not centered on the text. I might do a director's cut update™ to this game after the submission period ends, and replace some duct-taped mechanics with more sophisticated stuff
