That’s the intended behavior at the moment. The offset isn’t only a draw offset, it also offsets any hovering coordinates.
What I would maybe do in your case is put the label in a horizontal container, and handle the hovering callback on the container instead of the label (so the container should still have the same hit box)