Thanks for the feedback! The janky rotation when it's returning is not intentional, I couldn't figure out a way to solve it, the issue is caused by the tween still rotation to return the dial and being forcibly stopped by the user holding it, as the behind the scenes is still decreasing, it leads to some odd rotations.
And yes, treating it as a real one and going with the cursor on the holes is the best way to go. There's even some 'skip' moves you can do from numbers 1 to 5 where if you drag them straight to the stopped they'll rotate just fine, so you save a lot of time there.