Are you interacting initially with mouse? If yes, then try pressing 'E' instead. Mouse inputs gets recognized by unity twice. Once it's pressed and when it's released.
Turns out it was a connected controller causing the use button to go crazy lol. you suggesting the mouse actually made me think about that thank you.
