I'm not entirely sure what to make of this, but official controller support will come with proper navigation. Using a controller as a mouse should generally not be necessary when this support comes.
But perhaps I can add some sort of menu toggle to fully disable controller input in the settings