In that case you could deactivate the events till the clues are found and then activate them for either input system. While it is easier with mouse, running around and randomly hitting the action button is possible with keyboard too, if you allow for things to be found without clue.
Input devices are not equal. All have advantages and disadvanteges in comparison. A touch screen vs mouse for example means you can have several pointers hovering over the screen instead of one cursor to move. That's almost cheating in a whack a mole game.
For the mouse movement/action input you could also check, if the interaction-object is "visible" or "hidden". Hidden objects could only be activated while already standing there. Finding a clue would make the objects unhidden. I remember games where you could dig up treasure and such, but that was a disctinct different action than move/action. You could still brute force it, but it was basically impossible due to map size and time required for a dig attempt.