Yeah, the toggle actually sounds good. What I suggest is having a voice say that an action is done (like say you watered a crop) and then have one for navigational purposes that's actually a toggle. I also think you should still have something that tells you when you're entering a new area (town, other areas that aren't the farm or town, and then the farm itself) and just have the game say 'you entered the farm' or something similar. You could even go the shorter route and say entering farm too to make things easier.