I like the idea of using the wandering characteristic of NavMesh agents and combining it with waypoints. It makes the AI more natural movement and reduce complexity. Also, I suggest to use two rays and using Raycasting for the vision of the villagers instead of using the colliders. Good game!