That penalty is only there for your Slaves, unless the UI doesn't show it for the player character. Even if it's there, still hasn't stopped me from doing endless adventuring with a high agility character. Having finished the game I can say with confidence that the only thing I didn't manage to do was beat the last boss solo.
As to being Feared. In my last play through the only time I accumulated Fear was when I pressed a button by accident, the rest of the time I did pretty much what you mentioned, with the exception of putting them in jail until their Rebellious stat was down to 0 (as they'll steal money/food if they're not imprisoned whilst they have the stat above 0).