The Marksman Rifle and the Elminster Shotgun both lower Agility by 5, so you're currently reducing your total by 10. Baseball Bat also lowers it by 2.If you want a more Agility-focused setup, you need to use less or lighter weapons.
I didn't even think that weapons would reduce agility, that might be something to make more clear in the weapon item description.
