You gain Fatigue when using special skills, casting spells, and through some enemy attacks. Once your inventory is full, you can’t gain any more Fatigue or use abilities that cost Fatigue.
You can choose to fight with no armour or weapons but you’d only do 1d4 damage (unarmed) and you wouldn’t have any special abilities. You can absolutely choose to run a so-called speedrun build (naked with one weapon) to maximise the amount of special attacks you can use.