I agree with you. In my mind a spawn rate of something like 75% adults, 20% teens and 5% children for bandits would feel more realistic. I think children should spawn as bandits though because they too can have children.
Other things to consider would be that children should very rarely spawn as a random lone villager wandering and instead should spawn significantly more often in groups of travelers. Same logic for teens though not as dramatic as children. Their spawn rate should be a variable that increases the closer you get to a village/city (for civilized races).