I would consider outcome not improbable. Let's say you have a settlement surrounded by 4 infantries and the settlement is defended by a single infantry. If now one of your infantries attack, the odds on your side are attack 3 + 3*1 support = 6 against 4 defense *2 terrain = 8 . So your success probability is only 43%! If different unit types or counts are involved, the individual numbers need to be adjusted, but generally infantries within settlements make formidable strong points. Unless you manage to provide some artilleries to support your assault from behind...