Let me take your example to explain the probability calculations. The settlement (factor x2) is defended by an infantry (defense value 4), resulting in a total defense value of 8. The attacking infantry (value 3) is supported by two artillery units (2 times +3), yielding a total attack value of 9. Taken together, in 9 of 17 (=8+9) cases, the attacker should win, whereas in 8 of 17 cases the defender should be successful. The defending city may get additional support by adjacent infantry or near artillery units of the same party, even if they are currently invisible to the attacker. So in this example the attacker should have a small advantage of 9 vs. 8, but only if really no additional opponent units interfere. While the random algorithm is really unbiased towards any party, seemingly unfair overall results are not so improbable...
The debriefing statistics provides a few insights into your relative good or bad luck: If the length of the battles one bar exceeds the length of the odds bar, you were favored by luck in this match, otherwise the opposite holds true. The absolute odds score corresponds to your tactical skill. If it is greater than 0.5, then you had a tendency to fight in favorable situations, irrespective of the actual outcomes.