Starting with e.g. Physics 6 would not only take a bonus from a few classes but also it wouldn’t make sense when a person that barely know how to hold a sword (no weapon based class like fighter or archer) had a body refined as a highly skilled swordsman.
That is at least my opinion in this.