I went from version 6.1 to version 3.1, and am missing all the characters from before I think version 2. There's no school girls, shop owner, house wife, etc. I do have the Willow City characters and the maid/cat maid but that's it.
I think you need to start a new game and see if that helps.
