You are missing a lot of events regarding the characters. Talk to the fortune-teller to get hints on that. There is a port near the Asayiah palace. You need to equip the spirit stone there after midnight and talk to the ghost that appears there.
