I talk to him and in the end he does something with Celica, but I ask the fortune teller again and he says to talk to the mansion's servant. I'm really confused.
Try talking to Vivi more than once. You can also just fight monsters while wearing the armor.