Refusing to pay the 1000 gold to the spirits results in you being haunted.This can subtract gold and happiness from your kingdom.
Sorry but i have payed to tree kingdom,angry spirits, and water people (forget their proper name) but this didn't work and spirits still haunt me,
and after sometimelater they still wanted money, and i accept it but it didn't work
