I put the incense on my guys hand, but it just scares them away completely they don't even try to interact. How do i get them to interact and i stop the encounter?
You need to figure out what ghost it is, and then use the correct banishing tool to KO them and then bang them.
The 'How to Play' guide shows you how to use each banishing tool, because they're all a little different. Sometimes its just waiting for them to spawn and pressing F, sometimes its more involved.