I can’t seem to figure out how the capture mechanic is supposed to work. I have bought shackles from the market and used them on invaders in several situations, but they invariably escape. Is there a precondition that has to be met for them to work, or do I need to do something after they are shackled before they escape?

Invaders can always escape, but the weaker they are, the easier to capture. Try to lower their stats a bit before trying again.

The only required precondition is having enough Jail slots available.