If you remove a card from the fort through banishing or destroying, on your next turn (i.e. after you have finished 'taking stock'), you will play the next card to the space that's just been emptied. On the turn after that, assuming no other cards were then removed, you would return to wherever you would normally have to play the next card in the fort, if that makes sense.
Does this answer your question?