If you go back by foot you will do what you used to do before leaving, that's a great feature. But if you go back using a portal you will be put to Rest activity.

As a side note when you recruited Kurdan he will go to your manor by foot. If you used portal you will arrived one turn before him. Maybe a remark on his dialog could be nice. Or don't let him go back alone automatically.