Very interesting take on the Vagabond. For infamy you have to discard a matching quest card to score the extra point, is this to limit the total infamy points you score? That way you'll run out of quests at a certain point and they wont always match the clearing you're battling in?
For the item abilities do you have a place to keep items before you add them as a building? Is there a way to refresh the items on the board, or are they one time use? I imagine it might be hard to keep them as they could be an appealing target and get removed permanently if destroyed. And how do you keep track of which vagabond's are exhausted or not, do you flip them face down?
I like the aspect of letting each Vagabond take the actions of their listed starting items, that's an interesting way to give them some flexibility but still make each of them useful at different times.