Middle of streets in a cramped area, and yes I have tried doing the regarrison
I'm afraid that could be related to a bug around warbands getting stuck while pathing. An enemy warband wants the center of its formation to be in your town, but the warriors can't actually get there. Your warband is trying to fight the enemy, and is squaring off with where the enemy wants to be.
Unfortunately there's no fix for that yet, sorry about that.
A potential workaround would be to garrison that warband in another settlement, which would put them out of aggro range of the enemy warband's supposed position.