uh thats bugged, you should not have that past the fight, maybe try sleeping at the inn will fix it? But do that on a new save if you have a file from before the fight
I didnt have a problem with the oni twins, sure i could have done the fight faster but i managed to do it without dying and i think without losing spirit.