...yeah that's WAY too OP. And you're right, the memories thing doesn't really tie in. HOWEVER-if you wanted to make a more sort of-magnified, different version of Shinsou's quirk I may have a suggestion.
Ok here's my idea:
Emotional Manipulation.
Rather than having direct control over a person's actions, he has control over a person's emotions. Basically, by having them engage with him in anyway-easiest way is asking a question and having them answer it- he can gradually magnify or warp a person's emotions to affect their fighting style, maybe even causing them to turn against their opponent. It takes as little as a 1-word interactions to have the control begin. The more they interact with him, the stronger his control becomes. However, heightening someone else's emotional energy exhausts his own, making him feel both physically and mentally drained. The more people he controls or the greater the emotion he's controlling, the more drained he gets.