Good game. 

For what it's worth I think you're making the right choice in transitioning to a VN format. There's something I've learned about recently known as ludonarrative dissonance and conversely ludonarrative harmony. The dissonance is what happens when the gameplay of a game don't match the tone, theme and story and so pulls you out of the immersion or at least frames the experience in a way counter to the expectation.

Having a bunch of combat minigames at important parts, I think, would pull the player out of the experience I think you want to create. The wuxia AVN CrossRealms does a good job at creating intense fight scenes because it can create set pieces and control the flow. I recommend checking it out if you haven't.

But! There's a way to use that dissonance in effective storytelling, such as when you want the player/protag to step back, evaluate and learn. I think using the sandbox for that would let you control the flow and pacing quite effectively.

So yeah, I think you're making a good call. Both in abandoning the sandbox and in retaining it for the future to serve your purpose and better control the narrative flow.