You’re correct that something’s wrong. But the imbalance is entirely on my end, sorry about that!
Actually I just identified the bug: In the synthesizer, most voices use two oscillators and I’ve only been shutting down one of the pair when notes end. The others pile up and uselessly eat more and more CPU over time. Oops.
Thanks for reporting!