I see what you mean about smooth automation, I was curious and tried FMHeaven and Toxic Biohazard and I hear some stepping artifacts when automating the mod matrix - I guess they do not interpolate smoothly on a sample-accurate basis (if those are the right terms - I have no idea what I'm talking about). Personally I don't notice the artifacting or aren't bothered by it. Example where I modify one of the modulators to switch between harsh and soft tones: https://clyp.it/dgwzbzpj.
But in the realm of FM trackers, it is common to see FX commands that modify the current instrument in real time (Operator level, feedback, Multiplier). VGM Tracker, Deflemask, BambooTracker and FMComposer all have this capability. And obviously it only modifies it on a per-step basis, so it's not intended to be smooth at all, but still allows more expression in instruments.