I think you made the right choice, adding too much farming into the mix would make it too Harvest Moon-esque, and I like the idea of running a workshop rather than a farm in this instance.
The suggestions I've seen below about having an assistant running a farm while you reap the benefits seem like a good direction to go though.