I like this idea a lot! Farming games and action games are both fun, and this is a cool way to mesh them together. I wish there were more goals-- Maybe if there was rent that you have to pay, that increases over time? Right now, you can just focus on fighting, because growing crops only helps you grow more crops. Second of all, it would be cool if instead of using X to switch between fight mode and plant mode and using Z to fight or plant, I think it would make more sense if one button was for fighting, and the other was for gardening. You're already using two buttons anyways, I think it would flow better.
Nice job though! Fun game.