I think, as has been suggested, the farming be handled by an assistant you can hire and pay a salary to, or for free by your significant other if you have one. Keep up the good work. Its been fun so far.
Edit: A paid assistant could also have the added benefit of lowering the crafting time for raw materials like smelting ore or tanning leather.