Go to your home, click the little tool icon at the bottom, and to the left side there'll be a horse icon. From there you can train the horse to let you ride it (have carrots? on hand). Afterwards it'll give you an option to bring them with you in the same menu.