I understand your concerns and have already addressed some of them in the later versions of the game.
The Explore button gets faster as you play more in the game. The battle sequences are an integral part of the game and are not skippable. Your character gets stronger to a point where you go through them much faster.
I have played the game from the beginning multiple times, and I'll always improve and refine the balance.