That's fair, we had to rush a few parts of the game near the end so there's definitely not clear directions as to what to do.
If you are to give it a shot again, or if anyone else looking at the comments is confused as well, in the starting island you're supposed to interact with the villagers and then enter the ladder like tile which will take you to the caves. This order will be the same for the next 3-4 worlds as well.
The main concept is to go to every world's cave, get some of the ores while fighting away the monsters (you can make weapons using these ores, just talk to one of the villagers) and one of the ores will be yellow and glowing differently. This will be the ore required to make a key to open the portal, which should be positioned in one of the other rooms in the cave systems. In order to make the key, you just go grab the glowing ore, mine it (with any weapon by just hitting it), go back to the villagers and one of them will be willing to make you a key, after which you will be able to enter the portal (will need a bit of exploring of the caves in order to find it).
Every worlds villagers have a different weapon to offer to make, so I would definitely suggest getting some of the normal ores and playing around with the weapons as they are quite fun. You just have to switch among weapons using right click.
That's pretty much it. I hope you give it another shot, and we will definitely make sure to add more direction to the game so the player wouldn't be as lost. Thanks for the comment.