totally makes sense, the 4 color puzzles are very hard! they get easier with time if you stick with them, but it may take a lot of hints at first. there is a discord where people can help you with puzzles if you want to try and get better.
if you get really stuck on puzzles, monsters bring you treasure chests too. they're supposed to take a few minutes, so it's normal to feel a little frustrated if you can't get the answer right away. but, i'm sorry if this makes the game less fun.