Hello Jacob. Cursed rooms are designed to be a limit for your character to not get into very dangerous areas. You can un-curse them by getting that skill with Adventure Points. Also, remember that beating the totems is just to make the final fight easier, but you can face the last boss whenever you find the key to get to it, so destroying all the totems it is not "necessary" to beat the game.
Hope that helps! We will try to make this things clearer via tutorials and better text messages in-game in future updates.