Are you using the die that allows you to gain a card? It also could make you loose a card and loose a point if you roll it. I find that die to be essential to win the game.