Did you use the mirror (hold space) to detect ghosts and destroy them? They drop spirit flames that give warmth.
They will quickly drain your warmth if they aren’t defeated.
When you see a ghost in the cursed mirror, that's a reflection! Meaning, the ghost is being reflected. It's position in the real world is relative to the real you, the same way the reflected ghost is positioned relative to your reflection. Hope that made sense (you can't attack the reflection).