Note: it seems to happen when I affect the environment. Using plants, water to create steam, or temporary spells don't cause the issue. Tunnel and ice on lava do make it worse.
Edit: interestingly the problem goes away the further I get from where it started.