The distortion is not destructive in itself i'm just gradually modulating the base image with a perlin noise, so there's no real adding & substracting, just interpolating.
There are inconsistencies though as you noticed, they come from the moments where I store the current state and use it as a new base image. I should have offset the perlin noise based on the exact zoom position where the snapshot happened but I didn't. This makes the distortion vary depending on the exact position of the previous snapshot.