Basically as you zoom in the original map is gradually distorted by a small-scale perlin noise. Past a certain zoom level the result is stored and gets distorted by an even smaller-scale perlin noise. And this indefinitely
The distortion is not destructive in itself i'm just gradually modulating the base image with a perlin noise, so there's no real adding & substracting, just interpolating.
There are inconsistencies though as you noticed, they come from the moments where I store the current state and use it as a new base image. I should have offset the perlin noise based on the exact zoom position where the snapshot happened but I didn't. This makes the distortion vary depending on the exact position of the previous snapshot.