Thanks! Only thing I can say right now looking at the code is that it seems the current blending is depending on the scale of the image, instead of the outline width. That is probably why if I apply the outline after the existing transformation or before it changes the smoothness of the border. (my characters are zoomed out so the scale changes quite a bit). I still have to try the proxy transformation. That might solve my issues as it applies the outline transform after my zoom transform.
Oh I think I understand what you mean now - yes order of operations matters. If you apply the shader to a zoomed image, the outline won't be zoomed. If you apply the shader and then zoom the image, the outline gets zoomed as well. You can see the property order here: https://www.renpy.org/doc/html/atl.html#property-order So if you want to go out of order, you'll have to make an image that's got the shader already applied and then zoom that, for example.