I think I finally found the cause of this error and fixed it! https://rahisaurus.itch.io/rahituber/devlog/950908/update-2025-05-23-v1392
Awesome! I think Stretch has this bug too (I peeked at the code and I saw that lines 4117 and 4118 in LayerManager.cpp still use floating point precision for stretch, but I don't know if that's it), but with that turned off it looks like it's indeed fixed! Woohoo!