Thanks for playing. The physics took a lot of tweaking, but turned out a bit strange in the end. Turns out unity doesn't love it, when there are both huge and tiny physics objects in the scene!
Yeah, Unityâs physics engine doesnât like fast moving objects, had problems in other project too :D