Thank you for the feedback!
We used AnimatableBody3D with tweens and physics tickrate of 240 in order to not have balls flying through stuff. Sadly, these bodies don't have continuous collision detection, like rigidbody does. We were really really close to not making the game at all while dealing with a bug related to these, but found a workaround and felt that the physics in most cases worked good enough. One thing is that the ball often rolled quite slowly, which was a bit boring, but we never got around to fix that. Trying another physics backend would have been interesting!
The magnet levels are hard, I agree!