Indeed, car physics is a lot harder than you might expect. I tried using wheel colliders at first, but eventually decided to not use the built-in physics engine to apply forces. The wheels use physics materials, though, which are helpful. It's a lot easier to control that way, but then it introduces other issues like collider clipping that you have to account for.
Viewing post in Untitled Car game jam comments
if you want to use wheel colliders then you have to be very exact with the setup you are using in unity. And tbh wheel collider is like all in one thing, suspension,tire grip, moving the car stopping the car etc. If you want kinda like a smoother and relatively realistic driving experience you should try using wheel colliders. Alsooo the wheel colliders actually don't need physic materials. Everything is kinda self contained. I never had collider clipping issues with physic materials but I never explored them as much so am not sure too.