It sounds like everything might be working as expected -- this type of equation for tire forces is undefined when a wheel is at rest, because it is defined by slip/deformation. Under this model, a relaxed tire never really rests; it oscillates between gripping one way and another.
I recommend a minimum physics tick rate of 120Hz to mitigate the inherent error. Shorter intervals are better, at the expense of CPU. Then...hiding/clamping the fidgeting by brute force. The latter is a programmer's art. I owe a devlog to explain how I've addressed that problem, by request.