Thanks for the feedback!
I think I got carried away trying to add new features and explore different nice-to-haves (like ragdoll <-> bone animation transitions I didn't end up using too much of), where I could have definitely refined the physics more.
I did a refining pass early on that made it feel really responsive and less janky, but something I did later on re-introduced the jank (I think a lot of the jankiness could be fixed be re-disabling object-player collision while holding an object, and capping object travel speed).