Unfortunately, it turns out that in the vast majority of cases the noticeable performance drops happen because of Unity's physics system, which is single-threaded on the CPU and out of my control.
