I'm running on a Ryzen 5600X and CPU utilization is low with none of the cores maxed out - so it's clearly a bottleneck somewhere. On further testing I've found that it only becomes unusable when working with prefabs.
If I drop a prefab into the scene and unpack it, I can have over 1000 characters and it's super laggy but doesn't lock up the editor. If I leave it as a prefab every change takes 15-60 seconds (Unity pops up the "Hold on" window with STM's OnInspectorGUI and Unity's save scene and import assets taking the the longest). Multiple changes also get queued (like dragging to select a color from the color palette), which can lock up unity for several minutes at a time.
I tried creating completely new prefabs to test this and got the exact same behavior - it's at least 20x slower when working with prefabs. It doesn't matter if I'm working with a prefab in the scene or in prefab mode.