I think rather than overhauling the entire system it might work nicely to have many items that currently grant additional speed points to instead grant +1 (or -1 technically) to SP regen speed. Imagine in the extreme case that there are no, or very few items that grant raw SP and so only the most absurdly DEX specialized character could ever get more than 5-6.
I'm also somewhat uncomfortable about speed points when they are more than around 5-6 for the same points your making i.e. stopping time just feels somewhat odd once its more than 1-2T and also the crazy long time to regen them (I often forget to rest with SP heavy builds).