Another thing you might try is some 'coyote time' when walking of the ledges? Could be worth the experimentation at least? : )
I actually noticed that jumping right after walking off a ledge was a bit difficult, but I had no idea "coyote time" was a thing! I just added it and it feels much better. Thanks a lot for the suggestion! :)