I recommend adding coyote time, especially because the way the character behaves with ledges makes there functionally negative coyote time.
Yeah, it's not ideal- The character controller needs a lot more development to polish it up. Thanks for giving the game a try!