Ah, the journal (map thing) is actually completely broken, so its not just you. I thought I disabled the journal but I guess not!
Being able to quickly ascend platforms and ladders is somewhat intentional, but you're right, a cooldown before you can hop off the ladder again might make some sense. Thanks for downloading and commenting!