Thank you so much!
It seems the ladder bug has appeared again.
For me when I tried to replicate it, I could go up and down the ladder fine on keyboard, with or without the sword.
But only had issues going down the ladder with the controller when I had the sword.
I'm looking into it again, I'm unsure why this keeps happening.
Thank you for the feedback, I'll work on it! <3