Ah, i didn't consider that.
I may keep it on since fall damage seems to be inconsequential outside of these specific cases. (Not many places to fall from)
You can find two items that help reduce the damage from falls. I will not name them in order to avoid spoilers. Just tell me if you want to know.