With ladders you're standing in the same location whether you are holding or letting go. Perhaps the problem would be if someone is holding the ladder then walks away. Most games the play space will follow you in that situation but in your case they should lose there grip and fall landing within the correct coordinates.