I jumped on the big rock besides the one that is being held up by the small rock in the middle, where you can see the green key from, then to go back I wedged myself between two trees by the rocks that are away from the library and spammed jump until I was through.
I tried holding the yellow key with the right hand and clicking either of the doors. When that didn't work, I changed hands and tried again, then I started spamming left and right click while changing hands quickly and my right arm disappeared, leaving my left hand holding the key. I had the dagger equipped on the other hand.