Glad you noticed the score sound was pages flipping!
I spent most of the first two days of the jam just tweaking that upright assistance. I ended up simply applying torque to the books while they are being dragged, based on the current rotation of the book. This turned out to make for a great feeling of an "anchor" below the book. I also offset both the gripping and the pivot point a bit upwards while dragging.