This happened to me as well, but only with Arthur.
I was suspicious when Arthur gave me the Book of Memories and it wasn't physically accessible in the house except with the button on the side (unlike the chest and bed, which you can see in the house and click on directly to interact).
After getting the desk, I was told Arthur had a side quest for me, and of course, he gave me the same quest again.
But all the logs did not respawn so there's no way for me to complete it.
It didn't take away the Book from my house but it just bothers me that now the Log icon is stuck over the Feasterra section of my map screen.