very easy to fix. You simply have to take a follower with you, preferably someone from the Royal Court.
So, you probably have the power that allows you to teleport back to Selvanna. From there, go back to Lumis with the teleport sigil, sleep in your bed until the next day, then go ask someone from the Court to be your follower. Take the follower with you until you reach Death, and do everything as usual. The sex scene will begin after, and once it's over, another sex scene will happen between you and your current follower.
As for the memories, it's totally normal. I do not have access to them either, no one does.