Honestly yeah the memories thing is a bit messy but it's simple once you get it: They are set in the past and you could watch them when they pop up, because they will be referenced later on in the story.
The VN really is pretty vague, lets hope answers come as the plot develops.
Though everything else in your comment kinda feels like you tweaking over nothing.