Hi! The "spend time with..." actions don't affect which random scene you will get as of writing. They increase affinity with the chosen LI, which is used to choose some scenes (but not all) and, most importantly, the ending :)
Ohhh that makes a lot more sense. I went back a few times thinking I chose Cedric by accident x'D