I picked inner magic, just because it felt like what my Crown would be most drawn to. Beneath this comment is a screenshot detailing what inner and outer magic is about, in case you didn't know.
SPOILER
And yes, it is possible to become friends with a romantic interest whose specific route you didn't choose. (From Chapter 5 on, you will leave the Common Route and continue the story with a specific romantic interest in the main focus; keep that in mind.)
Regarding the choice concerning X that you mention: You don't throw away the connection if you choose not to find them attractive in that specific instant. But you should know, towards the end of Chapter 4 there might be options that you can't choose if you neglected your connection to X too much.