The latter. See the beginning of the “Chapter 4: Magic”:
Spells require a few minutes of uninterrupted calm and attention to cast, as well as requiring a set of detailed gestures and incantations. Consequently, Spells are generally impossible to cast in combat.
So, while not in combat, you can cast any Spell from your Tome, though in dangerous environments this will prompt a random encounter roll (see “Chapter 7: Hazards and Obstacles”, Random Encounters):
When characters explore, rest, cast unprepared Spells, or hesitate in a dangerous place, roll a d6.
And, of course, you can switch your prepared Spells “setup” at any Rest.