Why can you even say no to the guardians? Even if you don't care for that particular guardian, you still get a buff for each guardian recruited. So there isn't really a reason for there to be a no option.
Yeah, it definitely feels a bit pointless, especially in the later stages. You might not have noticed, but when you choose to recruit a guardian, you don’t actually destroy the planet. Destroying a planet gives you extra DP - it’s a small amount, but it’s super valuable at the start of the game. Plus, you get the added bonus of upsetting the guardian, which I personally really enjoy :D