Some only spawn in view. It used to be that they spawned outside it, but it was a little annoying.
Now only 1 does, and it disrupts your radar.
You also have to rotate the cam for more gems, so you can be put into a position where they spawn in the opposite end of your view that you want to turn. which I was fine with.