I wouldn't say you arent supposed to. some spoilerage ahead.
There is a place where you can leave the fence area and are partially encouraged to do so regarding a specific kind of item to collect.
