Perhaps switching to an orthographic camera would work. Would have to test to see how it looks and plays. (not confident on this option)
There are int versions of the Vector methods that may help. ( Vector3DInt ). Haven't tested them in this context.
How about having the village 'health'/completeness be a combination of the village's construction level and the amount of damage is done by the creatures. The village would start out at a minimal amount and get more complete as the game goes. The creatures could do damage as you already have.
For the village, you could have buildings being built near the fence, enough for the player to see something is happening. Then the next time you pan the screen you could show the buildings on that row/lane. When the creatures go over the fence the construction sounds would stop and you could occasionally have sounds of the villagers screaming and running away. Then a moment later the construction sounds start up again.