Definitely need a 1st-person fly camera "ASDF+QE" - like what's used in Minecraft when flying. Unity uses this when right-click is held. Magicavoxel implements this perfectly, giving the option to toggle between Orbit and Free cameras. Just please keep the pitch control linked to the camera's X axis. Programs that allow true free cameras are just dizzying.
A recommendation for the current orbit camera: hold the middle-mouse button and move mouse to move the camera target left/right/up/down relative to the camera's current angle"Quaternion". This would allow the target to lift and drop, so this tool could be used better for structures with roofs. Lastly, in orbit mode, could use the same 1st-person controls to control the camera's target, while the right-button still orbits the target. Again, this is how Magicavoxel does this and it is the best control scheme I've ever used.
Update with some simple code:
Code for panning the orbit camera:
// If the middle mouse button is held down
// Get pan speed (distance between camera and camera target divided by 25)
float panSpeed = Vector3.Distance(cameraTarget.transform.position, mainCamera.transform.position) / 25;
// Pan camera, based on mouse movement, relative to the direction the camera is facing
cameraTarget.transform.Translate(mainCamera.transform.right * -Input.GetAxis("Mouse X") * panSpeed);
cameraTarget.transform.Translate(mainCamera.transform.up * -Input.GetAxis("Mouse Y") * panSpeed, Space.World); }
Of course cameraTarget is the parent of the camera. I imagine you have the camera orbiting around it. mainCamera is pretty obvious =)
The panning code is fairly standard, but I never liked the scale in speed of most panning cameras, so I did a straight-forward pan-speed increase the further the camera is zoomed out.