A good idea, I like the concept of swapping between several different versions of the same place. Moving people in groups of 3 added some challenge since it means you have to temporarily bring other people out of the time they belong in if there's 2 or fewer people left in the wrong place.
As others have mentioned you can lock the mouse cursor in Unity to stop in going off-screen with first person controls. Teleporting the player back to the start when they time travel might also be good since you can break out of bounds currently!