It's not enough to have the key to enter the construction site area. You also have to go when no one is around to stop you. Try going to bed and coming back at night.
