Yes, press E to open the menu, the press the trash can in the bottom-left corner. Click on buildings to remove them. Careful- you don't get the resources back!
