Yes, you have to have people selected beforehand. So select some people, then press the destroy button, and then the building. The selected people will then start destroying the building you clicked.
