Ah, there's the issue. For buildings you have to right click on the building itself. I tried making it a key initially but which building got interacted with was unclear sometimes without how wide the sprite gets so I went with right click instead.
ahhhh, okay then.
