Am I meant to swap with the boxes I click on? If so, it doesn't seem to work reliably. Interesting idea, so I'd be happy to give it another go if I'm doing something wrong.
When near an item that is smaller or larger than you current size, you can right-click to swap sizes with that object :). Should work pretty good, so you are experiencing a bug I am sorry to hear that.
