Is there a modifier key that I can hit to simply click-and-drag to resize the elements?

Also, is there a way to turn off that annoying element-delete when trying to change the numbers in the scale or position fields? It's so obnoxious to click a field, hit backspace to change a single number, and then your entire element gets deleted. Not to mention there is no undo button.

This is absolutely ridiculous. Please someone tell me there is an easier way to do this.

There's no modifier key to click-and-drag resize, you'll have to change the scale values in the properties pane. Sounds like a bug, shouldn't delete your element when you're focused on a text field, will see that this gets fixed.