You can drop items in your inventory. If you press tab to view what you have and select an item, for most items, a drop button will appear under the description.
Thanks, glad to know I was just derping!