Thanks for playing! I definitely made it too difficult, or at least too swingy, the early-game--until you've got a decent weapon--is far too hard.
I experimented with having the walls collideable but it made the movement much more difficult and made dodging frustrating so I ditched it. Also, getting enemies to collide in an intelligent way with the walls didn't sound like a trivial addition! :)