In case you ever come back to this, some thoughts:
What shadowdweller said below is most important. The walls can only attack you when you're next to them, so you can take them out with ranged attacks. The problem here is how you get your ammo back without walking next to them... but there's some ways to solve that. One of those involves an item that is often for sale in Divinity at least, another an item that you probably have lying around--if there's anything whose effects you haven't identified, you should! (Speaking from experience--I finished Castle of Mirrors two months ago and only just now identified that item and realized it would have made my life easier.)
For the "stumbling into the wall monster" problem, I find the monster is much more visible in tile rather than text mode, so when I thought some were near I would toggle tile mode on to check.
Anyway, from where it sounds like you are, avoiding hallucination sounds better than trying to tough it out. I haven't found a dreamcoat myself, and the rosewood staff turned out not to be effective; you have to point it, and you can't point a staff while you're hallucinating. And if you can solve the shimmering wall issue, the Castle of Mirrors is probably less deadly than White Vertigo.