Ah! Thanks. I was just thinking last night it was odd there seemed to be no kit to stop this. There is kit to resist or grant immunity to most other things. A dreamcoat or rosewood staff, eh? I shall have to track them down, as they sound the best solution.
Sorry, haven't found it yet - currently at the Isle of White Vertigo and I suspect these items are back over Timmertaut way - in fact I think I may even have got a Rosewood staff there many moons ago when I first did that area, but got rid of it without knowing its functionality. However I have currently paused my game again and it may be a while before I continue. If I find a dreamcoat or rosewood staff and remember, I'll get back to you. If you find it first lemme know, I'd be curious what you find.
In case you ever come back to this, some thoughts:
What shadowdweller said below is most important. The walls can only attack you when you're next to them, so you can take them out with ranged attacks. The problem here is how you get your ammo back without walking next to them... but there's some ways to solve that. One of those involves an item that is often for sale in Divinity at least, another an item that you probably have lying around--if there's anything whose effects you haven't identified, you should! (Speaking from experience--I finished Castle of Mirrors two months ago and only just now identified that item and realized it would have made my life easier.)
For the "stumbling into the wall monster" problem, I find the monster is much more visible in tile rather than text mode, so when I thought some were near I would toggle tile mode on to check.
Anyway, from where it sounds like you are, avoiding hallucination sounds better than trying to tough it out. I haven't found a dreamcoat myself, and the rosewood staff turned out not to be effective; you have to point it, and you can't point a staff while you're hallucinating. And if you can solve the shimmering wall issue, the Castle of Mirrors is probably less deadly than White Vertigo.