I really like your concept! Though, I had a glitch where trying to move quickly made me clip through the wall, and I couldn't get out, though weirdly reading the other comments it seems maybe this is just me
It's rare but happens, sadly I found the solution 10 after the target. Here's why: Unity Rigidbody collisions can be ignored due to speed, the fix is to set the detection from discrete to continous in the Rigidbody options. Just... one... selection in a dropbox... yeah.