Yea, I totally agree. I tried to, but there was issues where objects being held would just no-clip through anything while being held. I had to allow for a collision detection to happen, so I just decided to drop the object if you hit it. It's really not what I wanted but it means you can't just skip puzzles with passing objects through walls. Didn't have time, but I will figure it out properly maybe after the jam!
Thanks!