It wouldn't be too much of a problem if it were the case. But you can in fact blow the wall up with bombs as someone pointed out! Bombs however only appear near the back-end of the game as loot from chests and rare drops.
To which end I wonder if anyone found or fought the hidden boss yet.