I'm pretty sure the game hangs because dun_room_pos_free only returns true if there's a tile of the wtd_FLOOR type on the position (which also uses the legacy tile function), and there's a number of loops to place enemies/treasure that will loop indefinitely checking random positions until they find a free spot. (Completely blank spots are treated as collisions to stop points outside of the dungeon to count as empty) If no spot is free since the collision-checking function is broken, it will loop forever, and since you now don't spawn the legacy floor tiles anymore, it will never find a free spot.
To fully fix this, you'd need to replace all the legacy tile functions in the engine with something adapted for your multi-layer solution:
The first 3 matches delete wall tiles and overwrites them with floor tiles at positions that should have a door, the next 3 matches are the tile rendering you've already replaced, the four subsequent ones are in the collision-checking function that's currently causing the crash, and the final match is just an effect object that you don't need to worry about at the moment.
My idea for the fix to dun_room_pos_free is to replace the wall/floor checks with checks to see if the tilemap at the wall / floor layers has a nonzero tile, since walls are checked before floors it should work without changing the order/priorities.
