What happens is that the tiles you draw on a tile layer are used to add 5 more squares on top of the one on the tile layer, and that repeated vertically a couple of times, to create a wall with a tiled texture. Only when squares should be visible to the player though.
The 3D camera is just your normal Game-Maker-way 3D camera, with some head movement and extra bells and whistles to have Wolfenstein 3D style shooting and interactions.
As for objects like doors and exits, those triangles are coded within the objects themselves and then given a more unique texture.
So the principle to handle 3D walls with either repeated or unique textures is in here. As for something like Daggerfall (especially the dungeons), I would say this template if studied well can be a good staring point to understand how to do it. Keep in mind Daggerfall is absolutely massive, even without randomised worlds, and this is not such a template. Performance will be the harder problem to solve when you want something on the scale of Daggerfall in Game Maker, but part of me thinks it's not impossible.