Effectively it comes down to how the color palette works. If you store the color data without using UVs, you have to split each polygon based on its color.
Say for example you have a large square that is subdivided into 4 smaller squares in voxels. Using materials, each of those smaller squares have to be independent polygons, leading to 8 polygons. If you use a texture, you can reduce that to 2 polygons.
There are still some reductions that can be made with a palette texture or materials, however, such as batching/instancing within Unity.
I'm currently working on a Unity plugin for the program, which will allow you to directly import your vox models through Unity with all the same properties as the standalone program. Might be of interest to you since you can batch materials using it.