Anti-aliasing is possible in DX9 mode through Nvidia Profile Inspector (SGSSAA). Use "0x004000C1" as the compatibility bits. I don't know if there are known compatibility bits for the Unity engine in DX11. But it would be good having some native AA options in 3DNes.
3DNes uses some AA already, but I don't know what type it is. I've tried hunting it in DX9 mode with the Helixmod dll and it seems it isn't a pixel or vertex shader, so maybe it is MSAA.