Playing 1.0.3 I don't believe custom characters are working in Endless mode, I set it to 'Custom characters only' and proceeded to see Nyx show up a couple times back to back.
(I do have custom characters saved and they were working in 1.0.2 In hindsight maybe Custom Characters Only isn't working, can anyone else test and verify?)