Probably related to the cartridge settings in the Ultimate 64 board. I switched the "Bus Operation Mode" from "Quiet" to "Dyn. & Writes" and it works stable now.
That's good to hear.
