I just posted a follow-up post that kind of explains what's going on.
https://ohsat-andrej.itch.io/speer/devlog/14890/quantum-collision-issues-followu...
