Very interesting. Haven't looked into Stream Diffusion yet. From your blogpost it sounds like the main issue was moving frame data from GPU to CPU and back to GPU. Maybe improving the pipeline to stay GPU only would give a significant performance boost. Have you looked into Unreal RDG API, yet?
Hello! Thanks for reading my post!
The main issue was more between Unreal Process and "python process" (I wrapped StreamDiffusion in a C++ exe).
So I had to find a way to give input frame to this "python process" and retrieve its output in Unreal main process. I used an OS pipe, but it wasn't efficient enough.
This project can be enhanced in many ways, but i moved forward on another project 😅