Gotta say this is the most innovative concept I've seen, and very well done! Reminiscent of your previous movie game jam game too.
I can see the loading times being a little long as this goes on. Regardless I'll be checking this out in the future just to see how much of a cacophony this descends into!