This is definitely an issue with worker GC not triggering when it should and not a direct memory leak.
I looked around for possible fixes and there is a complicated one that at least one developer has done where they terminate the worker after a certain amount of time and start a new one 'fresh'. I am not sure this is a good way to go yet, since they were only using one worker, not hundreds of workers like in this test case.
As you noticed, when there is less load in terms of the number of workers, the worker GC have time to activate and lower the memory requirements. As you note not a desired solution, but hopefully a work around for now. For example if in the test case, I delete 1/2 of the models and watch memory, I can se GC kick in on the workers as memory pressure rises. I do not see the same case with the screen full of models.
As a longer term project I will experiment with whether the tear down and worker creation idea would work (I need to do more like retain some of the state from the worker, so animations don't get messed up, so a few logistics to work out before I can start testing.)