Yeah I experienced the same issue. I guess it's reaching the limits of what the engine can support. I had to settle for only 16 Bytes memory for my project. Would be nice to have a way to "compile" chips so they don't take so much performance.
It would be better to use RGB display. By using two colors, you'll have 256 bytes each RGB display.