I’ve tested around and I think it’s just freezing over time. I kept the game open and it froze after 10–15 minutes. In another session, I tested if it could be because of tabbing out and in again, but this seems to not be the problem.
There have been some random crashes that very rarely occur, it could have just been that. Out of a lot of playtesting, I haven't had one but it seems like there's some perfect mix that breaks everything.