You're learning some hard lessons here, but well worthwhile. Game Design Documents are there to keep you in scope and keep the project going in a direction, which is super useful when you're hip deep in the minutiae of doing things.
With that, you can still use agile processes, but you really have to get the "what am I actually doing" part down first.
Go do the new project. Get some distance, and come back to ComCreatures with fresh eyes, and then look at what you've done, what you need to make it feature complete, and then you can iterate with purpose.
I've bought this game on Desura (RIP), Steam and now have it on itch due to a bundle. It's a simple game with a few variants for replayability, but for the stupid amount of fun it packs in there, it's totally worth the asking price.
