That's a really interesting way to make a game. I usually do the opposite (starting small and adding more stuff), but it feels like your idea is better when it comes to having a solid idea of what the game should be by the end.
My game for instance had a completely different concept than what it ended up being, mostly because I made a lot of changes to make it more fun during the development. Honestly I think both ways are good, but your way of making games is definitely more unique, and really interesting.