I will say, I began developing my game with a game idea that I was almost CONFIDENT that not everyone, not even a large majority, might enjoy.
So it depends on what you are making the game for.
- Do you want to make a successful game?
- Do you want to maximize how many people have fun?
- Do you want to draw in the largest possible crowd?
Give more thought to the fun aspect.
- Are you making it to learn?
- Are you making it to explore your idea?
- Are you looking to have +1 thing out there that you made?
- Are YOU HAVING FUN?
I say go ahead.
Good luck!
ps: It is a part and parcel of gamejams to have the product deviate from your original vision and/or the objectives of the game jam. But that will come easily to you as you gain experience. Hopefully in future jams, you will be able to start with a more cohesive and fun idea, as well as keep it in a proper direction!