I don’t think anyone can tell you whether or not to quit a job that isn’t working for you. That’s a decision that you have to make for your own reasons.
I can say, from experience and having tried it a few years ago: quitting your job to game dev full time is a big gamble, and you’re extremely likely to lose that bet.
These days, I have a full time job and work on my game every day. I made a commitment to do something on my game every day. Some days it’s a 5 minute bit of work and some days it’s a full 8 hours. This works for me, and I can only recommend that you find a sustainable balance for yourself.