Passion always take sacrifice. This is something that ive learned over the last couple of years. If you want to pursue the passion project, you just simply need to make time for it. There will never be a 'good' time, as there will always be other things that need doing.
Thats just sadly how it works unless you somehow win the lottery on time management :/