This is a common mistake made This has been made easier for versions 0.3 and up 

v0.3 and higher

• open the game folder 
• Goto Data
• and double click the Music Fix.bat file

• and your done!!

v0.2.6 and lower

if you closed the game from the X on top then you need to open it up again and press 6 (exit) on the main menu

if you did those things and the music is still playing then go into task mangaer and look for an app called "Microsoft ® Windows Based Script Host" and end that task 

Now if that doesn't help then a restart may be required