Every time the game tries to save. whether it's in gameplay or changing the settings, it completely closes the game and my progress is lost.

I'm on Windows 10 and I did install the program it said was required

I had the same issue, try running the game as administrator or put all the game files in a folder that doesn't require administrator privileges to edit content (e.g. Documents)

On Windows the game tries to save in the directory adjacent to the Celeste.exe. If it doesn't have permission to do so this could cause it to crash.