So after finding this on the community page due to having the same issue but this comment, most specifically
"or you are running it from a location where it doesn't have read access?"
It got me thinking maybe the whole folder didnt get extracted and it turns out that the "Content" folder got left out.
To address the issue I simply extracted the "Content" folder into the same place that I had the full "Celeste" file and for neatness I had made a folder for the extract.
Hopefully I help other people get access to the game as well, cheers.