Hi.
I'm not sure I fully understand your problem.
In linux as in windows, the association between the extension of a file and the program is made at the registry level in the OS. (On Linux it's a bit more complex, but the concept is similar.)
If your problem comes from the side that instead of distributing the final binary, you are distributing an installer (setup), the problem is that in the absence of a manifest, the itch app understands that the only .exe you have will be the game itself.
If I understood your problem correctly, you can fix it by adding a manifest file that explicitly told the itch app what to do with the files.
I don't know MAC, but I imagine there are programs that allow you to create virtual machines with windows. For example windows 11 tiny inside virtual box