The installer creates a shortcut to provide a quick way to launch Tiled. It should be optional (as should be whether to launch Tiled after installing), but I haven't figured out how to do these things yet. To avoid the installer (and to receive more frequent updates), use the itch.io app and choose the "snapshot" version of Tiled instead of the "release".
I can't tell you why Windows would give you such a confusing error message when trying to remove the shortcut. I have personally never had any problems simply deleting it...
Why ask this question as a reply to an old devlog? :-)