That's weird, I never got that issue in testing. Do you have any more information on when this is happening? I tried to reproduce the issue on the itch.io page and wasn't able to get it. Were you playing on Windows?
Thanks for the feedback, and I'm considering possibly developing this idea further!