Hey JiradTheRake, what kind of controller are you using?
I've noticed that other people who have issues with the controls tend to be using third party controllers.
There might be a way to fix it dev side, but I don't own any third party controllers, just the Xbox one that can be used on PC, so I can't exactly test and fix it properly. Unfortunately there's also no guarantee the same issue wouldn't happen for other third party brands.
Another user ran into a similar issue here; https://itch.io/t/2732802/hi-the-game-is-glitching-out-from-the-beginning-for-me
To save you time clicking on the link and reading the entire thread, here's how they fixed it;
well I got it working adding the game to steam and enabling steam input and mapping keyboard controls there to my PS4 controller! Don’t have an xbox one. Must just be something weird with ds4windows
Please let me know if this fixes the issue for you!