Thank you for telling us and for trying the game! yes there seems to be a bug when using the controller, I will make a patch (when the game jam ends)
