Thanks for the feedback!
Yeah, the controller doing this was a deliberate decision made by me, which didn't work out as I wanted.
I might fix it in the future, but in the meantime you can check out the next part (https://forbiddensaucery.itch.io/pyramid-2-demo) which has a more standard controller.