Fiddled around some more, tried with an Xbox360 and Xbox one controller and the same problem, so it's not the PS4 controller!
Looked into the default input mappings and they're completely bonkers, had to manually change it all and now it sort of works, however as I've not been able to play the game properly I've no idea what inputs to bind to which buttons so it's taking a lot of trial and error to get this right.