Got it working. Here's what I did:
1. Plug the game pad into the 3rd USB socket and unplug USB 1 (which had an empty USB hub attached), USB 2 is the memory stick.
2. Set joystick port 1 to Mouse and joystick port 2 to CD32 Pad.
3. Set memory to 2MB chip, 4MB fast (was maxed out) and Blitter to Normal (was on Wait).
Not sure if it was just one of those things. But it's working 👍