Thanks for the response, I didn't think to change the settings. Changing the shadow quality to low fixed the FPS issue for me.
Something that may be relevant is when on medium shadow quality a message appears in the top left that says "Virtual Shadow Map Page Pool overflow detected.