Thanks br0f,
some things you could try:
You could try removing the Psuedo Saturn Kai and loading the game straight from the official Fenrir interface to see if that makes a difference.
If you're using the Fenrir Loader Kai in the Fenrir, you could try reverting to the official Fenrir interface.
V1.001 and V1.000 had known long load times on Fenrir - this was fixed in V1.002. You may need to wait around 2 minutes for the game to load a level on these older versions. You could try loading up V1.001, giving it that 2 minutes to boot and load a level, and seeing if the experience is any different - this may give me some clues on further things I could look into.