I did some more "extensive" tests today and the culprit seems to be the 6GBs of VRAM my GPU has, which - no matter the settings, be it in game, or in Nvidia Control Panel - always stay at 5.5GB so it makes streaming impossible and GPU often has hiccups even if I am "just" playing. I think this is the bane of UE5 itself and I'm just unlucky. Doubtful this can be fixed, every game on this engine seems to just eat through the VRAM no matter what type of game is it or what settings are used.
From stuff that probably can be fixed, on save reload game seems to be constantly trying to load or unload some assets or post-process effects even if the build shaders option is chosen at the game's start. It causes stutters, drops frames noticeably and slows the game, Control Panel reports 100% 3D usage, can be manually "repaired" only by exiting the game and loading the save back from actual main menu.