I mean memory isn't supposed to be used to help you win the gain its just for a small boost or faster progression, you can win without spending anything in memories you just have to find the right path, just imagine if you did need memories too win it would increase the amount of grinding by so much