A way that you could keep the memorisation aspect but remove the "unfair" aspect of having to try the portals up to three times each, would be to have the solution at the start but fixed in place. That way you can beat it first try if you memorize well.