Decided to go and take a second look at the algorithm.
Potential Spoilers Ahead, go play the game first, you won't regret it.
Finished a second playthrough to re- experience the game today, all while recording the appearances of the algorithm text. I'm no cryptologist or Linguist, but it does seem like it could be a readable language to me. It's 2:40 AM as im posting this, safe to say I got pretty sucked into this little project, ahaha.
Took a good while, but I think I've managed to grab all the appearances of the red algorithm text. Took a bit of rearranging too, seeing as the text is all wobbly, flipped and mirrored when you encounter it in the game, but my sleep-deprived ass somehow managed to do it.
Here's what I've got so far if anyone else would like to take a crack at actually decoding the thing.