Just played the game again, and its a lot better. The slowdown sadly still happens late-game (floors 20+) though, but not to such an extreme degree. But MossShade's suggestion to close the game and reopen it also fixes the issue completely, so that makes things a little better.