The game is rendered at game boy resolution with the use of a render texture, then that texture is drawn scaled to the resolution of the game window, preserving aspect ratio.
I'm not sure i know what you mean by muddy; the render texture is drawn larger but with point filtering. If the filtering mode is somehow defaulting to another filtering mode, that could cause it to look muddy or blurry.
Some of the elements in the game are pixel art, and some are 3d models rendered at low resolution. Rotation and movement of 3d elements are constrained in a couple of ways, but ultimately the rasterization gives the effect of pixels dancing around in a way that's not consistent with sprite animation. If that's what you're seeing, that is consistent with my method and intentions.