Overview

Basic Concepts

• Shape: in-game meaningful graphical object - mario, cloud, pipe, bullet, brick etc...
• 2dShape: the 2d representation of Shape, that’s what we see in the original game. It is equivalent to a grid of 8x8px tiles.
• 3dShape: the 3d representation of Shape in real world which will created and rendered in the emulation process. 3dShape is not a unique 3d object, it is a set of 3d object(3dTile), each one correspond to a 8x8px tile (2dTile)
• Pattern: first of all, a Pattern is a Shape without a fixed position.
  • It will be compatible with a class of Shape which contains the same tileset and have the same tile constraint set. Examples of diffrent shapes belong to the same pattern in SMB: pipes with different height, clouds with diffrent sizes. 
  • Tile constraint set is the info of which tile is located on the right/bottom of which tile.
  • A pattern contains a 2dPattern(2dShape) and a set of 3dPattern(3dShape). Normally a 2dShape will be interpreted as a 3dShape but in some cases it will be interpreted as some 3dShapes. 
• A shape will be compatible with a pattern if its tileset is a sub set of 3dPattern tileset and 
• 2dPattern: is a 2dShape without a fixed position.
• 3dPattern: 3dShape without a fix position. 3dPattern is not defined by a set of parameters - with it we can create a set of 3dTile procedurally from a 2dPattern. Given a 3dPattern each tile(2dTile) from the 2dPattern will be linked to a 3dTile. It also contains a list of tags. 
• Script: a script is  set of Lua function with a name and a list of tags. If a script and a 3dShape has a common tag then they are linked together and we can use this script to customize the 3dShape each frame.
• 3d Profile: a list of pattern and script work together to guide the 3dification process of a specific game.

Scripting System

A script is a  set of special Lua function which is executed to customize 3dShape by modifying its parameters. A script is identified by a name and has a list of tags.

To hook itself into the 3dification process, a scripts must implement one or some of following special functions:

• Start(): after loading the rom or the save state, all Start functions will be executed.
• End(): right before finishing the emulation process of a given game, all End function will be executed.

At each frame, following functions will be executed in order:

• Update: all Update functions will be executed first.
• UpdateS: if a script is linked with a  3dShape (share a same tag) then the global variables shape and shape2D will be assigned to this 3dShape and its corresponding 2dShape then the UpdateS function of this sceipt will be executed.
• LateUpdate: all LateUpdate function will be executed  

The special tag "*" means that this UpdateS function will be linked to all 3dShape of the current frame.

Script code can access and modify setting and the current list of 3dShape via various global variables:

• variable shape of type Pattern3D (shape3D and pattern3D have the same type)  and variable shape2D of type Shape. shape and shape2D are only  determinist in UpdateS scope.
• variable Script of type ScriptManager
• variable Frame of type FrameManager (contains all 3dShapes of the current frame)
  • variable frame - the frame counter, shortcut of Frame.frameCounter
  • variable time – current game time in second, shortcut of Frame.time
• variable Setting of type SettingManager
  • variable Light - shortcut of Setting.Light
  • variable Layer - shortcut of Setting.Layer
  • variable Clipping - shortcut of Setting.Clipping
• variable Pattern of type PatternManager
• variable Palette of type PaletteManager
• variable Camera of type CameraControl
• variable gamepad, gamepad1, gamepad2 of type Nespad3D. gamepad1 and gamepad2 contain the state of respective gamepads, gamepad state will return true if either gamepad1’s or gamepad2’s state return true.  

Lua variables are the wrappers of C# objects. To access a field or a property, we use the dot syntax:

 Variable.FieldName

To invoke a method, we use the ':' syntax

Variable:MethodName()

To invoke a static method, we use the normal dot syntax:

Variable.MethodName()

For example:

Setting.BgColor = false;
Setting:Save();
Pattern.GetPatternWithTag("character");

Scripting API

Utility

public struct Vector3 { public float x, y, z;}
public class PVector3 {public int x, y, z;}
public class PVector4 {public float x, y, z, w;}
public class PMatrix4x4 { public float m00, m01,m02, m03, m10, m11,m12, m13, m20, m21,m22, m23, m30, m31,m32, m33;}
public class Color {public float r, g, b, a;}
public enum ZLayer { UNIDENTIFIED, UI, L1, L2, L3, L4, COUNT };
public enum Geo3DType
{
    Default,
    HCYLINDER,
    VCYLINDER,
    CUBE,
    HALFHCYLINDER,
    HALFVCYLINDER,
};
public enum SelectionMode { Shape3D, Tile };
public enum Alignment { FRONT, CENTER, BACK };
public enum RenderAlgo { Greedy, Marching };

ScriptManager

Provide APIs to access to scripts and read, write game memory.

public class ScriptManager
{
    Script Get(string name); // get script with name
    byte ReadMem(ushort address); // read from nes memory
    void WriteMem(ushort address, byte value); // write to nes memory
}
public class Script
{
    string Name { get; } // script name
    bool Enable { get; set; } // if the script is allowed to run
    bool Error { get; } // if there is any error in script code – auto detect by 3dSen
}

Frame Manager

Provide APIs to access to 3dShapes in current frame buffer.

public class FrameManager
{
    List<Pattern3D> GetShapesWithTag(string tag); // return a list of shape having the tag “tag” in the current frame buffer
    Pattern3D GetShapeWithTag(string tag); // return the first shape having the tag “tag” in the current frame buffer
    Pattern3D GetShape(int index); // return the index-th 3DShape in the current frame buffer 
    int shapeCount { get; } // return the 3DShape count in the current frame buffer
    int frameCounter { get; } // return the current frame counter 
    float time { get; } // return time in second of current frame buffer
}

CameraControl

Provide APIs to control the camera.

public class CameraControl 
{
    public float Distance {get; set} // the distance from the camera to the root (0,0,0)
    public PVector3 Rot {get;} //the camera rotation via three axis
}

Nespad3D

Provides APIs to access gamepad state.

public class Nespad3D
{
    bool UpPressed { get; }
    bool DownPressed { get; }
    bool LeftPressed { get; }
    bool RightPressed { get; }
    bool SelectPressed { get; }
    bool StartPressed { get; }
    bool APressed { get; }
    bool BPressed { get; }
    bool RotLeftPressed { get; }
    bool RotRightPressed { get; }
    bool RotUpPressed { get; }
    bool RotDownPressed { get; }
    bool ZoomInPressed { get; }
    bool ZoomOutPressed { get; }
}

PaletteManager

Provides APIs to access color palette info of the current frame.  For each frame there are 4 color palette for sprite and 4 color palette for background tiles. Each color palette has 4 colors so totally there are 32 indexed color. Each indexed color will refer to one of 64 NES colors.

public class Nespad3D
{    
    //return the color of the ith color [0..31]
    public static Color FrameColor(int i); 
    // return the color index in the NES color table [0..63] of the ith color
    public static int FrameColorIndex(int i); 
}

Shape

Provide APIs to access properties of a 2dShape

public class Shape
{
    int Count { get; } // the number of shape3D linked with this 2dshape
    Pattern3D Shape3D(int index = 0);  // get the index-th shape3D linked with this 2DShape
    IntVector2 TStart { get; } // start (top left) position in tile unit (8 pixels)
    IntVector2 TEnd { get; } // end position (bottom right) position in tile unit
    IntVector2 TSize { get; } // shape size in tile
    IntVector2 Start {get;} // start (top left) position in pixel
    IntVector2 End; // end position (bottom right) position in pixel
    IntVector2 Size { get; } // shape size in pixel
    bool Bg { get; } // true if created by background tiles
    bool Hiden { get; } // true if one of one of its sprite tiles is hidden 
    int Palette { get; } // palette index 
    int TileCount { get; } // tile count
    bool Completed { get; } // true if the shape has every tile in the pattern tile set 
    bool Inside { get; } // true if the shape is completely located inside the screen 
    bool AtBorder { get; } // true if shape lies on a screen border
    int Age {get;} // the current amount of time - in frame unit - that the linked pattern has at least one linked shape in output frame
}

Pattern3D

Provide APIs to access properties of a 3dShape/3dPattern

public class Pattern3D
{
    Shape Shape2D { get; } // the corresponding 2dshape
    int Index { get; } // its order in its linked 2dshape's 3dshape list 
    ZLayer Layer { get; set; } // shape layer
    bool UI { get; set; } // return true if Layer is UI, otherwise return false, setting it to true corresponds setting Layer to ZLayer.UI
    PVector3 Scale { get; }
    PVector3 Rot { get; }
    PVector3 Offset { get; }
    PVector3 Pivot { get; } // the pivot of scale and rotation operation float 
    Alpha { get; set; } // shape transparency 
    PVector3 DeformAmp { get; }
    PVector3 DeformPivot { get; }
    PMatrix4x4 DeformSpeed { get; }
    PMatrix4x4 DeformOffset { get; }
    bool Enable { get; set; } // if true, the shape will be rendered 
    bool Sp { get; } // return true if this is a sprite shape
    bool Bg { get; } // return true if this is a background shape 
    bool CastShadow {get; set} // shape will cast shadow or not
    bool ReceiveShadow {get; set} // shape will receive shadow or not
    Vector3 OriPos { get; } // the original shape position, calculated based on BottomLeft, TopRight and Pivot 
    Vector3 BottomLeft { get; } // the original bottom left coordination in pixel
    Vector3 TopRight { get; } // the original top right coordination in pixel
    Vector3 GetDeform(Vector3 pos); // return the deform vector at "pos" position based on Deform parameters and formula
    bool ContainsTag(string tag); // return true if shape contains the tag
    int CData[]; // 8 custom data of each pattern
}

SettingManager

Provide APIs to access setting parameters.

public class SettingManager
{
    public SelectionMode SelectionMode {get; set;}
    public class LightConfig
    {
        public Vector3 Direction {get; set;}
        public Color Color {get; set;}
        public float Intensity {get; set;}
    }
    LightConfig Light { get;}
    bool BgColor {get; set;} //if true, use the game background color else use CustomBgColor 
    Color CustomBgColor {get; set;}
    PVector4 Clipping {get;} //amount of clipped pixel at four borders Left, Bottom, Right, Top 
    bool MapEnable {get; set;} // show or hide the mini map 
    public class LayerP
    {
        public float Offset {get; set}
        public Alignment Aligned {get; set;}
    }
    public class LayerConfig
    {
        public LayerP this[int index] {get;}
        public ZLayer DefLayer {get; set;}
        public bool Auto {get; set;} // if true, 3dSen will auto assign layer for pattern if possible
    }
    LayerConfig Layer {get;}
    int GraphicQuality {get; set;} // valid values: 0, 1, 2
    int VrMode {get; set;} // 0: None, 1: OpenVr, 2: Oculus
    int VSync {get; set;} // enable or disable vsync
    int FrameRate {get; set;} // the frame buffer count per second if VSync is false float 
    GameSpeed {get; set;} // from -4 to 4, 0 is normal speed
    float Volume {get; set;} // from 0 to 1
    RenderAlgo RenderMode {get; set;}
    bool Titling {get; set;} // camera titling effect while pressing up, down, left, right button 
    bool Shadow {get; set;} // enable or disable shadow rendering at global level bool 
    SamePalette {get; set;} // if true, tiles in a shape must have the same palette
    int SpriteTolerance {get; set;} //be default sprite tiles in a shape must be 8px aligned but we could set a tolerance value to this constraint using this property
    bool EmptyTile {get;set;} // EmptyTile = false will filter empty tiles (tiles with all zero indexed color pixels)
    string Save2String(); // save setting to string
    void Load(string s); // load setting from string
    void Save(); // save setting to the permanent storage
    void Load(); // load setting from the permanent storage
}

Examples

Super Mario Bros - UI

In Super Mario Bros we want the status bar at the top of the screen to be always visible, and perpendicular to camera view. That can be done by set every involved pattern to layer UI. The issue is when the same shape is positioned in another area , we don’t want it in UI layer at all. So the static layer assignment won’t work. What we want here is if the shape is high enough (that means it is in the status bar region), its layer should be set to UI. That can be done by a small script tagged “*”:

function UpdateS()
    if shape.BottomLeft.y > 216 then
        shape.UI = true
    end
end

Legend Of Zelda – UI

The UI in Legend of Zelda is a little bit different. Normally the UI part is located at the top but when the start button is pressed, the UI will scroll down and show its full content. That means the UI region position isn’t fixed. To deal with it, we choose a 3dPattern that only appears at the bottom of UI region and doesn’t appear anywhere else, that 3dPattern will be tagged “ui_bottom”. Then we add a script with the tag "*".

function Start()
    saveInt = Light.Intensity
    saveDir = Light.Direction
    Light.Direction = Vector3(0.3, 0.3, 1)
    Light.Intensity = 0.7
end
function Update()
    ui_bottom = Frame:GetShapeWithTag("ui_bottom");
    if ui_bottom then
        bottom = ui_bottom.BottomLeft.y
    else
        bottom = 1000
    end
end
function UpdateS()
    if shape.BottomLeft.y >= bottom then
        shape.UI = true
    end
end
function End()
    Light.Intensity = saveInt
    Light.Direction = saveDir
end

Rotation effect

The code below is a script with a “rot” tag.

function UpdateS()
    shape.Rot.y = frame * 2;
end

If we want any shape to rotate around Y axis continuously, just assign tag “rot” for it.

Day-night effect

function Update()
    f = frame % 600
    if (f < 200) then
        Light.Intensity = 0.5
    else
        Light.Intensity = 1.2
    end
end

This script changes the light intensity periodically to emulate the day/night effect.