itch.io

2.0.8

• The web application will switch between playing the eye blinking animation track and setting the track time based on tracking data. This should result in a smoother blinking.
• Added a "blinking speed" property into the model's "Single Value Properties" to control the play back speed of blink. At default it should be "2" which is about half a second of blinking animation ( 1 second / 2 play speed ). Note that the duration will be shorter because the  animation track will stop at where the eye tracking state.  This animation stop is to prevent jumping from full eyes wide open to eye tracking state after the animation track finishes.

2.0.7

• Updated runtime to Spine 4.2.35 webGL with 4.1 binary backward support.
• Moved menu options to the right of the model display at the request of user for easier live stream changes.
• Fixed an attachment variable name bug that gave application errors.
• Fixed physics constraint bug that did not update correctly.

2.0.6

*  Updated to run Spine 4.2 beta physics constraints. Include Spine 4.1 backward compatibility that 4.2 beta broke.
* Stopped drawing landmarks when face and body tracking modes under "AI Settings" menu are unchecked.
* Increased smoothing for body tracking because there is a lot more shaking from partial body than full body tracking.
* Fixed model number input that broke .svp saving process.

2.0.5

* Optimized setting Spine animation for hand gestures only once.  Prevent continuously setting Spine animation for the same hand gesture animation.

* Added background thread support for body tracking AI on supported modern web browsers. The background thread start up the AI immediately without hindering the main thread. Users would experience shorter or no wait after starting the body tracking for the first time.

2.0.4

* Fixed "Showing Landmarks" canvas bug where it was not showing. The canvas was erasing faster than it can draw due to the 2.0.3 lag bug. Synchronized the erasing  only when it is ready to draw the next frame.
* Added hand gesture tracking for closed fist, point up, victory sign, I love you sign, thumbs up. thumbs down. Hands open is assumed to be the default hand position.

2.0.3

* Added a check box for enabling hand tracking. It is located under "AI Settings" menu > "AI Tracking Modes"  section > "Hand Tracking".
* Added "Debug Bounding Boxes" checkbox  under "Canvas Settings" menu.  Axis aligned bounding box and bounding polygon have separate color and opacity settings. Bounding polygons would help visualize collision shapes.
* Fixed a lag bug caused from turning on and off the camera/video.  The software activated body tracking too frequently in a short amount of time.  Mitigated by slowing down the activation from microseconds to milliseconds.

2.0.2

* Add a pointing downward behavior for the Spine model legs when the tracked legs are outside the video frame boundary. The leg positions are more assumptive and less reliable when outside of video frame boundary. The legs should default to pointing downward ( standing position ) to look more natural than flailing legs.
* Start to implement web parallel processing for browsers that support it. Multiple threads might speed up computation tasks.
* Defer 3d graph plotting task to optimize canvas rendering.
* Save 3d plot graph camera location when user rotates the graph. Note that the user can only rotate the graph when the graph is not being updated ("Update 3D Plot" is unchecked).
* Replace the word "body" with "torso"  in the animation names of "body_rotate", "body_scale_y", and "body_scale_x".  Make these animation names less ambiguous.

For desktop use. It requires a web camera. You want something like Open Broadcaster Software to record or live stream.

Uses modified Spine WebGL runtime 4.2.35 

Copyright (c) 2013-2022, Esoteric Software LLC

Integration of the Spine Runtimes into software or otherwise creating derivative works of the Spine Runtimes is permitted under the terms and conditions of Section 2 of the Spine Editor License Agreement: http://esotericsoftware.com/spine-editor-license

Otherwise, it is permitted to integrate the Spine Runtimes into software or otherwise create derivative works of the Spine Runtimes (collectively, "Products"), provided that each user of the Products must obtain their own Spine Editor license and redistribution of the Products in any form must include this license and copyright notice.

THE SPINE RUNTIMES ARE PROVIDED BY ESOTERIC SOFTWARE LLC "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ESOTERIC SOFTWARE LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, BUSINESS INTERRUPTION, OR LOSS OF USE, DATA, OR PROFITS) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THE SPINE RUNTIMES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Spine Vtuber Prototype uses these animations:

Note: Underscore in animation name can be substituted with hyphen ( - ) or space ( ).

Standby Animations

1. idle
2. breathe

Face Animations

1. face_roll_left
2. face_roll_right
3. face_pitch_down
4. face_pitch_up
5. face_yaw_left
6. face_yaw_right
7. left_brow_raise
8. left_eye_open
9. left_pupil_pitch_down
10. left_pupil_pitch_up
11. left_pupil_yaw_left
12. left_pupil_yaw_right
13. mouth_height
14. mouth_width
15. right_brow_raise
16. right_eye_open
17. right_pupil_pitch_down
18. right_pupil_pitch_up
19. right_pupil_yaw_left
20. right_pupil_yaw_right

Body Animations

1. torso_scale_y
2. torso_scale_x
3. torso_rotate
4. left_upper_arm_scale
5. left_upper_arm_rotate
6.  left_lower_arm_scale
7. left_lower_arm_rotate
8. right_upper_arm_scale
9. right_upper_arm_rotate
10. right_lower_arm_scale
11. right_lower_arm_rotate
12. left_upper_leg_scale
13. left_upper_leg_rotate
14. left_lower_leg_scale
15. left_lower_leg_rotate
16. right_upper_leg_scale
17. right_upper_leg_rotate
18. right_lower_leg_scale
19. right_lower_leg_rotate

Hand Gesture Animations

1. left_close_fist
2. right_close_fist
3. left_point_up
4. right_point_up
5. left_victory
6. right_victory
7. left_iloveyou
8. right_iloveyou
9. left_thumb_up
10. right_thumb_up
11. left_thumb_down
12. right_thumb_down

All the standby and face animations need to be created for the model for it to work in Spine Vtuber Prototype. If you are missing any of the required animations, Spine Vtuber Prototype will pop-up alert you about it. The other animations are optional but are required if you use tracking that corresponds to them.

Each animation represents the range of motion for a given parameter except for idle and breathe. For example "mouth height" represents the mouth range from closed to open, where the first frame is the mouth closed, and the last frame you set is the mouth 100% open. You can set any FPS and create multiple keyframes on the timeline to represent the range of movement. You want to to animate within one second. Any animation past one second could be ignored by this application. For example, if you set your Spine file for 60 FPS, you should not try to animate past 60 frames.

The right and left sides are based on the character model's point of view.

The rotation for the body animations start from the your right and rotate counter-clockwise. At 25% animation the body part should be point upward: 50% pointing to your left, 75% pointing downward.

The body parts with rotate and scale are 2 dimensional transformations.  More faux 3 dimensional transform are still being discussed as Spine at the core is still 2 dimensional.

2.0.0

* Initial Release.

2.0.1

* Added a check box for enabling body pose tracking. It is located under "AI Settings" menu > "AI Tracking Modes"  section > "Body Tracking".
* Added a button for 3d plot graph showing body pose. It is named "Update 3D Plot" and is located under "Camera Settings" menu.

1.1.6

• Add a returning animation option for when custom expressions are active and then resetting custom tracks by pressing expression slot 0.
• Add an assignment button to bind custom expression slots to face expression recognition AI  results under 'Experimental' menu.

1.1.5

• Add "Backface culling" checkbox to "Canvas Settings" menu. The backface of attachments are invisible if checked. The back side is kept consistent with Spine editor.
• Add "Flip Horizontal" checkbox to "Canvas Settings" menu .  When checked, the world X-axis is flipped.
• Add empty animation tracks: custom tracks 1 to 4.
• Add customizable expression buttons 0 to 20 underneath the rendering canvas. Button 0 resets all custom tracks thus removing all customizable expressions active. The buttons are usable once the expression slot has been setup.
• Add customizable expression setup interface under "Model Settings" menu.
  • Add customizable expression slot drop-down menu ( 1 to 20 ) in the setup. Each slot allow you to setup multiple custom track index ( 1 to 4 ), transitional animation, and animation loop that follow after the transitional animation.
  • Include a button to add a setup interface row for adding more custom tracks ( 1 to 4 ), transitional animations, and animation loops.
  • Add a button to remove the last setup interface row. You do not want any rows with incomplete information as you will not be able to finish setting up the customizable expression slot.
  • Add a button to assign all the custom tracks ( 1 to 4 ), transitional animations, and animation loops to the customizable expression numbered slot ( 1 to 20 ).
  • Each setup interface row has three parts: custom track index ( 1 to 4 ), transitional animation,  animation loop. Custom track index ( 1 to 4 ) and transition animation are required for setup while animation loop is optional. The transition animation and animation loop input fields are drop-down menus that lists all the animation found in file ( .json | .skel ). Transition animation does not loop and the animation loop plays after the transitional animation. There is not any mix duration between transition animation and animation loop.
  • The customizable expression slots are saved into and can be loaded from SVP file.
• Remove animation selection from Single Value Properties drop-down list with "Model Settings" menu.
  
  

1.1.4

• Update to Spine WebGL 4.1.24 .
• Add skeleton debug mode into scene render. The checkbox is located under "Canvas Settings" menu.
• Add debug bones checkbox under "Canvas Settings" menu. This include options for bone center color and bone line color, and bone line width ( minimum of 0 and max of 10 ).
• Add debug region attachments checkbox under "Canvas Settings" menu. This include option for region attachment line color.
• Add debug mesh triangle checkbox under "Canvas Settings" menu. This include the option for mesh triangle line color and mesh line opacity ( minimum of 0 and max 100 ).
• Add debug clipping checkbox under "Canvas Settings" menu. This include the option for clipping line color.
• Add left/right pupil pitch/yaw threshold settings to "Single Value Properties" drop-down list under "Model Settings" menu.
• Fix bug that did not load pre-multiplied alpha when it was assigned false.

1.1.3

• Separate error catching on Spine animation tracks.
• Allow "left eye open" Spine animation track to keyframe both eyes while detecting eye wink.
• Add wink threshold to determine the ease of detecting eye winks. Range from 0 ( easiest ) to 1 ( hardest ). Setting located in Model Settings > Single Value Properties > wink threshold. This property can be saved into and loaded from svp file.
• Fixed "brow raise" incorrect landmark tracking.

1.1.2

• Allow model to be loaded into default pose even though the model lacks the required animations.
• Add face expression AI to test into Experimental menu. "Face Expression AI Active" checkbox starts and stops the facial expression recognition. The camera has to be active for the AI. This experiment outputs the recognized facial expression state and list all the possible  facial expressions and their respective confidence level.
• Add "Change Facial Expression Threshold" button to Experimental menu to adjust the confidence threshold for each possible facial expression. Neutral expression will be ignored in the future releases.
  

1.1.1

• Add "Start Record" button to Experimental menu. This button will start  motion capture the model(s) poses. User will need on enter the amount of delay between each capture (milliseconds).
• Add slowly flashing "Recording" label in the Experimental menu. This will signal when the application is motion capturing.
• Add "Stop Record" button to Experimental menu. This button will stop motion capturing.
• Add "Play Keyframes" button to Experimental menu. This button will play back the motion capture without any in-betweening.
• Add "Save Motion Capture" button to Experimental menu. This button saves the current motion capture. The user will be asked to enter a name for the motion capture animation and the amount of delay between each keyframe (milliseconds). The user will also be asked to save a JSON file that can be re-imported back into the Spine editor. User can save multiple motion capture animations in a JSON file by choosing "Cancel" in the save dialog window after each motion capture session. Then save the JSON file at the end when user is ready to re-import back to Spine editor.

1.1.0

• Override spine.SkeletonData.findAnimation method to take in regular expression string parameter. It will allow regular expression pattern use for finding animation names.
• Add an experimental menu for unfinished features and allow others to give feedback during development
• Add "Motion Capture Snapshot" to experimental menu to test saving motion capture pose as a animation in JSON. The JSON then could be re-imported as data into Spine editor. Only works with Spine JSON file imported into Spine Vtuber Prototype. Spine skel binary will not work because the Spine Vtuber Prototype needs to duplicate the Spine JSON. Bone and draw order keyframes are currently operational. This is a requested unfinished feature.
• Add a slider (range) to control the video player opacity in the video player settings menu. At full transparency, the video player can be controlled by the right-mouse context menu.

1. Should be stable.
2. Free and accepting donations.
3. Donations on itch.io is fine. You can find me on Spine unofficial Discord server run by Erika.

1.0.9

• Add premultiplied alpha setting to Single Value Properties. Takes the value of "false" or "true".  Each model can be rendered either premultiplied alpha or straight alpha.
• Exand *.svp file to save premultiplied alpha setting.
• Add the ability to load a video for face tracking. You can find it in the Video Player Settings menu. Video player accepts *.mp4, *.webm, and *.ogg as long as your web browser support those formats and corresponding video codecs.
• Allow the required animation in Spine to be placed in a folder. All the required animations need to be in the same folder. You cannot put the required animations in multiple different folders.

1.0.8

• Allow spaces in Spine animation names to be substituted with hyphens ( - ) or underscores ( _ ).
• Reduce draw face mesh execution time from 5-7 milliseconds to 0-1 milliseconds. Increase frame rate for Spine model while face mesh is shown.
• Expand .svp JSON file to include exporting/importing viewport position, viewport zoom, Spine model canvas height and width, Spine model canvas background color.
• Add title heading for loading state.
• Fix permanently disappearing input field after switching between certain model properties .
• Add viewport zoom property to be editable under 'Single Value Properties'.

1.0.7

• Removed the low-pass filter for face tracking.
• Added unweighted moving average smoothing for face tracking.

1.0.6

• Added a button, located right next to the property drop down menu, to save the model's settings. The model number will determine which settings the button will save from. The downloadable file is formatted as a JSON with ( .svp ) file extension to avoid confusion with Spine's exported JSON file.
• The drag-and-drop zone and upload file button now reads ( .svp ) file as a JSON text.

1.0.5

• Moved the names of uploaded files from bottom of the page into the Drag and Drop Zone area. This allows easier viewing of loaded files.
• Changed how calculations are applied. This allows animators to use multiple keyframes on the timeline for each animation track. Animating within one second is recommended. Previously, each track only allowed one keframe on the timeline.

1.0.4

• Added circumvention steps to prevent the application from freezing when the AI stops recognizing a face.
• Increased the initial values of vectors that would otherwise cause jarring animations.
• Added circumvention to stop moving the model for when a face is no longer tracked but the AI still produces empty tracking data.
• Decreased face recognition threshold in hopes of increasing facial detection range.

1.0.3

• Rearranged model property drop-down list.
• Added "maximum number faces" option to model property drop-down list. This property will allow more than one face to be tracked using a single web camera.
• Added "Single Value Properties" label to the left of model property drop-down list. It should be clearer about the drop-down list if it was not before.
• Updated the track layer for the model property "animation" option".
• Hide value input box for animation and skin properties.
• Add setting for "face pitch strength", "face yaw strength", "face roll strength", "mouth height strength", "mouth width strength", "left brow strength", "right brow strength", "left eye strength", "right eye strength", "left pupil pitch strength", "left pupil yaw strength", "right pupil pitch strength", "right pupil yaw strength" next to "Single Value Properties".

1.0.2

• Changed canvas resize functionality.
• Decreased the strength of jitter filter for the eye pupils.

1.0.1

• Initial release.

Videos

Animation Smoothing Function Test

https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/1e351697-997f-420a-b339-5...

Recorded Video into Camera Feed

https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/1e351697-997f-420a-b339-55dc604511c0/dfayw2p-8bac9980-e12a-4d89-b384-0978995dbef1.1080p.0ddcac221fef40d48b9618a24a4f0d73.mp4

Misaki ( Spine Team )

Cydoni with an O ( https://www.youtube.com/channel/UCFSKg1v6WVHHOLyz7bl3wqg )

Erika ( Spine Team )

Spine Vtuber Prototype uses these animations:

Note: Spaces in animation name can be substituted with hyphens ( - ) or underscores ( _ ).

1. breathe
2. face pitch down
3. face pitch up
4. face roll left
5. face roll right
6. face yaw left
7. face yaw right
8. idle
9. left brow raise
10. left eye open
11. left pupil pitch down
12. left pupil pitch up
13. left pupil yaw left
14. left pupil yaw right
15. mouth height
16. mouth width
17. right brow raise
18. righteye open
19. right pupil pitch down
20. right pupil pitch up
21. right pupil yaw left
22. right pupil yaw right

All these animations need to be created for the model for it to work in Spine Vtuber Prototype. If you are missing any of the required animations, Spine Vtuber Prototype will pop-up alert you about it.

Each animation represents the range of motion for a given parameter except for idle and breathe. For example "mouth height" represents the mouth range from closed to open, where the first frame is the mouth closed, and the last frame you set is the mouth 100% open. You can set any FPS and create multiple keyframes on the timeline to represent the range of movement. You want to to animate within one second. Any animation past one second could be ignored by this application. For example, if you set your Spine file for 60 FPS, you should not try to animate past 60 frames.

Spine Vtuber Prototype uses these animations:

1. breathe
2. face pitch down
3. face pitch up
4. face roll left
5. face roll right
6. face yaw left
7. face yaw right
8. idle
9. left brow raise
10. left eye open
11. left pupil pitch down
12. left pupil pitch up
13. left pupil yaw left
14. left pupil yaw right
15. mouth height
16. mouth width
17. right brow raise
18. righteye open
19. right pupil pitch down
20. right pupil pitch up
21. right pupil yaw left
22. right pupil yaw right

All these animations need to be created for the model for it to work in Spine Vtuber Prototype. If you are missing any of the required animations, Spine Vtuber Prototype will pop-up alert you about it. You can set any FPS and create multiple keyframes on the timeline to represent the range of movement. You want to to animate within one second. Any animation past one second could be ignored by this application. For example, if you set your Spine file for 60 FPS, you should not try to animate past 60 frames.