Ever thought of doing it with your OC and uploading that to Steam, and giving access to custom content through workshop, so people can put things like Frieren there?
It's pretty tricky adding mod support, as I'm using paid addons for all the physics simulations. It might need a lot of work. But I am thinking about it!
For physics you can try to look into cocos creator or cocos2d-x. There free and open source game engines. With cocos2d-x you can change the code to your liking. On top of that cocos is royalty free.
Edit: it supports .fbx files natively to the engine.
