Thank you for the suggestions, they are really helpful!
I really agree with all of them and will implement them in the next version.
About number 3, I am actually having a hard time making it work well. I am currently defining two planes, one parallel to the Y axis with 0 in the Z coordinate. and one perpendicular to it with 0 in the Y coordinate.
So when someone right click the mouse I raycast to check which plane he hit, and return the world position of that point (if he hit the first plane I will not check the second).
This is the best I could come up with to make the world position of the raycast somewhat make sense with what you are seeing.
I also tried defining the planes at an angle, which helped with keeping all coordinates, but I felt like what was happening was less intuitive.
If you have an idea of how to improve this part I will be really happy to hear!
I actually started this project in Unity for myself. And I was just using DrawGizmos and the inspector.
Than at some point I started to feel like it had a solid foundation and I decided to work on making it run in a build as well.
Again, thanks a lot for the detailed feedback, I appreciate it.
And your project looks really cool as well :D