My Disappointing Automation Experience

This entry is adapted from a post I wrote on the Unity forums expressing disappointment.  I wanted to write a high-level overview of something I did that worked really well, but it was not meant to be.  Continuous integration isn't all that exciting for someone who isn't a developer, but it does help me with a lot of housekeeping so I can focus on making a better game.  However, there were hurdles; for now I've shelved this effort and am resuming work on the actual game.  Read on for more.

This past week I've been trying to set up a continuous integration (CI) pipeline for Chromavaders. I'm currently using the Personal edition of Unity 2019.2.2, but will upgrade to Unity Plus if I need to. I want my pipeline to automatically do these three things, so I don't have to do them by hand:

• To ensure that the tests pass when I make changes to the code
• To ensure that my game builds on all the platforms I support
• To deploy to all platforms and storefronts I plan to support

I looked into Unity Cloud Build, but unfortunately that's not an option for me, for these reasons:

• Chromavaders uses Burst, which cannot be cross-compiled, and therefore cannot be built on Unity Cloud Build
• Unity Cloud Build does not support building for consoles, which I intend to do
• I'm using Trimmer to build multiple variations of my game, and Unity Cloud Build doesn't support some of the features that Trimmer uses

Instead, I decided to try out the GitHub Actions beta for these reasons:

• I already have a paid GitHub plan, so I get 3000 minutes included per month
• It supports Windows, macOS, and Linux, so I can build for any supported Unity platform on the same pipeline
• If I want to add some build step external to Unity (like importing files or generating code), I can do that at my leisure

GitHub Actions has been a great experience for me; it's doing everything I want it to and doesn't get in my way with technical limits. Seriously, give it a try...with a project unrelated to Unity. Here's why.

Unity recently published some improvements to their test framework, and has expressed a renewed interest in CI. These new efforts led me to believe that I could set something up that would at least fit my requirements, even if there were some warts along the way. I was mistaken.

Unity's licensing mechanism is 100% getting in the way of me building the CI pipeline that I want. Currently, a Unity license must be obtained manually, either in the editor or through the process described here and here. For manual use (i.e. when actually designing a game), this is fine; I had no issue installing and licensing Unity for my development machine.

The problem is that you still have to do this when installing Unity on a CI runner. Most CI services don't make promises about a runner's identifying details (network address, hostname, serial numbers, other OS identifiers). As a result, a Unity license for a CI runner could become useless at any time without warning. In fact, that's exactly what's been happening with my own pipeline. This undermines every effort the Unity developers make to offer a smooth automation experience. What's the point of automated builds, tests, or deployments if I have to manually license Unity every time I run my CI pipeline?

I don't have a problem with downloading and installing Unity on the CI runners, nor do I with actually invoking it on the command line. It's rougher around the edges than I'd prefer, but at least it fits my needs. All I want is to be able to use Unity for automated builds, test, and deployments in a CI runner without any manual intervention; the best way I can see that is to remove or totally automate the licensing process for CI pipelines.

Or, if Unity's legal team would get nervous, perhaps they could consider special CI-only builds of Unity that don't need to be licensed, but can only be run headless? Such a build couldn't be used for designing content, and thus would not be worth pirating.

I love Unity and it's generally a bliss to work with, but in this case I'm incredibly disappointed. I know the dev team can do better.

No new screenshots this week because I spent my time working on my attempt at a CI pipeline; it's external to the game, but still in its service.  You can always try out the new build I recently released and take screenshots of that.  Or, if you really want to see a picture, here's my cat Pepe.