Or, building Unity for mobile VR in the cloud.
This may sound ridiculous and/or insane. But quality and repeatability are important parts of a mature software development process. So we set CI/CD up for Unity.
For the unfamiliar, Unity is a GUI-driven 3D game engine and editor package. The good side of it is that it targets pretty much every platform: web, mobile, game consoles, desktop, etc. The bad side is that it’s not designed for headless use, and crashes a lot in GUI mode.
These days, you can get an executable for the three main OS families (Windows, Mac OS, and Linux). All the code is written in C#, but the language version (and subsequent feature set) is bound by Mono. Suffice it to say C# version 2 doesn’t really have the richness of version 6.
This post is about the technical hurdles we encountered trying to set a build pipeline up (in September 2016). There are (or will be) some more blog posts (by different people) about other aspects of this project.
“Crazy, am I? We’ll see whether I’m crazy or not.”
Part 1: make with the cloud, already
Setting up the cloud infrastructure was easy. Just reuse some existing patterns and templates, and you’re done.
We ended up with a VPC, 3 subnets (to match Sydney’s AZs), and security groups to let in only a few known IPs, an ASG+ELB for one EC2 instance (Jenkins), and an IAM role & instance profile.
We want automation to be self-sustaining, so Packer and Ansible come along for the ride. This is where everything gets a bit hairier . . .
Part 2: self-sustaining CI box
So, we started with a Linux AMI (from AWS) . . . say 14.04, because it has good Ansible support. Start adding Ansible roles, finding existing ones on github where possible.
The Packer template has provisioners to install Ruby and Ansible, so we don’t need roles for them. So let’s add a Packer role, so Jenkins can build itself. Easy, just download the distro, unpack it, and clean up. Kicking goals already.
Let’s add Jenkins next, so we have a one-click job to bake new AMIs. Found ansible-role-jenkins, and a matching ansible-role-java (prerequisite). It’s OpenJDK 1.7, and the Jenkins role seems ok with it.
Now we need something to save our Jenkins jobs, so new boxes don’t have to be set up manually. With a bit of crafty searching, people on the internet will tell you what files to save. Then everything won’t work, and you’ll have to keep searching. We got it eventually, though.
Now, add a matching import script, and bake a fresh AMI. Update the AMI ID in the CloudFormation stack for Jenkins, and we’re good to continue.
Part 3: command-line Unity
Regardless of platform, Unity has the same command-line interface. It’s not documented super-well, but it’s a starting point. So, we try to build a script locally, to build our project. Our local dev boxes already have all the right bits and pieces, so this script should be quick and easy.
Weird, when I run Unity (on my PATH) it quits immediately. No detail, and almost no log. As it happens (and unlike every other command-line tool), Unity must be run from its own install directory, not your project dir. Also, you can specify a log file by a flag. If it’s set to a reasonable path, the log will go there, and set it to an empty string to put it on standard out. Talk about sensible defaults.
So, we end up with:
cd $unity_path
./Unity \
-quit -batchmode -nographics \
-projectPath "$project_path" \
-executeMethod BuildTool.${build_target} \
-logFile "" | tee ${project_path}/unity.log
A few esoteric flags, and some extra tee but it seems to work. So, we commit that, and try it out from Jenkins. No dice. It quits immediately, with no build artifacts. More searching.
The error does quickly yield some results, though. Even with “batch mode” and “no graphics” options, the command-line build still pops up system dialogs, which barf catastrophically when there’s no display. So, we add xvfb to the Ansible playbook, and continue. But dead end, again.
This took a lot more to hunt down, and didn’t even look like an actual solution: there are some npm packages bundled with Unity, and they need to be extracted to a surprise location (replete with jutsu hand gestures and Lovecraftian incantation).
Now it builds all the way to native Mono, and falls over when trying to do the Android bits!
Part 4: Unity builds Android
Did I mention that the target platform is Android? So, we’re also going to need some Android SDK love. This installation is actually very quick and easy. It’s a much smoother CLI, and only one gotcha (echo 'y' piped to the installation command).
But when we try to build again, Unity can’t find any Android bits. This is odd, they’re all on the PATH, like any *nix program would expect. Try adding ANDROID_SDK_ROOT, JAVA_HOME, and even JAVA_SDK_ROOT, to no avail. These are all the vars that we have to set up for the GUI Unity Editor, so what gives?
A bit more crafty searching reveals that Unity doesn’t use PATH (always invokes fully qualified executables), or read any environment variables in (like the paths it wants). So, we have to modify the build script to explicitly read in the environment variables, and set some Unity system variables. As I mentioned before, unlike every other command-line tool.
Anyway, now we get a different error. Unity doesn’t like our Java and/or Android version. Butts. Our Ansible role only goes up to Java 7 for this Linux version. Throw it all away . . .
Part 5: better luck next time
So, we started with a Linux AMI (from AWS) . . . say 16.04, because it’s newer. Ansible doesn’t work. Jenkins doesn’t work. Everything is terrible.
Actually, it’s not that bad. Things are easy enough to fix, but OpenJDK 8 isn’t playing nice with Android. I’m dreading writing some hacky script to accept the Oracle terms, to get the URL, to download the thing. Even though it’s “free” and open-ish.
But before I forge ahead, I have a quick look on the internet . . . hey wow, someone’s already made an ansible-java8-oracle role! Saviour.
And what do you know? It works, first try. Suspicious. But put it on a phone, and it runs. Best news all month (yes, it really took a month of wall time, between actual dev work, to nail all the problems). Many textures are busted, but we can change their compression type.
“It’s alive, it’s moving, it’s alive […] IT’S ALIVE!”
So, it ended up all working. Tests run, too. That’s probably a whole other story.
Unity building (through xvfb) for Gear VR, in Linux with C# through Mono, via Jenkins, all in AWS.
But did we learn anything?
Packer and Ansible is still a great combination for CI in AWS.
I will probably recommend against Jenkins in future – it could probably do everything, but it needs a lot of config. A different CI architecture (such as Buildkite‘s) with separation of job management from build agents would help a bunch.
The community around Unity is vibrant, but generally very low on the capability maturity scale. Almost all answers on the forum are “here’s a hacky fix I did” and replies of “this worked” or “this didn’t work for me”. On top of that, the Linux version (especially headless) is relatively flaky and surprising.
Unity has been good for building and prototyping things quickly. It has an extremely low barrier to entry, which is evidently a double-edged sword. For a full-featured long-lived product, I’d definitely try another platform, maybe Unreal engine.
Legacy
Here are some Ansible roles and build script samples to help other people get up and running quicker. Very bare-bones, it does just enough for this job, not really much more.