How This All Started
I figured we could grow with an all-in-one box for a long time. But one day, builds started failing, and both the times that builds would last before failing and the steps that would cause failure were never consistent. In this case, a Spark-based project that uses Gradle as its build tool was choking. First things first… let’s check out the build logs:


Time for Migration
So now comes the time to offload the builds to a separate host. Things have changed since we last added resources for Gitlab. Our current host is an Amazon Web Services EC2 instance of m3.xlarge type. To give the runner host a little bit more horsepower, we’re going to utilize a newer m4.* instance type. However, arbitrarily spinning up new instances for builds can get expensive. Expenses matter, but so do working builds, so we need to find a middle ground.
Configuring the Runner
Now that we’ve got our cloud settings tuned to our liking, it’s time to configure gitlab-runner to use the new configuration. Start by installing or updating the runner service. Our environment uses Ubuntu 16.04 instances on EC2-backed hosts, so your mileage may vary. For other platforms, check out the docs. Installation Add the GitLab repo and install the runner: $> curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash $> sudo apt-get install gitlab-ci-multi-runner Minor Opinionated Derail: curling scripts from the internet is always a risk, especially when the instructions have you pipe it directly to sh. Make sure when following instructions like these (which I’ve copied from the official GitLab docs) that you trust the host and you can take that a step further by navigating to the resource manually in the browser and checking out the script before you run it. It is convenient, but I am not a fan. Register the runner. You’ll need a registration token for this step. You will likely want to use a Shared runner so that all of your projects can use this autoscaling setup, not just a Specific runner which is tied to a project’s scope. $> sudo gitlab-runner register Follow the instructions generated by the command, which will prompt you for the token we just mentioned and some other information. I won’t regurgitate all of the steps here, but the steps for installing a runner in your environment are available as part of the official GitLab documentation. When prompted for the executor, we’re looking for docker+machine. Configuration The registration step above should create (or update, if you already had runners) a configuration file at /etc/gitlab-runner/config.toml. We’ll take this in sections to better discuss why each parameter is needed. Let’s start with the top section - concurrency.






- Spot Instancesare temperamental things. If your region runs out of Spot capacity and docker-machine gets rejected by AWS, you’re going to have a lot of stuck builds. Mitigation? Either weather the storm, do some cross-region wizardy, or go back to the more expensive version by removing the amazonec2-request-spot-instance
- Roles: it would be nice to do a lot of the things we’re doing with an IAM user with roles instead.
- Docker on Dockerbuilds are still a requirement for our organization, which allows containers to use the host’s Docker daemon which can be hard to debug sometimes. Mitigation? Endure privileged mode or create a separate shell runner to run Docker builds. One thing to remember if you go with the shell runner is that you’re still running on the GitLab host. You want to separate your build into stages to ensure that all your Dockerfile is doing is using COPY to put resources in the container instead of building them (with webpack or some other build tool).