Introduction

GitLab CI/CD is a software development tool used for building, testing and deploying your code in a fully automated way. Any GitLab (or GitHub) CI/CD pipeline is composed of jobs, configured by executors and ran by machines created locally or remotely by runners. To make it possible to host a pipeline on Qarnot, we've developed a Docker Machine driver. This allows to use GitLab's docker+machine executor which implements an autoscale algorithm and provides many useful options that we will discuss in this blog post. The runner will create worker VMs (machines) on Qarnot Cloud to run your CI/CD jobs.

Version

Here are the supported versions on Qarnot:

If you are interested in another version, please send us an email qlab@qarnot.com.

Prerequisites

The runner will launch computation tasks, so please make sure to:

• Create a Qarnot account here
• Retrieve your authentication token here

Additionally, you'll need to:

• Install Docker
• Create a GitLab account

Launch a use-case

In this section, you will learn how to launch the runner that will start machines on Qarnot and you will test it out on the default pipeline of a blank GitLab project.

Setup a sample repository

Create a blank GitLab project and go to Settings > CI/CD and expand the Runners section. There, disable the shared runners so that your future pipeline will run on Qarnot. In this section you'll also find the registration token needed to register the runner and the URL of your GitLab instance (defaults to "https://gitlab.com/"). Note: CI/CD might be disabled by default in your project. To enable it, go to Settings > General > Visibility, project features, permissions and be sure to turn on the CI/CD button.

Configure your GitLab Runner

This Docker image contains everything needed on the client side to register and launch a GitLab runner powered by the docker+machine executor. You can tweak the runner's autoscale parameters and pass a custom configuration file to the Docker container to suit your needs. Here is a template file explaining all you need to know about GitLab's autoscale configuration in order to create your own. If it happens not to be enough, you can always refer to GitLab's doc. Keep in mind that it is a template: it is meant to show as much parameters as possible. config.template.toml Here is an example of a configuration file for a runner that runs at most 10 jobs concurrently, has at most 10 machines powered on, does not launch more than 3 machines at a time, leaves idle machines on for 30 seconds, and runs alpine:latest by default if no image is provided for a job: config.default.toml Warning: the attributes MachineName, token, or BucketName should not be changed.

Pull the image

You can pull the docker image directly from the Docker hub: DOCKER_MACHINE_VER=v0.16.2-gitlab.19 && GITLAB_RUNNER_VER=v15.7.2 && docker pull qarnotlab/gitlab-runner-launcher:dmach_$DOCKER_MACHINE_VER-gitrun_$GITLAB_RUNNER_VER 

Launch the runner

First, create a config directory and add your configuration file to it. Then, go back to the parent directory of config and launch the runner using the following template: DOCKER_MACHINE_VER=v0.16.2-gitlab.19 && GITLAB_RUNNER_VER=v15.7.2 && docker run --rm -it -e EMAIL="" -e QARNOT_TOKEN="" -e REGISTRATION_TOKEN="" -e CONFIG="" -e VPN="" -v "$PWD"/config:/config -v "$PWD"/runner-logs:/runner-logs qarnotlab/gitlab-runner-launcher:dmach_$DOCKER_MACHINE_VER-gitrun_$GITLAB_RUNNER_VER Note that you do not have to specify every single environment variable. Here is a complete description, along with the default values:

Please note that the list above only contains variables that are either specific to Qarnot, or required by gitlab-runner. You can provide additional variables to the container, as long as gitlab-runner recognizes them. To get the list of variables that gitlab-runner recognizes, the best option is probably to run gitlab-runner --help and gitlab-runner register --help in this image (specify --entrypoint bash in your docker run command to avoid launching the runner). Here is a list of variables you may need:

Warning: if you interrupt or stop the container, the runner will be unregistered from your GitLab instance, and the caching bucket as well as all Qarnot tasks will be deleted. If the machine powering the runner crashes, you will have to manually delete the tasks. You can do that using Tasq or one of our SDKs (Python, C#, NodeJS, CLI).

Launch a pipeline

On GitLab, go to Build > Pipelines, and select Try test template. This will create for you a very basic .gitlab-ci.yml file (thus defining a pipeline for your project) that you can use to test out the runner. Next, select Commit changes and go back Build > Pipelines. Note: If, for some reason, the Try test template button does not appear, you can always commit this file to the root directory of your project (make sure to name it .gitlab-ci.yml) and select Run pipeline in Build > Pipelines. You should see your pipeline running:

Results

The runner's logs are timestamped and made available in the runner-logs directory. You can also see the VMs being launched and the cache bucket on Tasq:  

After a little while, the pipeline should end successfully:

Advanced use cases

Private GitLab instances

If your GitLab instance is running behind a VPN, you will need Qarnot VMs to be able to reach it. To do so, you can provide an OpenVPN client configuration file in the config directory, and pass its full name to the container using the VPN environment variable. Warnings:

• The runner launcher (aka the container) also needs access to the GitLab instance.
• Depending on your runner configuration file, your VPN config might need to allow more than one connection, otherwise the VMs will constantly steal the connection from each other and will never reach the instance. The number of simultaneous connections your client config allows must be greater than or equal to the number of concurrent machines launched by the runner.

Currently, only credential-less connections using OpenVPN are supported.

Shared & group runners

Everything in this tutorial is also valid for shared and group runners, but you will have to retrieve the registration token correctly: see The scope of runners.

Wrapping up

That’s it! If you have any questions, please contact qlab@qarnot.com and we will help you with pleasure!