Enhancing Drone.io with System Containers

September 24, 2019

Drone is a modern continuous-integration system built with a containers-first architecture. As such, Drone’s active components run within Docker containers, and the same applies to most of the continuous-integration (CI) pipelines that it executes.

After playing around with various CI/CD solutions, we have learned to appreciate the simplicity of Drone’s ecosystem, and the flexibility that it offers by enabling self-hosted / customized deployments.

Nevertheless, we have found some challenges when trying to expedite our own CI pipeline execution, while keeping our build infrastructure secure. And this seems to be a common pattern across various organizations that we have interacted with.

After doing some research we found that the problem comes down to the approach utilized by Drone to interact with the Docker daemon (a.k.a. dockerd) in charge of executing CI pipeline tasks.

Our objective here is to spell out the pros & cons of the existing approaches, and to propose a more efficient and secure method through the utilization of Nestybox’s system containers.

Contents

Drone & Dockerd interaction

In Drone’s multi-agent setups, the drone-server hands out pipeline tasks to its connected drone-agents. In agentless scenarios, the drone-server itself takes care of executing the required pipeline steps. In either case, there is always some level of interaction between Drone binaries and a Docker daemon to spawn slave containers that can execute CI pipeline tasks.

There are a couple of well-known approaches being utilized for this interaction:

We have fully covered the differences between DinD and DooD solutions in a previous article. Thus, our goal here will be to analyze the pros & cons of these approaches as they are utilized by Drone, and how they can impact Drone’s overall performance.

Drone’s DooD Configuration

In this configuration, all the Drone components that require Docker services refer to the host’s dockerd process. This applies to the docker-server and docker-agent containers, as well as slave containers that require dockerd interaction to complete their tasks (i.e. “docker build”, “docker push”, etc).

For example, in the image below, the drone-server (or drone-agent) in container A interacts with the host Docker daemon to launch slaves containers B, C and D. Container D’s goal is to build a docker image, and for this to happen D makes use of the host Docker daemon to launch container E.

This Drone configuration has one important advantage over the Dood + DinD one described below: image-caching. By making use of a centralized and persistent Docker daemon we can drastically reduce the building time required to execute Drone’s docker-pipelines. Previously imported image layers will be kept in Docker’s caching subsystem, which will dramatically expedite subsequent build tasks.

Unfortunately, this comes at a high cost, as we are exposing the host’s Docker daemon to Drone’s build environment. This is a non-starter for publicly hosted drone instances (e.g. drone.io), but it also poses a problem for private deployments with security requirements.

Drone’s DooD + DinD Configuration

As in DooD configuration, in this configuration Drone interacts with the host’s dockerd to spawn slave containers. However, for tasks that require docker-image handling, Drone relies on popular plugins such as drone-docker, which incorporates a dockerd in its container image.

This approach attempts to address DooD configuration security concerns, but it does so at the expense of the image-caching variable mentioned above. This a consequence of the life-span of the dockerd instance being utilized in the inner container (container D in the image below), matching the one of the task being executed (i.e. creation of container E as part of “docker build” instruction). Thereby, we cannot reuse previously built layers, nor can we push the image being produced into any local / centralized cache, so subsequent builds won’t be ever able to leverage image-caching capabilities.

Unfortunately, without image-caching, many docker-based pipelines are not tenable, as a comparatively large chunk of the image-compilation time is usually spent fetching the required image layer dependencies.

Last but not least, notice that for this approach to work, the DinD container must be initialized with privileged flag, which as described here, poses a serious security risk.

So isn’t there a better solution that can truly conjugate building-efficiency and system-security requirements? Yes, there is …

Proposed Solution

Our proposed solution relies on the use of Nestybox’s system containers and the application of the following basic guidelines:

See below a visual representation of the solution just described.

What do we gain with this?

Proposed Solution Setup

The solution discussed above can be easily tested through the execution of the following docker-compose recipe, and by creating a Drone pipeline such as the one displayed further below.

Notice that there are only three minor differences between the docker-compose recipe below and the one of traditional approaches:

Note that technically speaking, in Drone’s multi-agent setups only the drone-agent container needs the presence of a Docker daemon. Thus, for the drone-server configuration below, we could have just relied on the traditional docker-compose recipe (i.e. no need for a customized image and runtime); we are displaying them here to cover the agentless scenario too.

$ cat docker-compose.yml
version: '2.3'

services:
  drone-server:
    image: nestybox/ubuntu-bionic-drone-server:latest
    runtime: sysbox-runc
    ports:
      - 80:80

    environment:
      - DRONE_OPEN=true
      - DRONE_GITHUB=true
      - DRONE_GITHUB_SERVER=https://github.com
      - DRONE_SERVER_HOST=my-drone.server.com
      - DRONE_SERVER=http://my-drone.server.com
      - DRONE_GITHUB_CLIENT_ID=da96f2060001a68100ed
      - DRONE_GITHUB_CLIENT_SECRET=4601234fe84b5b738a2954b40ecf03ece01231a1
      - DRONE_RPC_SECRET=my-secret
      - DRONE_AGENTS_ENABLED=true
      - DRONE_USER_CREATE=username:nestybox,admin:true,token:55f24eb3d61ef6ac5e83d55017860000

  drone-agent:
    image: nestybox/ubuntu-bionic-drone-agent:latest
    runtime: sysbox-runc
    restart: always
    depends_on:
      - drone-server
    environment:
      - DRONE_SERVER_HOST=my-drone.server.com:80
      - DRONE_RPC_SECRET=my-secret

In terms of pipeline definition we should take the following points into account:

$ cat .drone.yml
...
steps:
- name: build
  image: golang:latest
  commands:
  - go build

- name: docker-build
  image: docker
  volumes:
  - name: cache
    path: /var/run/docker.sock
  commands:
    - docker build -t my-repo/new-image .

volumes:
- name: cache
  host:
    path: /var/run/docker.sock

Refer to this simple GitHub repository (forked off of Drone-demos project) if need more details about the docker-compose recipe and the pipeline configuration being utilized in this example.

Proposed Solution Test

The scenario described above can be easily brought up by following these basic steps:

That’s all, hope it helps!

Check our Free Trial!

If you want to give our suggested proposal a shot, please refer to this link for a free-trial. Your feedback will be much appreciated!