K8s.io KinD inside a Sysbox Container

January 10, 2022

Contents

• Intro
• Background on KinD and Motivation
• Running KinD inside an Unprivileged Sysbox Container
• Running KinD inside an Unprivileged Sysbox Pod
• Sysbox Container Image
• Conclusion

Intro

This article will show how to run the K8s.io KinD tool inside an unprivileged container or pod launched with the Sysbox container runtime, as shown in the figure below.

As shown above, the Sysbox container acts as a secure environment inside of which the KinD tool runs and deploys one or more Kubernetes clusters.

The benefits of doing this are the following:

• Containerizing KinD itself (i.e., the KinD tool and all associated config is packaged in a container).

• Improving the isolation between the KinD cluster and the host machine, by surrounding the insecure privileged containers deployed by KinD with a secure (unprivileged) Sysbox container.

• Avoiding the limitations and complexity of KinD + Rootless Docker.

• Running KinD inside Kubernetes pods securely (without privileged pods or heavier VM-based approaches like KubeVirt).

This setup is useful and popular, particularly when using containers as development environments or in CI pipelines, because it allows you to prepackage KinD and run it securely.

Background on KinD

The K8s.io KinD tool allows users to easily deploy Kubernetes clusters using containers, as shown in the figure above.

That is, each container acts as a node in the Kubernetes cluster, and the cluster is made up of several such containers interconnected by a container network.

KinD is a very useful tool for testing on Kubernetes, as it allows you to provision clusters quickly and efficiently in your local machine using light-weight containers instead of heavier VMs.

Motivation for running KinD inside a Sysbox Container

Normally KinD runs on a host (e.g., bare-metal or VM), but in this article we are running KinD inside a Sysbox container.

The reason is that KinD deploys the cluster using privileged Docker containers by default, and these are pretty insecure (they provide very weak isolation between the container and the host).

However, by placing the KinD cluster inside a Sysbox container, you can significantly harden the isolation because the privileged containers are only privileged inside the Sysbox container. The Sysbox container itself is unprivileged (aka rootless) and provides strong isolation.

Though there is an option to use KinD with Rootless Docker (which also hardens isolation), this comes with the complexity and limitations associated with Rootless Docker. Plus it does not provide a solution when you want to run KinD inside Kubernetes pods securely (i.e., you need a privileged pod).

With Sysbox, the approach is simpler, with fewer limitations, and allows you to run KinD securely inside Docker containers or Kubernetes pods.

Running KinD inside an Unprivileged Sysbox Container

To run KinD inside an unprivileged container launched with Docker + Sysbox, follow these steps:

1) Install Sysbox on your host as described in the Sysbox User Guide.

For example, on my Ubuntu 20.04 (aka Focal) host, I installed Sysbox with:

$ wget https://downloads.nestybox.com/sysbox/releases/v0.4.1/sysbox-ce_0.4.1-0.ubuntu-focal_amd64.deb
$ sudo dpkg -i sysbox-ce_0.4.1-0.ubuntu-focal_amd64.deb

2) Deploy a container using Docker and the Sysbox runtime:

This is the container inside of which the KinD tool and associated cluster will live, as shown in the figure at top of this article.

In this example I chose the image nestybox/ubuntu-focal-systemd-docker because it includes systemd and Docker inside the container (Dockerfile), but you can choose another image if you wish. Just make sure the image includes Docker inside the container.

$ docker run --runtime=sysbox-runc -d --rm --name kind_container --hostname kind_container nestybox/ubuntu-focal-systemd-docker

Once the container is running, you can exec into it and should see Docker running inside of it:

$ docker exec -it kind_container /bin/bash

root@kindcontainer:/# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

You can verify this is an unprivileged (rootless) container, where the root user inside the container is mapped to an unprivileged user at host level:

root@kindcontainer:/# cat /proc/self/uid_map
         0     165536      65536

For example, in this container the root user (0) is mapped to unprivileged user 165536 on the host, thereby hardening isolation between the container and the host.

3) Install KinD inside the container, following the instructions in the K8s.io KinD website:

$ docker exec -it kind_container /bin/bash

/# cd /tmp
/# curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-linux-amd64
/# chmod +x ./kind
/# cp ./kind /usr/local/bin/.

If all is good, you should be able to run the kind command inside the container:

/# kind version
kind v0.11.1 go1.16.4 linux/amd64

We are now ready to deploy our first KinD cluster inside the unprivileged Sysbox container.

Unfortunately there is one wrinkle here regarding the Docker image that the KinD tool must use to deploy the Kubernetes cluster (inside the Sysbox container). The table below describes this:

[1] For KinD clusters that deploy Kubernetes < v1.22, the official kindest/node image does not work inside Sysbox containers due to a problem in the image’s entrypoint. The nestybox/kindestnode image corrects this problem (see its Dockerfile).

[2] For KinD clusters that deploy Kubernetes >= v1.22, the official kindest/node image does work. However, you need Sysbox > v0.4.1 (to be released in late Jan ‘22) as there is a bug in Sysbox that prevents KinD from deploying the cluster.

With this in mind, let’s create a KinD cluster. In this example I am using image nestybox/kindestnode:v1.20.7.

Inside the Sysbox container, do the following:

/# kind create cluster --image=nestybox/kindestnode:v1.20.7

Creating cluster "kind" ...
 ✓ Ensuring node image (nestybox/kindestnode:v1.20.7) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

If all goes well, in less than 2 minutes you’ll have a KinD cluster running entirely inside the unprivileged Sysbox container!

You can then install kubectl inside the Sysbox container and start using the cluster as needed. Refer to the k8s.io KinD website for more info on how to configure the cluster (e.g., create multi-node clusters, etc.) Note that if you create a multi-node cluster, then all nodes of the cluster would live inside the Sysbox container.

Running KinD inside an Unprivileged Sysbox Pod

With Sysbox, you can also deploy KinD inside a Kubernetes pod securely, without using privileged pods.

This is very useful when using KinD in Kubernetes-based CI pipelines for example, or when running development environments inside Kubernetes pods.

To deploy KinD inside an unprivileged Kubernetes pod, follow these steps:

1) Install Sysbox on your Kubernetes cluster as described in the Sysbox User Guide.

For example, I created a Google Kubernetes Engine (GKE) cluster with Ubuntu-based nodes, and installed Sysbox with:

$ kubectl label nodes <node-name> sysbox-install=yes
$ kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml

2) Deploy a pod using Docker and the Sysbox runtime:

This is the pod inside of which the KinD tool and associated cluster will live. The pod spec is below:

apiVersion: v1
kind: Pod
metadata:
  name: ubu-focal-systemd-docker
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: ubu-focal-systemd-docker
    image: registry.nestybox.com/nestybox/ubuntu-focal-systemd-docker
    command: ["/sbin/init"]
  restartPolicy: Never

3) Verify it’s an unprivileged (rootless) pod:

$ kubectl exec -it ubu-focal-systemd-docker -- /bin/bash

/# cat /proc/self/uid_map
         0     362144      65536

This confirms this is an unprivileged pod: the root user inside the container (0) is mapped to an unprivileged user (362144) at host level. The mapping extends for a range of 65536 user IDs.

This means you can now deploy KinD inside the pod securely: the privileged containers that KinD will use to deploy the Kubernetes cluster are well isolated from the underlying host by the unprivileged Sysbox container.

4) Install KinD inside the pod:

$ kubectl exec -it ubu-focal-systemd-docker -- /bin/bash

/# cd /tmp
/# curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-linux-amd64
/# chmod +x ./kind
/# cp ./kind /usr/local/bin/.

5) Create the KinD cluster inside the pod:

/# kind create cluster --image=nestybox/kindestnode:v1.20.7

Creating cluster "kind" ...
 ✓ Ensuring node image (nestybox/kindestnode:v1.20.7) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

That’s it! You can then install kubectl inside the Sysbox container and start using the cluster as needed.

Sysbox Container Image

In the examples above, after we deployed the Sysbox container, we had to install KinD inside the container manually.

We did this for illustration purposes but in a real setup, you would create an image for the Sysbox container that comes preconfigured with KinD, Kubectl, and any other tool you need.

As a reference on how to do this, there is a sample Dockerfile here.

Conclusion

This article showed you how to run the K8s.io KinD tool inside a Sysbox container. There are several benefits to doing so, such as containerizing an entire KinD cluster and hardening isolation from the underlying host.

We hope you find this useful! Feel free to add any questions or comments below.

To learn more about Sysbox, refer to the Sysbox repo, or you can join the Sysbox Slack Channel and ask questions in there too.

Resources

• Sysbox GitHub Repo
• K8s.io KinD Website
• nestybox/kindestnode image
• nestybox/kindestnode Dockerfile