Comparison: Sysbox and Related Technologies

October 06, 2020

Intro

In the world of cloud native software, things move fast and there are always several alternatives to accomplish a given task, with different trade offs for each.

It’s no surprise then that software engineers often ask us about the differences between Sysbox and related technologies.

This article sheds a bit of light on this. It gives a quick comparison between Sysbox and alternative technologies with the goal of helping you choose the best option based on your particular use case and requirements.

Recall that Sysbox enables Docker and similar tools to deploy “VM-like” containers (aka system containers) capable of running systemd, Docker, K8s, etc., inside the container, with ease and strong isolation from the underlying host (i.e., no privileged containers). This allows you to use containers in new ways, and gives you an easy, fast, and much more efficient alternative to VMs to run the above mentioned software.

As you’ll see, Sysbox is pretty unique: it is (to the best of our knowledge) the only container runtime that enables deployment of “VM-like” containers using Docker (and soon Kubernetes), easily and without resorting to very unsecure privileged containers.

TL;DR

The table below lists the technologies described in this article and how they compare with each other.

The “OCI-based” column indicates if the solution is compatible with the Docker & Kubernetes ecosystem.

The “Container Isolation Strength” column indicates how hard an attacker would need to work to break out of the container. Privileged containers offer weak isolation, containers using the Linux user-namespace offer much stronger isolation, and containers wrapped by VMs offer even stronger isolation.

For a more detailed description of each of these tools, keep reading.

Contents

• LXD
• Footloose
• K8s.io KinD
• Kindbox
• Rootless Docker
• Kata Containers
• Ignite
• gVisor
• Podman
• Kaniko
• Conclusion

LXD

LXD is a container manager that enables deployment of “VM-like” containers using the venerable LXC library.

Similarities to Sysbox

• Both LXD and Sysbox enable the deployment of “VM-like” containers.

• Both support running rootless containers.

• Both support running Docker inside containers.

Differences from Sysbox

• OCI Compatibility: LXD is not OCI compatible, meaning it uses a container image format that is not compatible with Docker or Kubernetes. Sysbox on the other hand is OCI-based, which enables you to deploy “VM-like” containers using OCI tools (e.g., Docker, Kubernetes) and leverage the OCI ecosystem (storage and networking plugins, container monitoring tools, etc.)

• Maturity: LXD has been around for at least 5 years so it’s more mature than Sysbox. Having said this, Sysbox is well designed and tested, and integrates with mature OCI-based container managers such as Docker.

Recommendation

• If you want to deploy “VM-like” containers using a mature solution and don’t mind these being incompatible with Docker or Kubernetes, LXD is appropriate.

• If you want to deploy these same type of containers while leveraging Docker, Kubernetes, and the entire OCI ecosystem, Sysbox is a better option.

Footloose

Weaveworks Footloose is a tool that creates “VM-like” containers capable of running systemd, ssh, etc. Footloose sits on top of Docker.

Similarities to Sysbox

• Both Footloose and Sysbox enable deployment of “VM-like” containers easily.

Differences from Sysbox

• Work at different layers: Footloose is a tool that sits on top of Docker. Sysbox is a container runtime that sits below Docker. This means that with Footloose you must learn its CLI. With Sysbox you continue to use Docker (or any other OCI-based runtime) as usual.

• Container Isolation: Footloose requires the use of unsecure privileged containers if you wish to run Docker in the containers. Sysbox is capable of running Docker, Kubernetes, etc. without privileged containers.

• Functionality: Footloose does not support running K8s inside Docker. Sysbox does.

Recommendation

• In general we recommend using Sysbox to deploy VM-like containers, as the containers will be capable of running systemd, Docker, K8s, etc., will have strong isolation (rootless), and you can deploy them directly with Docker (and soon K8s).

K8s.io KinD

The K8s.io KinD tool makes it easy to deploy Kubernetes inside Docker containers. That is, each container acts as a Kubernetes (K8s) node. This is very useful for local testing, CI/CD, development, and learning.

Similarities to Sysbox

• Both KinD and Sysbox can be used to deploy K8s in Docker.

Differences from Sysbox

• Work at different layers: KinD sits above Docker and works by doing special configurations on containers to enable them to run K8s. Sysbox sits below Docker (it’s a new type of “runc”) and works by creating containers that can run system software (K8s, Docker, systemd, others) seamlessly and securely.

• Container Isolation: KinD requires the use of privileged containers. These offer very weak isolation from the underlying host; they are not recommended for enterprise use. In contrast, Sysbox is capable of running K8s inside rootless containers (isolated via the Linux user-namespace).

• Custom images: KinD requires custom container images that incorporate a specialized entrypoint to prepare the container. With Sysbox, no specialized images or custom entrypoints are required: K8s runs inside the container just as it would on a VM or bare-metal host. This gives you full control and more flexibility.

• Flexibility: KinD is specific to running K8s in containers and you are restricted by the capabilities of the tool. With Sysbox, you can run K8s in containers using Docker itself, giving you more flexibility on how you configure the cluster.

Possible Integration

Given that KinD runs above Docker and Sysbox runs below Docker, it’s possible to integrate KinD with Sysbox. In fact the KinD maintainers have been nice enough to open an issue to investigate a possible integration. This would allow KinD to run without using privileged containers as well as other functional benefits. Refer to the issue for more details.

Recommendation

• If you want to run K8s in Docker and don’t mind using unsecure privileged containers or KinD’s specialized container images, KinD is a good option.

• If you want to run K8s-in-Docker using strongly isolated containers and want to have full control over the container image and cluster configuration, then Docker + Sysbox is a better option.

Kindbox

The Kindbox tool, developed by Nestybox, is at a high level similar to K8s.io KinD: it eases the deployment of K8s inside Docker.

What makes Kindbox different is that it’s designed to leverage the Sysbox runtime, meaning that the K8s cluster will be strongly isolated from the host (no privileged containers) and you can choose any container image for the K8s nodes.

Kindbox is a wrapper around docker run commands to deploy K8s-in-Docker using rootless containers. With Kindbox, you have full control over the container image, you can resize the cluster dynamically, and you can easily modify it to suit your needs.

Here is a video showing how it works.

Similarities to Sysbox

• N/A (they are complementary tools).

Differences from Sysbox

• N/A (they are complementary tools).

Recommendation

• If you want to easily deploy a K8s cluster using secure Docker containers, Kindbox is your best option.

Rootless Docker

Rootless Docker runs Docker on the host without requiring root privileges. It’s meant to protect the host from Docker itself, by confining it into a sandbox via the Linux user-namespace. The containers generated by the rootless Docker are by extension also confined to that sandbox.

Similarities to Sysbox

• Both Sysbox and rootless Docker can generate “rootless” containers (i.e., containers whose root user is mapped to an unprivileged user on the host).

Differences from Sysbox

• Purpose: Rootless Docker is meant as a way to isolate Docker on the host. Sysbox is meant to create “VM-like” containers inside of which you can run software such as systemd, Docker, and Kubernetes (seamlessly and with strong isolation).

• Trust boundary: With rootless Docker, the host Docker as well as the containers it generates are untrusted. With Sysbox, the host Docker and Sysbox are trusted entities, but the containers generated by these can run untrusted software.

• Functional restrictions: With Rootless Docker, the Docker on the host suffers several limitations due to the fact that it’s running without true root privileges. With Sysbox, the Docker on the host is not functionally constrained, so you can use it to its full potential. Furthermore, Docker instances running inside the containers generated by Docker + Sysbox are not constrained due to the way Sysbox creates the container.

Recommendation

• If you want to protect against vulnerabilities in the Docker daemon at host level, Rootless Docker is appropriate, though it will limit what you can do with Docker.

• If you want to launch containers inside of which you can run Docker (or any other software), Sysbox is more appropriate.

Kata Containers

Kata Containers is a container runtime whose purpose is to improve container isolation by deploying containers inside lightweight virtual machines (VMs).

Similarities to Sysbox

• Both Kata and Sysbox are OCI-based container runtimes.

Differences from Sysbox

• Purpose: Kata is meant to increase container isolation. Sysbox is meant to create “VM-like” containers.

• Container Isolation: Kata uses VMs to enhance container isolation (each container gets a dedicated VM + guest OS kernel). In contrast, Sysbox uses the Linux user-namespace to isolate containers (all containers share the host’s Linux kernel). Because of this, it’s fair to say that Kata provides stronger container isolation than Sysbox.

• Efficiency & Performance: Though we’ve not done a performance analysis between Kata and Sysbox, the fact that Kata wraps containers inside VMs necessarily poses the performance and efficiency penalty of the VM (i.e., guest OS + virtualized hardware overhead). This is mitigated by the fact that Kata uses lightweight VMs, but it’s not clear by how much, specially for IO workloads. In contrast, Sysbox is a pure OS-virtualization technology, so there is no guest OS and no virtual hardware overhead.

• Hypervisor requirement: Kata requires a hypervisor. This is fine on bare-metal machines but may not be fine on cloud instances (as it would require nested hardware virtualization which is not offered by all cloud vendors, and even if offered reduces the efficiency of your costly cloud VMs). In contrast, Sysbox does not require a hypervisor so it can run on top of VMs without problem and with little overhead.

Recommendation

• If you are running on bare-metal and wish to have the strongest possible isolation around your containers, Kata is a good choice.

• If you want strong container isolation but without resorting to VMs, Sysbox is the better choice.

Ignite

Weaveworks Ignite is a tool that deploys container images inside micro-VMs, using the Firecracker virtual machine monitor. Its purpose is to enable the deployment of workloads inside light & fast VMs using a container-based frontend.

Similarities to Sysbox

• You can use Ignite to deploy “VM-like” containers, since by definition Ignite runs the container contents as a real VM.

• Both Sysbox and Ignite work with OCI container images (e.g., Docker images).

Differences from Sysbox

• Container Isolation: Ignite uses Firecracker VMs to run the container contents and isolate them from the underlying host. In contrast, Sysbox uses the Linux user-namespace to isolate containers. Because of this, it’s fair to say that Ignite provides stronger container isolation than Sysbox.

• Efficiency & Performance: We’ve not done a performance analysis of Ignite, but the use of micro-VMs with Firecracker should certainly help reduce the overhead of VMs compared to other hypervisors. Having said this, as with all VMs there is a guest OS and emulation of hardware in software, so this should incur an overhead compared to running containers on bare-metal with Sysbox, in particular for IO-bound workloads.

• Hypervisor Requirement: Ignite requires a hypervisor (Firecracker + KVM). This means it works well on bare metal, but may not be a viable option on a cloud instance (since it would require nested hardware virtualization). Sysbox does not require a hypervisor so it works equally well on bare-metal or inside cloud VMs.

Recommendation

• If you are running on bare-metal and wish to deploy VMs using a Docker-like frontend, Ignite is a good choice.

• If you want to deploy VM-like containers (but without actual VMs), using Docker (and soon K8s), Sysbox is the better choice.

gVisor

gVisor is software that is meant to increase container isolation by reducing the attack surface between the container and the host kernel. It does this by inserting a “user-space kernel” inside each container that prevents direct access between the container and the underlying kernel and instead “funnels” the large kernel API into a reduced set of interactions with the host kernel.

Similarities to Sysbox

• Both gVisor and Sysbox are OCI-based container runtimes.

• Both enhance container isolation (but use very different techniques as described below).

Differences from Sysbox

• Purpose: gVisor is meant to increase container isolation. Sysbox is meant to deploy “VM-like” containers.

• Container Isolation: gVisor increases container isolation via a user-space kernel inserted into each container. Sysbox increases container isolation via the Linux user-namespace. Note that these techniques are not mutually exclusive and could in theory be combined.

• Performance: Though we’ve not done a performance analysis of gVisor, given that it inserts a user-space kernel inside the container, it likely reduces the performance of the container, both in the control and data paths. Sysbox on the other hand adds very little overhead compared to regular containers

Recommendation

• If you wish to reduce the attack surface between containerized apps and the Linux kernel, and do this without VMs, gVisor is appropriate. It will likely reduce the performance of your containers though.

• If you wish to run “VM-like” containers with strong isolation via the Linux user namespace (i.e., rootless), Sysbox is more appropriate.

Podman

Podman is a daemonless container engine that provides an alternative to Docker.

Similarities to Sysbox

• Both can be used to deploy rootless containers.

Differences from Sysbox

• Purpose: Podman is meant to provide a daemonless alternative to Docker. Sysbox is meant to enable Docker (and similar) to deploy VM-like containers.

• Work at different layers: Podman is a container manager that sits on top of OCI-based container runtimes. Sysbox is an OCI-based container runtime.

Possible Integration

Since Podman is an OCI-based container manager and Sysbox is an OCI-based container runtime, in theory Podman can integrate with Sysbox. We plan to explore this in the future.

Recommendation

• Podman is powerful and flexible; use Podman as an alternative to Docker based on your needs.

• Podman does not by itself create “VM-like” containers capable of running Docker, K8s, etc. For this, using Docker + Sysbox is a better choice at this time, until Podman + Sysbox integration is supported.

Kaniko

Kaniko is a tool that builds container images using a Dockerfile, but without the need for the Docker daemon. Its purpose is to enable users to build Docker images in environments where the Docker daemon can’t run easily or securely (e.g., inside Kubernetes pods).

Similarities to Sysbox

• Both enable building container images in environments in which this was previously hard or unsecure. But they do this in different ways (see below).

Differences from Sysbox

• Purpose: Kaniko is a tool designed to build containers. Sysbox is a tool designed to run containers (it’s a container runtime).

• Work in different ways: Kaniko builds container images without resorting to the Docker daemon, to get around the fact that up to recently Docker was not able to run inside containers or pods (without resorting to unsecure privileged containers). Sysbox on the other hand creates containers capable of running the Docker daemon (as well as other similar software) easily and securely. With Sysbox, you can run Docker inside a container and use it to build an image; and moreover, you can even deploy the built image inside the container (i.e., Docker-in-Docker).

Recommendation

• If you want to only build container images inside a container or pod, Kaniko is a good option.

• If you want to build and run container images inside a container, Sysbox is the better option. Also, with Sysbox, the entity building the image inside a container is the Docker daemon itself, so you avoid any incompatibilities that may exist between Docker and Kaniko in the way images are built.

Conclusion

We hope this article helps you make a decision on the best tool to use for your specific use case around containers.

As you can see, Sysbox is unique in that it’s the only container runtime that enables deployment of “VM-like” containers using Docker (and soon Kubernetes), easily and without resorting to very unsecure privileged containers.

We’ve striven to be fair in our comparison. If you know of any tool we missed, or feel we’ve not done a fair comparison, please add your comments at the end of this blog and we will address them. Thanks!