dind.md

Sysbox Quick Start Guide: Docker-in-Docker

This section shows examples for running Docker inside system containers.

The User Guide describes this functionality in deeper detail.

Contents

• Deploy a System Container with Docker inside
• Deploy a System Container with Systemd, sshd, and Docker inside
• Deploy a System Container with Supervisord and Docker inside
• Persistence of Inner Container Images using Docker Volumes
• Persistence of Inner Container Images using Bind Mounts
• Caching Docker Images among multiple Docker-in-Docker Instances

Deploy a System Container with Docker inside

We will use a system container image that has Alpine + Docker inside. It's called nestybox/alpine-docker and it's in the Nestybox DockerHub repo. The Dockerfile is here.

1. Start the system container:

$ docker run --runtime=sysbox-runc -it --hostname=syscont nestybox/alpine-docker:latest

2. Start the inner Docker:

/ # which docker/usr/bin/docker/ # dockerd > /var/log/dockerd.log 2>&1 &

3. Verify Docker started correctly:

/ # tail /var/log/dockerd.logtime="2019-10-23T20:48:51.960846074Z" level=warning msg="Your kernel does not support cgroup rt runtime"time="2019-10-23T20:48:51.960860148Z" level=warning msg="Your kernel does not support cgroup blkio weight"time="2019-10-23T20:48:51.960872060Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"time="2019-10-23T20:48:52.146157113Z" level=info msg="Loading containers: start."time="2019-10-23T20:48:52.235036055Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"time="2019-10-23T20:48:52.324207525Z" level=info msg="Loading containers: done."time="2019-10-23T20:48:52.476235437Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: failed to set opaque flag on middle layer: operation not permitted" storage-driver=overlay2time="2019-10-23T20:48:52.476418516Z" level=info msg="Docker daemon" commit=0dd43dd87fd530113bf44c9bba9ad8b20ce4637f graphdriver(s)=overlay2 version=18.09.8-cetime="2019-10-23T20:48:52.476533826Z" level=info msg="Daemon has completed initialization"time="2019-10-23T20:48:52.489489309Z" level=info msg="API listen on /var/run/docker.sock"/ # docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

4. Start an inner container:

/ # docker run -it busyboxUnable to find image 'busybox:latest' locallylatest: Pulling from library/busybox7c9d20b9b6cd: Pull completeDigest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51eStatus: Downloaded newer image for busybox:latest/ #

As shown, Docker runs normally inside the secure system container and we can deploy an inner container (busybox) without problem.

The Sysbox runtime allows you to do this easily and securely (no complex Docker run commands, no insecure Docker privileged containers!).

Deploy a System Container with Systemd, sshd, and Docker inside

In the prior example we did not have Systemd (or any other process manager) in the container, so we had to manually start Docker inside the container.

This example improves on this by deploying a system container that has both Systemd and Docker inside.

We've also added an SSH daemon in into the system container image, so that you can login remotely into it, just as you would on a physical host or VM.

We will use a system container image called nestybox/ubuntu-bionic-systemd-docker:latest which is in Nestybox DockerHub repo. The Dockerfile is here.

1. Start the system container:

$ docker run --runtime=sysbox-runc -it --rm -P --hostname=syscont nestybox/ubuntu-bionic-systemd-docker:latestsystemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)Detected virtualization container-other.Detected architecture x86-64.Welcome to Ubuntu 18.04.3 LTS!Set hostname to <syscont>....[ OK ] Started Docker Application Container Engine.[ OK ] Reached target Multi-User System.[ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes...[ OK ] Started Update UTMP about System Runlevel Changes.Ubuntu 18.04.3 LTS syscont consolesyscont login:

2. Login to the container:

In the system container image we are using, we've configured the default console login and password to be admin/admin. You can always change this in the image's Dockerfile.

syscont login: adminPassword:Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 5.0.0-31-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantageThis system has been minimized by removing packages and content that arenot required on a system that users do not log into.To restore this content, you can run the 'unminimize' command.The programs included with the Ubuntu system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted byapplicable law.To run a command as administrator (user "root"), use "sudo <command>".See "man sudo_root" for details.admin@syscont:~$

3. Verify that Systemd has started Docker:

admin@syscont:~$ systemctl status docker.service● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-10-24 00:33:09 UTC; 8s ago Docs: https://docs.docker.com Main PID: 715 (dockerd) Tasks: 12 CGroup: /system.slice/docker.service └─715 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sockadmin@syscont:~$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

4. Start an inner container:

admin@syscont:~$ docker run -it busyboxUnable to find image 'busybox:latest' locallylatest: Pulling from library/busybox7c9d20b9b6cd: Pull completeDigest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51eStatus: Downloaded newer image for busybox:latest/ #

Good, it works!

5. Now let's ssh into the system container.

In order to do this, we need the host's IP address as well as the host port that is mapped to the system container's sshd port.

In my case, the host's IP address is 10.0.0.230. The ssh daemon is listening on port 22 in the system container, which is mapped to some arbitrary port on the host machine.

Let's find out what that arbitrary port is. From the host, type:

$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESe22773df703e nestybox/ubuntu-bionic-systemd-docker:latest "/sbin/init" 16 seconds ago Up 15 seconds 0.0.0.0:32770->22/tcp sad_kepler

6. From a different machine, ssh into the system container:

$ ssh admin@10.0.0.230 -p 32770The authenticity of host '[10.0.0.230]:32770 ([10.0.0.230]:32770)' can't be established.ECDSA key fingerprint is SHA256:VNHrxvsHp4aJYH/DQjvBMdeoF0HBP2yKtWc815WtnnI.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '[10.0.0.230]:32770' (ECDSA) to the list of known hosts.admin@10.0.0.230's password:Last login: Thu Oct 24 03:47:39 2019To run a command as administrator (user "root"), use "sudo <command>".See "man sudo_root" for details.admin@syscont:~$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Great, the ssh worked without problem.

This is cool because you now have a system container that is acting like a virtual host with Systemd and sshd. Plus it has Docker inside so you can deploy application containers in complete isolation from the underlying host.

Deploy a System Container with Supervisord and Docker inside

Systemd is great but may be too heavy-weight for some use cases.

A good alternative is to use supervisord as a light weight process manager inside a system container.

We will use a system container image called nestybox/alpine-supervisord-docker:latest. Nestybox DockerHub public repo. The Dockerfile, supervisord.conf, and docker-entrypoint.sh files can be found here.

1. Start the system container:

$ docker run --runtime=sysbox-runc -d --rm -P --hostname=syscont nestybox/alpine-supervisord-docker:latestf3b90976ad0550fc8142568d988c8fa65c54864d04c1637e88323a32f87cf3af

2. Verify that supervisord started all services inside the system container.

From the host, type:

$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESf3b90976ad05 nestybox/alpine-supervisord-docker:latest "/usr/bin/docker-ent…" 2 seconds ago Up 1 second 0.0.0.0:32776->22/tcp sleepy_shamir $ docker exec -it sleepy_shamir psPID USER TIME COMMAND 1 root 0:00 {supervisord} /usr/bin/python2 /usr/bin/supervisord -n 7 root 0:00 /usr/sbin/sshd -D 8 root 0:02 /usr/bin/dockerd 36 root 0:03 containerd --config /var/run/docker/containerd/containerd. 980 root 0:00 ps

As shown, supervisord is running as the init process and has spawned sshd and dockerd. Cool.

3. Now let's ssh into the system container.

In this example the host machine is at IP 10.0.0.230, and the system container's ssh port is mapped to host port 32776 as indicated by the docker ps output above. The login is root:root as configured in the image's Dockerfile.

$ ssh root@10.0.0.230 -p 32776The authenticity of host '[10.0.0.230]:32776 ([10.0.0.230]:32776)' can't be established.RSA key fingerprint is SHA256:/p++Ju2yo5SF1obEV4TeI+Fq6Q2DBErdboO287aSNp0.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '[10.0.0.230]:32776' (RSA) to the list of known hosts.root@10.0.0.230's password:Welcome to Alpine!The Alpine Wiki contains a large amount of how-to guides and generalinformation about administrating Alpine systems.See <http://wiki.alpinelinux.org/>.You can setup the system with the command: setup-alpineYou may change this message by editing /etc/motd.syscont:~#

4. Run a Docker container inside the system container:

syscont:~# docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESsyscont:~# docker run -it busyboxUnable to find image 'busybox:latest' locallylatest: Pulling from library/busybox7c9d20b9b6cd: Pull completeDigest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51eStatus: Downloaded newer image for busybox:latest/ # syscont:~#

Persistence of Inner Container Images using Docker Volumes

The Docker instance running inside a system container stores its images in a cache located in the /var/lib/docker directory inside the container.

When the system container is removed (i.e., via docker rm), the contents of that directory will also be removed. In other words, inner Docker's image cache is destroyed when the associated system container is removed.

It's possible to override this behavior by mounting host storage into the system container's /var/lib/docker in order to persist the inner Docker's image cache across system container life-cycles.

In fact, not only do inner Docker images persist; inner containers will also persist (thought they will need to be restarted).

Here is an example:

1. Create a Docker volume on the host to serve as the persistent image cache for the Docker daemon inside the system container.

$ docker volume create myvolmyvol $ docker volume listDRIVER VOLUME NAMElocal myvol

2. Launch the system container and mount the volume into the system container's /var/lib/docker directory.

$ docker run --runtime=sysbox-runc -it --rm --hostname syscont --mount source=myvol,target=/var/lib/docker nestybox/alpine-docker/ #

3. Start Docker inside the system container:

/ # dockerd > /var/log/dockerd.log 2>&1 &

4. Pull an inner container image (e.g. busybox):

/ # docker pull busyboxUsing default tag: latestlatest: Pulling from library/busybox7c9d20b9b6cd: Pull completeDigest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51eStatus: Downloaded newer image for busybox:latest/ # docker image lsREPOSITORY TAG IMAGE ID CREATED SIZEbusybox latest 19485c79a9bb 7 weeks ago 1.22MB

5. Create an inner container:

/ # docker run -d --name inner-container busybox tail -f /dev/null56ccb4bb33280f3f670956f6f06afde08e8219eb56c9d79a8b0e5d925ecee96d/ # docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES56ccb4bb3328 busybox "tail -f /dev/null" 4 seconds ago Up 2 seconds inner-container

6. Exit the system container.

This causes the inner container to be automatically stopped.

The contents of the system container's /var/lib/docker will persist since they are stored in volume myvol.

7. Start a new system container and mount myvol into it:

$ docker run --runtime=sysbox-runc -it --rm --hostname syscont --mount source=myvol,target=/var/lib/docker nestybox/alpine-docker

8. Start Docker inside:

/ # dockerd > /var/log/dockerd.log 2>&1 &

9. Verify that the inner Docker images persisted:

/ # docker image lsREPOSITORY TAG IMAGE ID CREATED SIZEbusybox latest 19485c79a9bb 7 weeks ago 1.22MB

There they are!

10. Verify that the inner Docker container persisted:

/ # docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES56ccb4bb3328 busybox "tail -f /dev/null" 32 seconds ago Exited (255) 3 seconds ago inner-container/ # docker start inner-containerinner-container/ # docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES56ccb4bb3328 busybox "tail -f /dev/null" 42 seconds ago Up 1 second inner-container

There is it is!

As shown, the inner container images and even inner containers persisted across the life-cycle of the system container.

This is cool because it means that a system container can leverage an existing Docker image cache stored somewhere on the host, thus avoiding having to pull those inner Docker images from the network each time a new system container is started.

There are a couple of important caveats to keep in mind:

• A Docker volume mounted into the system container's /var/lib/docker must only be mounted on a single system container at any given time.

  • This is a restriction imposed by the Docker daemon, which does not allow its image cache to be shared concurrently among multiple daemon instances.

  • Sysbox will check for violations of this rule and report an appropriate error during system container creation.

• A Docker volume mounted into the system container's /var/lib/docker will inherit any files present in that same directory as part of the system container's image. Such files would be present when using system containers that have preloaded inner container images.

  • In other words, if the system container comes preloaded with inner images, those will be automatically transferred to the Docker volume when the system container starts, and will persist across the system container life-cycle.

  • Note that this behavior is different than when bind-mounting host directories into the system container /var/lib/docker (see next section).

Persistence of Inner Container Images using Bind Mounts

This section is similar to the prior one, but uses bind mounts instead of Docker volumes when launching the system container.

The steps to do this are the following:

1. Create a directory on the host to serve as the persistent image cache for the Docker daemon inside the system container.

$ sudo mkdir /home/someuser/image-cache

2. Launch the system container and bind-mount the newly created directory into the system container's /var/lib/docker directory.

$ docker run --runtime=sysbox-runc -it --rm --hostname syscont --mount type=bind,source=/home/someuser/image-cache,target=/var/lib/docker nestybox/alpine-docker/ #

3. Start Docker inside the system container and pull an image (e.g., busybox):

/ # dockerd > /var/log/dockerd.log 2>&1 &/ # docker pull busyboxUsing default tag: latestlatest: Pulling from library/busybox7c9d20b9b6cd: Pull completeDigest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51eStatus: Downloaded newer image for busybox:latest/ # docker image lsREPOSITORY TAG IMAGE ID CREATED SIZEbusybox latest 19485c79a9bb 7 weeks ago 1.22MB

4. Exit the system container.

5. Start a new system container and bind-mount the my-image-cache directory as before:

$ docker run --runtime=sysbox-runc -it --rm --hostname syscont --mount type=bind,source=/home/someuser/image-cache,target=/var/lib/docker nestybox/alpine-docker

6. Start Docker inside the system container and verify that it sees the images from the bind-mounted cache:

/ # dockerd > /var/log/dockerd.log 2>&1 &/ # docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES/ # docker image lsREPOSITORY TAG IMAGE ID CREATED SIZEbusybox latest 19485c79a9bb 7 weeks ago 1.22MB

There are a couple of caveats to keep in mind here:

• A host directory bind-mounted into the system container's /var/lib/docker must only be mounted on a single system container at any given time. This is a restriction imposed by the inner Docker daemon, which does not allow its image cache to be shared concurrently among multiple daemon instances. Sysbox will check for violations of this rule and report an appropriate error during system container creation.

• A host directory bind-mounted into the system container's /var/lib/docker will "mask" any files present in that same directory as part of the system container's image. Such files would be present when using system containers that have preloaded inner container images.

  • This behavior differs from when Docker volume mounts are mounted into the system container's /var/lib/docker (see prior section).

Caching Docker Images among multiple Docker-in-Docker Instances

Running Docker inside system containers leads to the situation where multiple Docker instances are running on the same host machine (i.e., each instance running inside a system container).

A common need in this scenario is to have a local Docker image cache that can be shared among those Docker instances, to reduce network traffic generated by the Docker daemons pulling those images from a remote repository (e.g., Dockerhub).

To achieve this, one may be tempted to create a single host directory or volume that is mounted into the /var/lib/docker of all the system containers that are running Docker. But this is not correct because each Docker instance must each have a dedicated /var/lib/docker data-store (i.e., it can't be shared). This is a restriction imposed by the Docker daemon.

A correct way to achieve this is to host a local image registry using the open-source Docker registry, and point the Docker instances running inside the system containers to it. This local registry acts as a "pull through cache", meaning that images that are not present in the local registry are pulled from the network (e.g., Dockerhub) and cached in the local registry for subsequent accesses. More information on how to set this up can be found here.