Pre-Installing Docker Images Inside Exported Docker Containers
By Mahyar Mirrashed • 8 minutes read •
Chances are, while working with Docker, you’ve never used the docker container export command. It lets you export a Docker container’s file system as a TAR archive.
This command can be very handy in scenarios where you may want to, for example, create a custom distribution of Ubuntu Linux and want to have some files available/modified on the first boot.
But if you’re building a file system with Docker, is there a way that you can have some Docker images pre-installed in there?
Although the problem itself sounds trivial… and the solution below appears simple… the discovery of this process was far from trivial or simple. 😔
This is a recount of how I discovered a process for pre-installing Docker images in an exported Docker container file system.
Why pre-install in the first place?
At one of my previous companies, we were producing hardware-software solutions at scale. In these scenarios, when you are reprogramming thousands of controllers, it is imperative to have a production process that is as simple as possible.
When steps are added beyond just flashing the OS, the potential for production errors increases dramatically. It is a lot easier to detect if the OS is flashed than checking that each manual installation step is completed. Even the process of adding tests to ensure that the steps were carried out properly has the potential for error, not to mention even more time spent on the assembly line/process.
Thus, the ability to have Docker images pre-installed would save time in multiple areas and ensure overall higher product integrity.
How to run Docker-in-Docker (DinD)?
Now that we’ve justified why we need to have the Docker images pre-installed in our OS, we need to figure out how to actually pre-install them.
Note
Note that, technically speaking, we are not creating an operating system with the
docker container exportcommand. Instead, we are creating a root file system (RFS) which will need a Linux kernel to run it. It’s a difference worth mentioning.
One solution would be to save the image via docker image save and place them into a known location on the file system. Then, an on-boot script could run something like docker image load and “pre-install” the images for us. Normally, this would be an acceptable and straightforward approach. But it doesn’t really pre-install the images, and would offload a “downtime” onto the customer which was long enough to be deemed unacceptable by some standards.
Another solution would be to docker image pull and have the images pre-installed and pre-loaded in the container’s file system. Then, on boot, the images would be “ready-to-go” and the application could start immediately. Since this solution addresses the shortcoming of our previous approach (with docker image save), we will concentrate our efforts here.
Ultimately, though, both solutions require us having the Docker daemon running inside our Docker container: something called Docker-in-Docker (DinD).
To to this, we create a Dockerfile specifying our base image, and installing the docker.io dependency. We’ll also add the ca-certificates package so that Docker can properly authenticate images pulled from the internet.
FROM ubuntu:latest
RUN \
apt-get update && \
apt-get install -y - no-install-recommends \
ca-certificates \
docker.io
Afterward, we build the Docker image and run the container in daemonized (background) and in privileged mode.
Warning
Obviously, running with
--privilegedis dangerous and isn’t needed if you add the correct Docker capabilities, but that would overcomplicate our example.
Once the container is running, we can launch our Docker daemon with the docker exec dind-container dockerd command.
root@e2a29fc382e7:/# dockerd
INFO[2024–07–02T20:03:11.655652796Z] Starting up
INFO[2024–07–02T20:03:11.656633046Z] containerd not running, starting managed containerd
…
INFO[2024–07–02T20:07:38.914296544Z] Loading containers: start.
INFO[2024–07–02T20:07:39.161619295Z] Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option - bip can be used to set a preferred IP address
INFO[2024–07–02T20:07:39.200449711Z] Loading containers: done.
INFO[2024–07–02T20:07:39.205607045Z] Docker daemon commit=24.0.7–0ubuntu4 graphdriver=vfs version=24.0.7
INFO[2024–07–02T20:07:39.205672003Z] Daemon has completed initialization
INFO[2024–07–02T20:07:39.226264836Z] API listen on /var/run/docker.sock
If we ran dockerd in an unprivileged container, we would get this unhelpful error when launching the Docker daemon.
root@e2a29fc382e7:/# dockerd
INFO[2024–07–02T20:03:11.655652796Z] Starting up
INFO[2024–07–02T20:03:11.656633046Z] containerd not running, starting managed containerd
…
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)
(exit status 4)
Why not run Docker directly inside of the Dockerfile?
Importantly, Docker (more accurately, the Docker daemon) cannot be started during an image build. It’s only available in a running container. Therefore, we have to docker run the image and then docker exec to get a running, functional Docker daemon.
Thus, to get a functioning DinD solution, we have to run the following command in our running container:
Once the daemon’s running, we can run or pull whatever images we want. and, when we’re done, we stop/kill the Docker daemon with the pkill dockerd command.
Though, at this point, if we were to export the container’s file system and flash that onto the device, we’d see that no images would be pre-installed… And that has everything to do with Docker storage drivers.
What are Docker storage drivers?
Docker’s storage drivers define how images and containers are stored and managed on your host machine (where the Docker daemon is running). By default, Docker, and most other container runtimes, use the Overlay File System (a.k.a. OverlayFS) for storing image/container contents.
The OverlayFS essentially merges (overlays) multiple folders in specific order and presents them as a single, unified file system for running applications. (You can read more about the details here).
To interact with OverlayFS, Docker has the overlay2 storage driver. On all modern operating systems, it’s the default driver so long as the OverlayFS exists/is accessible.
So, although we had gotten DinD working, pulled/ran our Docker images inside the container, and even observed the size of the container file system export get larger, the Docker images were nowhere to be found once they were flashed on our controllers.
Why?
Because of OverlayFS’s accessibility in a DinD environment…
OverlayFS and DinD are not well-matched…
For reasons that I’ll gloss over, it turns out that the OverlayFS is not accessible in a DinD environment. And while I think it’s technically possible to get DinD to use the operating system’s OverlayFS, it’s not ideal for a host of reasons, such as:
- You would have to volume in your host’s OverlayFS and Docker volumes are not by default included in an exported container;
- Copying only the required layers from your OverlayFS corresponding to the ones for your pulled Docker images is nearly impossible; and
- Exposing your OverlayFS to the privileged Docker container is unwise and could lead to unintended consequences.
So, where did the pulled images go if not on the container file system?
Well, after many hours of combing through the file system contents, and digging around, the clue was in the Docker daemon start-up logs.
Running the Docker daemon in the container via dockerd &, and not suppressing output, we see something very interesting…
INFO[2024–07–02T20:48:29.113351512Z] containerd successfully booted in 0.015496s
ERRO[2024–07–02T20:48:29.136024512Z] failed to mount overlay: invalid argument storage-driver=overlay2
ERRO[2024–07–02T20:48:29.136083887Z] exec: "fuse-overlayfs": executable file not found in $PATH storage-driver=fuse-overlayfs
INFO[2024–07–02T20:48:29.137347887Z] Loading containers: start.
INFO[2024–07–02T20:48:29.199884512Z] Loading containers: done.
INFO[2024–07–02T20:48:29.216828762Z] Docker daemon commit=24.0.7–0ubuntu4 graphdriver=vfs version=24.0.7
INFO[2024–07–02T20:48:29.216949387Z] Daemon has completed initialization
INFO[2024–07–02T20:48:29.237140554Z] API listen on /var/run/docker.sock
That innocuous error message, exec: "fuse-overlayfs": executable file not found in $PATH storage-driver=fuse-overlayfs, was a hint to the underlying storage driver problem when use classic DinD.
Since the OverlayFS was not accessible in the DinD environment, and this other storage driver it attempted, fuse-overlayfs, was unavailable, the Docker daemon reverted to the most primitive image storage format: VFS.
We can confirm this! We flash that exported file system onto our device (e.g. mini PC), set the storage driver to VFS, and then restart the Docker daemon.
And bam! There are our Docker images.
root@controller:/home/halcyon# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest ee301c921b8a 14 months ago 9.14kB
Now the question became: how do we get our downloaded Docker images in the container to be stored in the OverlayFS format, not VFS? While some initial attempts were made to migrate VFS to OverlayFS, that was found to be an unattainable solution; it needed to be simplified.
Reviewing the fuse-overlayfs error
VFS, as you might expect, is quite primitive, and for reasons that it is primitive, it is also quite slow and optimized. You’d have a hard time running Docker on VFS. So let’s see if there’s a way we can fix that.
Returning to the exec: "fuse-overlayfs": executable file not found in $PATH storage-driver=fuse-overlayfs error message, it seems that Docker was attempting to use the fuse-overlayfs driver before reverting to the VFS driver.
FROM ubuntu:latest
RUN \
apt-get update && \
apt-get install -y - no-install-recommends \
ca-certificates \
docker.io \
fuse-overlayfs
Doing the same steps as before (pulling Docker images in the DinD environment), and examining the directory structure that the fuse-overlayfs driver was using, we see that it actually matches the structure of the overlay2 driver that the final, flashed device would be using!
root@8f3d3cdaf284:/# tree -L 2 -d /var/lib/docker
/var/lib/docker
| - buildkit
| | - content
| ` - executor
| - containerd
| ` - daemon
| - containers
| - fuse-overlayfs
| | - 9b52b3eeb61782b2ccdad82b1aad0d7829bc2f6c32b2bd42c803d851c8029e11
| ` - l
| - image
| ` - fuse-overlayfs
| - network
| ` - files
| - plugins
| | - storage
| ` - tmp
| - runtimes
| - swarm
| - tmp
` - volumes
21 directories
Compare it with the directory structure of our normal overlay2 file system.
root@controller:/home/halcyon# tree -L 2 -d /var/lib/docker
/var/lib/docker
├── buildkit
│ ├── content
│ └── executor
├── containers
├── image
│ └── overlay2
├── network
│ └── files
├── overlay2
│ ├── a03d1fb547b69317dde8b53798b0c7917fa51ceac1a48b87c66f5a8ce648ad83
│ └── l
├── plugins
│ ├── storage
│ └── tmp
├── runtimes
├── swarm
├── tmp
└── volumes
21 directories
Given that it’s a similar structure, we can see if renaming the folders and copying them into place would cause them to work on the flashed file system.
# transfer .tar to controller
# extract out /var/lib/docker/fuse-overlayfs and /var/lib/docker/image/fuse-overlayfs onto controller file system
# rename fuse-overlayfs folders to overlay2
On the controller, since there was no storage driver defined in the /etc/docker/daemon.json file, the Docker daemon would default to using the /var/lib/docker/overlay2 folders we had copied over from Fuse. And, lucky for us, that worked!
root@controller:/home/halcyon# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest ee301c921b8a 14 months ago 9.14kB
So, to recap, what we’ve essentially done here, is to stop the Docker daemon in the container process, and rename two folders.
To avoid jinxing it, let’s attempt flashing this updated exported container file system.
root@controller:/home/halcyon# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest ee301c921b8a 14 months ago 9.14kB
Summary
So ultimately, our Dockerfile installs both the docker.io and fuse-overlayfs packages on the base Ubuntu Linux image.
FROM ubuntu:latest
RUN \
apt-get update && \
apt-get install -y - no-install-recommends \
ca-certificates \
docker.io \
fuse-overlayfs
And then, we get the Docker daemon inside the container to download images in the overlay2 Docker driver format.
All that’s left now to do is profit! 🤣