Docker inside containers
Table of Contents
Some CI pipelines and integration tests need to build or run Docker
images. I run Docker as a sidecar service: the official docker:dind
image runs dockerd in a separate container, and the client container
only needs the Docker CLI with DOCKER_HOST pointing at the sidecar.
When running Docker-in-Docker inside a Dagger container, there is a
DNS subnet collision to handle. The outer Dagger engine places the
container on a 10.87.0.0/16 network with a DNS server at
10.87.0.1. When a nested Dagger engine starts inside the DinD
container, it creates its own dagger0 bridge in the same
10.87.0.0/16 range. That route captures all traffic to
10.87.0.1, making the DNS server unreachable from containers managed
by the inner engine.
The subnet is hard-coded in Dagger's source (DefaultCIDR =
"10.87.0.0/16" in github.com/dagger/dagger/network) with no
configuration option to change it. A session-level config file is
tracked upstream but
not implemented yet. Until then, the workaround below is necessary.
The fix lives in the sidecar wrapper script: before starting
dockerd, save the current DNS server and tell Docker (via
daemon.json) to hand out 172.17.0.1 (the Docker bridge gateway)
as the DNS server for containers. The wrapper then adds iptables DNAT
rules that forward DNS queries arriving at 172.17.0.1:53 to the
real DNS server. Containers managed by the inner Docker — including a
nested Dagger engine — can always reach the bridge gateway, so DNS
works transparently.
1. Docker daemon sidecar
The docker:dind image handles cgroup setup, storage driver
selection, and dockerd lifecycle. The wrapper script adds DNS
forwarding on top.
TLS is explicitly disabled (DOCKER_TLS_CERTDIR set to an empty
string) because the daemon only talks to the client container inside
Dagger's sandbox. Docker warns that "in future versions this will
be a hard failure." This is safe to ignore: Dagger owns this
daemon and will update their images when Docker enforces TLS. Every
Dagger user sees the same warning.
The stock dockerd-entrypoint.sh emits a few warnings on
nf_tables-only kernels (no legacy iptables modules loaded):
cat: can't open '/proc/net/arp_tables_names': No such file or directory iptables v1.8.11 (nf_tables) sed: write error
These come from Docker's own iptables backend detection logic, not
from our wrapper. The entrypoint probes
/proc/net/{ip,ip6,arp}_tables_names to decide whether to use
legacy iptables or nf_tables, then unconditionally prints iptables
--version. On kernels that only have nf_tables loaded (no
ip_tables / arp_tables modules), the probes fail and the
modprobe fallback produces the sed error. Reproducible with
plain Docker — no Dagger involved:
docker run --rm --privileged docker:27-dind sh -c 'dockerd-entrypoint.sh dockerd 2>&1 | head -10'
This is harmless: the nf_tables fallback works correctly. Upgrading
to docker:28-dind does not help — the same entrypoint code is
present.
@function
def dind_service(self) -> dagger.Service:
"""Return a Docker daemon running as a sidecar service.
Wraps the docker:dind entrypoint with DNS forwarding so
nested Dagger engines can resolve names despite the
10.87.0.0/16 subnet collision.
"""
return (
dag.container()
.from_(self.pinned(self._dind_engine_image))
.with_env_variable("DOCKER_TLS_CERTDIR", "")
.with_env_variable("TINI_SUBREAPER", "")
.with_new_file(
"/etc/docker/daemon.json",
'{"dns": ["172.17.0.1"]}',
)
.with_exec([
"mv",
"/usr/local/bin/dockerd-entrypoint.sh",
"/usr/local/bin/dockerd-entrypoint-orig.sh",
])
.with_new_file(
"/usr/local/bin/dockerd-entrypoint.sh",
_DIND_DNS_ENTRYPOINT,
permissions=0o755,
)
.with_mounted_cache("/var/lib/docker", dag.cache_volume("dind-docker"))
.with_exposed_port(2375)
.as_service(
use_entrypoint=True,
insecure_root_capabilities=True,
)
)
2. Preparing a DinD client container
The client container only needs the Docker CLI — the daemon runs in
the sidecar. The CLI binary is copied from the same docker:dind
image (statically linked, works on any Linux).
@function
def dind_container(
self,
base: dagger.Container | None = None,
) -> dagger.Container:
"""Return a container with the Docker CLI installed.
If base is provided, the CLI is added to it.
Otherwise uses Lib.dind_ubuntu_image.
"""
if base is None:
base = dag.container().from_(self.pinned(self._dind_base_image))
docker_cli = (
dag.container().from_(self.pinned(self._dind_engine_image)).file("/usr/local/bin/docker")
)
return base.with_file("/usr/local/bin/docker", docker_cli)
3. Running commands with dockerd
The client container binds the Docker daemon sidecar and sets
DOCKER_HOST so the CLI talks to it over TCP. The daemon runs in
the sidecar with elevated privileges — the client container does not
need insecure_root_capabilities.
@function
def dind_with_docker(
self,
cmd: str,
ctr: dagger.Container | None = None,
) -> dagger.Container:
"""Run a shell command inside a container with Docker available.
Binds a Docker daemon sidecar and sets DOCKER_HOST so the
Docker CLI in the container talks to the sidecar over TCP.
"""
if ctr is None:
ctr = self.dind_container()
return (
ctr
.with_service_binding("docker", self.dind_service())
.with_env_variable("DOCKER_HOST", "tcp://docker:2375")
.with_exec(["bash", "-c", cmd])
)
4. Running the test suite inside DinD
Dogfooding: build a container that has everything needed to run the
project tests (Python, pytest, Dagger CLI) on top of the DinD base,
then execute ./test-host.sh with dockerd available.
@function
def dind_run_tests(
self,
src: Annotated[
dagger.Directory,
DefaultPath("."),
Ignore(["**",
"!src/lib/", "!src/ralph.yml", "!sdk/", "!tests/", "!examples/",
"!dagger.json", "!.daggerignore", "!pyproject.toml",
"!test-host.sh"]),
],
) -> dagger.Container:
"""Run the project test suite inside Docker-in-Docker (dogfooding).
Builds a test-ready container from the DinD base, installs Python,
pytest, and the Dagger CLI, mounts the project source, and runs
./test-host.sh with dockerd available.
Source is mounted last so package installs are cached.
"""
ctr = self.dind_container()
ctr = (
ctr
.with_exec(["apt-get", "update"])
.with_exec([
"apt-get", "install", "--yes",
"curl", "python3", "python3-pip", "python3-venv",
])
.with_exec([
"pip3", "install", "--break-system-packages",
"pytest", "pytest-asyncio",
])
.with_exec([
"bash", "-c",
"curl -fsSL https://dl.dagger.io/dagger/install.sh"
" | DAGGER_VERSION=0.20.3 BIN_DIR=/usr/local/bin sh",
])
.with_directory("/work", src)
.with_workdir("/work")
.with_directory("/work/.git", dag.directory())
)
cmd = (
"echo '=== dockerd ready ===' && "
"docker info --format 'Docker: {{.ServerVersion}}' && "
"echo '=== running test suite ===' && "
"./test-host.sh"
)
return self.dind_with_docker(cmd=cmd, ctr=ctr)
5. Running org-babel in a container
The host emacs version affects org-babel output (whitespace,
formatting). To get reproducible results we run run-host.sh,
tangle-host.sh, and init-examples-host.sh inside containers
with a pinned emacs.
Two container flavors:
emacs_container— lightweight: just emacs, git, python, ruff. Used bydind_tanglewhich doesn't need Docker or Dagger.dind_emacs_container— adds emacs, ruff, the Docker CLI, and the Dagger CLI. Used bydind_run_organddind_init_exampleswhose babel blocks calldagger call.
@function
def emacs_container(
self,
src: dagger.Directory,
) -> dagger.Container:
"""Return a lightweight container with emacs, git, python, and ruff.
~elpa-htmlize~ is installed so ~ox-html~ exports can produce
syntax-highlighted source blocks. ~curl~ is there so
~export-html-host.sh~ can fetch water.css into the shared
=.tangle-deps/= cache.
"""
return (
dag.container()
.from_(self.pinned(self._dind_base_image))
.with_exec(["apt-get", "update"])
.with_exec([
"apt-get", "install", "--yes",
"emacs-nox", "elpa-htmlize", "curl",
"git", "python3", "python3-pip", "python3-venv",
])
.with_exec([
"pip3", "install", "--break-system-packages", "ruff",
])
.with_directory("/work", src)
.with_workdir("/work")
)
@function
def dind_emacs_container(self) -> dagger.Container:
"""Return a DinD container with emacs, git, python, ruff, and dagger CLI.
Does NOT mount source — callers add it last so the entire
tool install chain is cacheable.
"""
ctr = self.dind_container()
return (
ctr
.with_exec(["apt-get", "update"])
.with_exec([
"apt-get", "install", "--yes",
"curl", "emacs-nox", "git", "python3", "python3-pip", "python3-venv",
])
.with_exec([
"pip3", "install", "--break-system-packages", "ruff",
])
.with_exec([
"bash", "-c",
"curl -fsSL https://dl.dagger.io/dagger/install.sh"
" | DAGGER_VERSION=0.20.3 BIN_DIR=/usr/local/bin sh",
])
)
Each function snapshots /work before running the script, then
returns only the diff — files actually modified by the script. This
avoids clobbering files the user may be editing locally.
Each function declares its source files via DefaultPath(".") +
Ignore(["**", "!pattern", ...]) — a per-function allowlist. The
engine filters the context directory before the function runs,
producing a directory whose graph identity depends only on the
non-ignored files. This avoids a known Dagger caching bug where
dag.current_module().source() changes its graph identity when
unrelated files are added or removed, busting the exec cache even
though the filtered content is identical. See
studies/caching/readme.org for the full analysis and reproducer.
5.1. Tangling org files
Tangle only needs emacs — no Docker, no Dagger. Runs
./tangle-host.sh in the lightweight emacs_container.
@function
def dind_tangle(
self,
src: Annotated[
dagger.Directory,
DefaultPath("."),
Ignore(["**",
"!src/", "!tests/", "!examples/", "!studies/", "!.clk/",
"!*.org", "!*.sh", "!*.el",
"!dagger.json"]),
],
) -> dagger.Directory:
"""Tangle org files inside a container and return only the modified files."""
ctr = self.emacs_container(src=src)
ctr = ctr.with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
before = ctr.directory("/work")
after = ctr.with_exec(["./tangle-host.sh"]).directory("/work")
return before.diff(after)
5.2. Running org-babel blocks
Runs ./run-host.sh with dockerd available (babel blocks call
dagger call) and returns only the modified files.
@function
def dind_run_org(
self,
src: Annotated[
dagger.Directory,
DefaultPath("."),
Ignore(["**",
"!src/", "!examples/", "!studies/",
"!tests/dagger", "!tests/ralph-log-filter",
"!tests/ralph_log_sample.txt",
"!*.org", "!*.sh", "!*.el",
"!dagger.json", "!.daggerignore", "!pyproject.toml",
"!sdk/"]),
],
files: list[str] | None = None,
no_cache: bool = False,
) -> dagger.Directory:
"""Run org-babel blocks inside a DinD container and return only the modified files.
If files is given, only those org files are processed.
"""
ctr = self.dind_emacs_container()
ctr = (
ctr
.with_directory("/work", src)
.with_workdir("/work")
.with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
)
before = ctr.directory("/work")
cmd = "./run-host.sh"
if no_cache:
cmd += " --no-cache"
if files:
cmd += " " + " ".join(files)
after = self.dind_with_docker(cmd=cmd, ctr=ctr).directory("/work")
return before.diff(after)
5.3. Initializing examples
Runs ./init-examples-host.sh with dockerd available and returns
only the modified files.
@function
def dind_init_examples(
self,
src: Annotated[
dagger.Directory,
DefaultPath("."),
Ignore(["**",
"!src/lib/", "!examples/",
"!*.sh", "!*.el",
"!dagger.json", "!.daggerignore", "!pyproject.toml",
"!sdk/"]),
],
from_scratch: bool = False,
no_cache: bool = False,
) -> dagger.Directory:
"""Init example modules inside a DinD container and return only the modified files."""
ctr = self.dind_emacs_container()
ctr = (
ctr
.with_directory("/work", src)
.with_workdir("/work")
.with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
)
before = ctr.directory("/work")
cmd = "./init-examples-host.sh"
if from_scratch:
cmd += " --from-scratch"
elif no_cache:
cmd += " --no-cache"
after = self.dind_with_docker(cmd=cmd, ctr=ctr).directory("/work")
return before.diff(after)
5.4. Exporting org files to HTML
Exports org files to HTML for GitHub Pages. Uses the lightweight
emacs_container (no Docker needed) and returns the _site/
directory containing the HTML output.
@function
def export_html(
self,
src: Annotated[
dagger.Directory,
DefaultPath("."),
Ignore(["**",
"!src/*.org", "!tests/testing.org",
"!examples/*/readme.org",
"!*.org", "!*.sh", "!*.el",
"!dagger.json"]),
],
) -> dagger.Directory:
"""Export org files to HTML with noweb expansion for GitHub Pages."""
ctr = self.emacs_container(src=src)
ctr = ctr.with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
return ctr.with_exec(["./export-html-host.sh"]).directory("/work/_site")
dagger call dind-with-docker --cmd="docker info --format '{{.OSType}}'" stdout
6. Image references for pin management
See alpine for the convention. These properties alias the
user-facing dind_image and dind_ubuntu_image fields so the
_*_image discovery convention picks them up.
@property
def _dind_engine_image(self) -> str:
return self.dind_image
@property
def _dind_base_image(self) -> str:
return self.dind_ubuntu_image