Docker inside containers

Table of Contents

Some CI pipelines and integration tests need to build or run Docker images. I run Docker as a sidecar service: the official docker:dind image runs dockerd in a separate container, and the client container only needs the Docker CLI with DOCKER_HOST pointing at the sidecar.

When running Docker-in-Docker inside a Dagger container, there is a DNS subnet collision to handle. The outer Dagger engine places the container on a 10.87.0.0/16 network with a DNS server at 10.87.0.1. When a nested Dagger engine starts inside the DinD container, it creates its own dagger0 bridge in the same 10.87.0.0/16 range. That route captures all traffic to 10.87.0.1, making the DNS server unreachable from containers managed by the inner engine.

The subnet is hard-coded in Dagger's source (DefaultCIDR = "10.87.0.0/16" in github.com/dagger/dagger/network) with no configuration option to change it. A session-level config file is tracked upstream but not implemented yet. Until then, the workaround below is necessary.

The fix lives in the sidecar wrapper script: before starting dockerd, save the current DNS server and tell Docker (via daemon.json) to hand out 172.17.0.1 (the Docker bridge gateway) as the DNS server for containers. The wrapper then adds iptables DNAT rules that forward DNS queries arriving at 172.17.0.1:53 to the real DNS server. Containers managed by the inner Docker — including a nested Dagger engine — can always reach the bridge gateway, so DNS works transparently.

1. Docker daemon sidecar

The docker:dind image handles cgroup setup, storage driver selection, and dockerd lifecycle. The wrapper script adds DNS forwarding on top.

TLS is explicitly disabled (DOCKER_TLS_CERTDIR set to an empty string) because the daemon only talks to the client container inside Dagger's sandbox. Docker warns that "in future versions this will be a hard failure." This is safe to ignore: Dagger owns this daemon and will update their images when Docker enforces TLS. Every Dagger user sees the same warning.

The stock dockerd-entrypoint.sh emits a few warnings on nf_tables-only kernels (no legacy iptables modules loaded):

cat: can't open '/proc/net/arp_tables_names': No such file or directory
iptables v1.8.11 (nf_tables)
sed: write error

These come from Docker's own iptables backend detection logic, not from our wrapper. The entrypoint probes /proc/net/{ip,ip6,arp}_tables_names to decide whether to use legacy iptables or nf_tables, then unconditionally prints iptables --version. On kernels that only have nf_tables loaded (no ip_tables / arp_tables modules), the probes fail and the modprobe fallback produces the sed error. Reproducible with plain Docker — no Dagger involved:

docker run --rm --privileged docker:27-dind sh -c 'dockerd-entrypoint.sh dockerd 2>&1 | head -10'

This is harmless: the nf_tables fallback works correctly. Upgrading to docker:28-dind does not help — the same entrypoint code is present.

@function
def dind_service(self) -> dagger.Service:
    """Return a Docker daemon running as a sidecar service.

    Wraps the docker:dind entrypoint with DNS forwarding so
    nested Dagger engines can resolve names despite the
    10.87.0.0/16 subnet collision.
    """
    return (
        dag.container()
        .from_(self.pinned(self._dind_engine_image))
        .with_env_variable("DOCKER_TLS_CERTDIR", "")
        .with_env_variable("TINI_SUBREAPER", "")
        .with_new_file(
            "/etc/docker/daemon.json",
            '{"dns": ["172.17.0.1"]}',
        )
        .with_exec([
            "mv",
            "/usr/local/bin/dockerd-entrypoint.sh",
            "/usr/local/bin/dockerd-entrypoint-orig.sh",
        ])
        .with_new_file(
            "/usr/local/bin/dockerd-entrypoint.sh",
            _DIND_DNS_ENTRYPOINT,
            permissions=0o755,
        )
        .with_mounted_cache("/var/lib/docker", dag.cache_volume("dind-docker"))
        .with_exposed_port(2375)
        .as_service(
            use_entrypoint=True,
            insecure_root_capabilities=True,
        )
    )

2. Preparing a DinD client container

The client container only needs the Docker CLI — the daemon runs in the sidecar. The CLI binary is copied from the same docker:dind image (statically linked, works on any Linux).

@function
def dind_container(
    self,
    base: dagger.Container | None = None,
) -> dagger.Container:
    """Return a container with the Docker CLI installed.

    If base is provided, the CLI is added to it.
    Otherwise uses Lib.dind_ubuntu_image.
    """
    if base is None:
        base = dag.container().from_(self.pinned(self._dind_base_image))
    docker_cli = (
        dag.container().from_(self.pinned(self._dind_engine_image)).file("/usr/local/bin/docker")
    )
    return base.with_file("/usr/local/bin/docker", docker_cli)

3. Running commands with dockerd

The client container binds the Docker daemon sidecar and sets DOCKER_HOST so the CLI talks to it over TCP. The daemon runs in the sidecar with elevated privileges — the client container does not need insecure_root_capabilities.

@function
def dind_with_docker(
    self,
    cmd: str,
    ctr: dagger.Container | None = None,
) -> dagger.Container:
    """Run a shell command inside a container with Docker available.

    Binds a Docker daemon sidecar and sets DOCKER_HOST so the
    Docker CLI in the container talks to the sidecar over TCP.
    """
    if ctr is None:
        ctr = self.dind_container()
    return (
        ctr
        .with_service_binding("docker", self.dind_service())
        .with_env_variable("DOCKER_HOST", "tcp://docker:2375")
        .with_exec(["bash", "-c", cmd])
    )

4. Running the test suite inside DinD

Dogfooding: build a container that has everything needed to run the project tests (Python, pytest, Dagger CLI) on top of the DinD base, then execute ./test-host.sh with dockerd available.

@function
def dind_run_tests(
    self,
    src: Annotated[
        dagger.Directory,
        DefaultPath("."),
        Ignore(["**",
                "!src/lib/", "!src/ralph.yml", "!sdk/", "!tests/", "!examples/",
                "!dagger.json", "!.daggerignore", "!pyproject.toml",
                "!test-host.sh"]),
    ],
) -> dagger.Container:
    """Run the project test suite inside Docker-in-Docker (dogfooding).

    Builds a test-ready container from the DinD base, installs Python,
    pytest, and the Dagger CLI, mounts the project source, and runs
    ./test-host.sh with dockerd available.
    Source is mounted last so package installs are cached.
    """
    ctr = self.dind_container()
    ctr = (
        ctr
        .with_exec(["apt-get", "update"])
        .with_exec([
            "apt-get", "install", "--yes",
            "curl", "python3", "python3-pip", "python3-venv",
        ])
        .with_exec([
            "pip3", "install", "--break-system-packages",
            "pytest", "pytest-asyncio",
        ])
        .with_exec([
            "bash", "-c",
            "curl -fsSL https://dl.dagger.io/dagger/install.sh"
            " | DAGGER_VERSION=0.20.3 BIN_DIR=/usr/local/bin sh",
        ])
        .with_directory("/work", src)
        .with_workdir("/work")
        .with_directory("/work/.git", dag.directory())
    )
    cmd = (
        "echo '=== dockerd ready ===' && "
        "docker info --format 'Docker: {{.ServerVersion}}' && "
        "echo '=== running test suite ===' && "
        "./test-host.sh"
    )
    return self.dind_with_docker(cmd=cmd, ctr=ctr)

5. Running org-babel in a container

The host emacs version affects org-babel output (whitespace, formatting). To get reproducible results we run run-host.sh, tangle-host.sh, and init-examples-host.sh inside containers with a pinned emacs.

Two container flavors:

  • emacs_container — lightweight: just emacs, git, python, ruff. Used by dind_tangle which doesn't need Docker or Dagger.
  • dind_emacs_container — adds emacs, ruff, the Docker CLI, and the Dagger CLI. Used by dind_run_org and dind_init_examples whose babel blocks call dagger call.
@function
def emacs_container(
    self,
    src: dagger.Directory,
) -> dagger.Container:
    """Return a lightweight container with emacs, git, python, and ruff.

    ~elpa-htmlize~ is installed so ~ox-html~ exports can produce
    syntax-highlighted source blocks.  ~curl~ is there so
    ~export-html-host.sh~ can fetch water.css into the shared
    =.tangle-deps/= cache.
    """
    return (
        dag.container()
        .from_(self.pinned(self._dind_base_image))
        .with_exec(["apt-get", "update"])
        .with_exec([
            "apt-get", "install", "--yes",
            "emacs-nox", "elpa-htmlize", "curl",
            "git", "python3", "python3-pip", "python3-venv",
        ])
        .with_exec([
            "pip3", "install", "--break-system-packages", "ruff",
        ])
        .with_directory("/work", src)
        .with_workdir("/work")
    )
@function
def dind_emacs_container(self) -> dagger.Container:
    """Return a DinD container with emacs, git, python, ruff, and dagger CLI.

    Does NOT mount source — callers add it last so the entire
    tool install chain is cacheable.
    """
    ctr = self.dind_container()
    return (
        ctr
        .with_exec(["apt-get", "update"])
        .with_exec([
            "apt-get", "install", "--yes",
            "curl", "emacs-nox", "git", "python3", "python3-pip", "python3-venv",
        ])
        .with_exec([
            "pip3", "install", "--break-system-packages", "ruff",
        ])
        .with_exec([
            "bash", "-c",
            "curl -fsSL https://dl.dagger.io/dagger/install.sh"
            " | DAGGER_VERSION=0.20.3 BIN_DIR=/usr/local/bin sh",
        ])
    )

Each function snapshots /work before running the script, then returns only the diff — files actually modified by the script. This avoids clobbering files the user may be editing locally.

Each function declares its source files via DefaultPath(".") + Ignore(["**", "!pattern", ...]) — a per-function allowlist. The engine filters the context directory before the function runs, producing a directory whose graph identity depends only on the non-ignored files. This avoids a known Dagger caching bug where dag.current_module().source() changes its graph identity when unrelated files are added or removed, busting the exec cache even though the filtered content is identical. See studies/caching/readme.org for the full analysis and reproducer.

5.1. Tangling org files

Tangle only needs emacs — no Docker, no Dagger. Runs ./tangle-host.sh in the lightweight emacs_container.

@function
def dind_tangle(
    self,
    src: Annotated[
        dagger.Directory,
        DefaultPath("."),
        Ignore(["**",
                "!src/", "!tests/", "!examples/", "!studies/", "!.clk/",
                "!*.org", "!*.sh", "!*.el",
                "!dagger.json"]),
    ],
) -> dagger.Directory:
    """Tangle org files inside a container and return only the modified files."""
    ctr = self.emacs_container(src=src)
    ctr = ctr.with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
    before = ctr.directory("/work")
    after = ctr.with_exec(["./tangle-host.sh"]).directory("/work")
    return before.diff(after)

5.2. Running org-babel blocks

Runs ./run-host.sh with dockerd available (babel blocks call dagger call) and returns only the modified files.

@function
def dind_run_org(
    self,
    src: Annotated[
        dagger.Directory,
        DefaultPath("."),
        Ignore(["**",
                "!src/", "!examples/", "!studies/",
                "!tests/dagger", "!tests/ralph-log-filter",
                "!tests/ralph_log_sample.txt",
                "!*.org", "!*.sh", "!*.el",
                "!dagger.json", "!.daggerignore", "!pyproject.toml",
                "!sdk/"]),
    ],
    files: list[str] | None = None,
    no_cache: bool = False,
) -> dagger.Directory:
    """Run org-babel blocks inside a DinD container and return only the modified files.

    If files is given, only those org files are processed.
    """
    ctr = self.dind_emacs_container()
    ctr = (
        ctr
        .with_directory("/work", src)
        .with_workdir("/work")
        .with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
    )
    before = ctr.directory("/work")
    cmd = "./run-host.sh"
    if no_cache:
        cmd += " --no-cache"
    if files:
        cmd += " " + " ".join(files)
    after = self.dind_with_docker(cmd=cmd, ctr=ctr).directory("/work")
    return before.diff(after)

5.3. Initializing examples

Runs ./init-examples-host.sh with dockerd available and returns only the modified files.

@function
def dind_init_examples(
    self,
    src: Annotated[
        dagger.Directory,
        DefaultPath("."),
        Ignore(["**",
                "!src/lib/", "!examples/",
                "!*.sh", "!*.el",
                "!dagger.json", "!.daggerignore", "!pyproject.toml",
                "!sdk/"]),
    ],
    from_scratch: bool = False,
    no_cache: bool = False,
) -> dagger.Directory:
    """Init example modules inside a DinD container and return only the modified files."""
    ctr = self.dind_emacs_container()
    ctr = (
        ctr
        .with_directory("/work", src)
        .with_workdir("/work")
        .with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
    )
    before = ctr.directory("/work")
    cmd = "./init-examples-host.sh"
    if from_scratch:
        cmd += " --from-scratch"
    elif no_cache:
        cmd += " --no-cache"
    after = self.dind_with_docker(cmd=cmd, ctr=ctr).directory("/work")
    return before.diff(after)

5.4. Exporting org files to HTML

Exports org files to HTML for GitHub Pages. Uses the lightweight emacs_container (no Docker needed) and returns the _site/ directory containing the HTML output.

@function
def export_html(
    self,
    src: Annotated[
        dagger.Directory,
        DefaultPath("."),
        Ignore(["**",
                "!src/*.org", "!tests/testing.org",
                "!examples/*/readme.org",
                "!*.org", "!*.sh", "!*.el",
                "!dagger.json"]),
    ],
) -> dagger.Directory:
    """Export org files to HTML with noweb expansion for GitHub Pages."""
    ctr = self.emacs_container(src=src)
    ctr = ctr.with_mounted_cache("/work/.tangle-deps", dag.cache_volume("tangle-deps"))
    return ctr.with_exec(["./export-html-host.sh"]).directory("/work/_site")
dagger call dind-with-docker --cmd="docker info --format '{{.OSType}}'" stdout

6. Image references for pin management

See alpine for the convention. These properties alias the user-facing dind_image and dind_ubuntu_image fields so the _*_image discovery convention picks them up.

@property
def _dind_engine_image(self) -> str:
    return self.dind_image
@property
def _dind_base_image(self) -> str:
    return self.dind_ubuntu_image

Author: root

Created: 2026-04-18 Sat 21:16

Validate