Building images

build builds a Dockerfile with a builder of nspawn’s own, or runs mkosi on an mkosi configuration. A directory with a Dockerfile (or dockerfile, Containerfile), -f naming one, - on standard input or any of the Dockerfile flags below make it a Dockerfile build; a directory with mkosi.conf or mkosi.conf.d and no Dockerfile, or the mkosi flags, an mkosi build. Either result is stored like a pulled image and can be started right away or pushed. A build of a tag replaces the image of that reference under the tag’s local name as long as nothing ran from it, whatever it was pulled or built as: the tag moves to the new build, so a script that builds and runs its image builds it again. A machine that has the name (one that ran, starts or runs), or an image of another reference, keeps it, and the build goes beside it with the same reference, into the image an earlier build of the tag put there or else under a free name (app-2), which a note says; run, create, compose and the Docker API find it by the reference, the image before the machine. An mkosi build given --name is refused instead when a machine or an image of another reference has that name. --force replaces whatever has the name, a machine that ran included, and refuses one that runs. -f names the Dockerfile, so --force has no short form.

Dockerfiles

sudo nspawn build [DIRECTORY|-] [-t TAG]... [-f DOCKERFILE] [--build-arg KEY[=VALUE]]...
                  [--target STAGE] [--no-cache] [--no-cache-filter STAGE]... [--pull]
                  [--label KEY=VALUE]... [--network default|none|host] [--add-host HOST:IP]...
                  [--allow network.host] [--secret id=ID[,src=PATH|,env=VAR]]...
                  [--shm-size SIZE] [--ulimit NAME=SOFT[:HARD]]... [-q] [--progress TYPE]
                  [--iidfile FILE] [--push] [--no-verify] [--platform PLATFORM] [--load]
                  [--backend BACKEND] [--mode MODE] [--force]

The Dockerfile is read as BuildKit’s frontend reads it (a # syntax= of docker/dockerfile:1, 1.N, 1.N.M or labs is taken, another frontend image is refused), and run by a builder of nspawn’s own: every stage the target needs (--target, the last one otherwise), multi-stage builds, COPY --from, ARG and --build-arg, ONBUILD, heredocs, COPY --chown --chmod --parents --exclude --link, ADD of local archives (gzip, bzip2, xz, zstd or plain tar, unpacked) and of http(s) URLs (--checksum), and RUN --mount of the cache, bind, tmpfs and secret kinds, and --build-arg SOURCE_DATE_EPOCH=SECONDS for the time of the image and of the history entries the build adds. The context is the directory given. The command line packs it as the calling user, everything owned by root, times to the nanosecond and file capabilities kept, and leaves out what .dockerignore excludes (Dockerfile.dockerignore next to the Dockerfile wins over it); - takes a context tar, unpacked with its owners, or a Dockerfile with no context, from standard input.

An archive that ADD unpacks goes into the destination entry by entry: an entry replaces what is at its path unless both are directories, a symlink on the way is followed inside the destination, a parent the archive does not list is made 0755, and its owners (unless --chown), modes, times, FIFOs, device nodes and extended attributes are kept. A file ADD fetches from a URL has the time of its Last-Modified, the epoch without one.

A static program built in one stage and copied into an empty image:

FROM alpine:3.22 AS build
RUN apk add --no-cache gcc musl-dev
COPY hello.c /src/
RUN gcc -static -o /hello /src/hello.c

FROM scratch
COPY --from=build /hello /hello
ENTRYPOINT ["/hello"]

$ sudo nspawn build -t hello:1 .
docker.io/library/alpine:3.22: manifest 3e9b4b680bfc with 1 layer(s)
#1 [build 1/3] RUN apk add --no-cache gcc musl-dev
#1 0.070 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/APKINDEX.tar.gz
...
#1 1.387 OK: 172 MiB in 29 packages
#1 DONE 8.5s
#2 [build 2/3] COPY hello.c /src/
#2 DONE 0.2s
#3 [build 3/3] RUN gcc -static -o /hello /src/hello.c
#3 DONE 0.2s
#4 [stage-1 1/2] COPY --from=build /hello /hello
#4 DONE 0.7s
#5 [stage-1 2/2] ENTRYPOINT [/hello]
#5 DONE 0.0s
image hello-1 is ready: nspawn run hello-1, nspawn push hello-1

$ sudo nspawn run --rm hello:1
hello from a scratch image

Every step runs in a root of its own, an overlay of the layers so far, and what it changes becomes a layer. A RUN runs under systemd-nspawn with the confinement of an app machine (the default capabilities, the system call filter, no user namespace so that files keep their owners), as PID 1 with the hostname buildkitsandbox, in a network namespace the build sets up on the default bridge; RUN --network=none has none, and --network=host needs --allow network.host. A RUN --mount=type=tmpfs is mode 1777.

Steps that ran before with the same inputs come from the build cache, so the same build again takes a moment and gives the same image ID:

$ sudo nspawn build -t hello:1 .
#1 [build 1/3] RUN apk add --no-cache gcc musl-dev
#1 CACHED
#2 [build 2/3] COPY hello.c /src/
#2 CACHED
#3 [build 3/3] RUN gcc -static -o /hello /src/hello.c
#3 CACHED
#4 [stage-1 1/2] COPY --from=build /hello /hello
#4 CACHED
#5 [stage-1 2/2] ENTRYPOINT [/hello]
#5 DONE 0.0s
image hello-1 is ready: nspawn run hello-1, nspawn push hello-1

A cache entry keeps the layers its step ran on, so a --no-cache build of another Dockerfile with the same first steps does not invalidate this one’s later steps. --no-cache builds every step again, --no-cache-filter STAGE the steps of a stage, and --pull asks the registry for the base images even when they are here.

--rm, --force-rm, -m, --memory-swap, -c, --cpu-period, --cpu-quota, --cpuset-cpus, --cpuset-mems, --compress, --squash, --isolation and --security-opt, which Makefiles still give, are taken and ignored, three of them (--isolation, --security-opt, --squash) with a warning.

A failing step ends the build with process "/bin/sh -c false" did not complete successfully: exit code: 1, and leaves nothing behind; a step whose process a signal ends exits with 128 plus the signal’s number, and a build whose command line goes away (Ctrl-C) stops the same way.

The build cache

$ sudo nspawn builder du
Reclaimable:	1.112GB
Total:		1.141GB

builder du -v lists every entry, builder prune removes the cache entries no image shares (-a all of them, --filter until=24h, --keep-storage 10g), system df counts the cache and system prune removes it.

What the builder does not do

No SSH forwarding (--ssh), no output other than a local image (--output, --cache-to), no named, remote or git contexts, no ADD of a git repository, no attestations (--provenance=false and --sbom=false are taken), only this host’s platform, no RUN --security=insecure or RUN --device, no --check or --call, and no linter: a # check= directive or BUILDKIT_DOCKERFILE_CHECK is read and its words checked, and no rule runs. Stages run one after another; COPY --link copies as a plain COPY; /run is a tmpfs during a RUN, so what a step writes there is not kept, and systemd-nspawn adds container=systemd-nspawn, container_uuid, USER and LOGNAME to a step’s environment. Each step starts from the layers as the image keeps them, so what a layer does not carry (ACLs and extended attributes other than file capabilities) is gone for the next step. FIFOs and device nodes of a context directory are left out of it; those of a context tar are kept. A RUN of USER 0:GID or USER root:GROUP runs with group 0, and a RUN as root without root’s supplementary groups: systemd-nspawn sets no group of its own for root.

mkosi

sudo nspawn build [DIRECTORY] -t TAG [--name NAME]
                  [-d DISTRIBUTION] [-r RELEASE] [--profile PROFILE]...
                  [--backend BACKEND] [--mode MODE] [--force] [--keep-output]
                  [-- MKOSI_ARGUMENTS...]

build runs mkosi on a directory that has a mkosi.conf or a mkosi.conf.d (the current directory by default), with --format=oci, and imports the result exactly as pull would import an image from a registry. The service runs mkosi, so mkosi has to be installed where the service finds it (/usr/bin). The same configuration tree that works with mkosi on its own works here; --distribution, --release, --profile and anything after -- are passed through.

The command line that mkosi receives is:

mkosi --directory=DIRECTORY --format=oci --compress-output=zstd
      --output-directory=/var/lib/nspawn/builds/mkosi-NAME-TIMESTAMP-SUFFIX
      --cache-directory=/var/lib/nspawn/cache/mkosi
      --workspace-directory=/var/lib/nspawn/cache/mkosi-workspace/mkosi-NAME-TIMESTAMP-SUFFIX
      --image-id=REPOSITORY [--distribution=...] [--release=...] [--profile=...]...
      [MKOSI_ARGUMENTS...] --force build

The / characters of the repository are replaced by - in the image id. The package cache is kept between builds; the workspace is deleted at the end, and the output directory after the import unless --keep-output is given.

When mkosi is done, nspawn finds the OCI layout it produced, adds a few annotations to the manifest (org.opencontainers.image.version with the tag when mkosi wrote none from ImageVersion=, org.opencontainers.image.ref.name with the full reference, and org.nspawn.builder), verifies every blob against its digest, copies the blobs into the store and installs the image with the chosen backend, with build as its origin. The image can be started right away, cloned with create or pushed. The labels of its OCI configuration, which mkosi writes from OciLabels=, are kept with it: nspawn inspect shows them as image_labels, and they travel with the image when it is pushed.

A minimal configuration for a Fedora machine that boots systemd, taken from the test suite of nspawn:

[Distribution]
Distribution=fedora
Release=44

[Output]
ImageId=e2e-built
OciLabels=org.nspawn.e2e=built

[Content]
Bootable=no
SELinuxRelabel=no
RootPassword=root
Packages=
        systemd
        systemd-networkd
        systemd-resolved
        dbus-broker
        passwd
        util-linux

systemd-networkd is what configures host0 on the bridge, and dbus-broker (or dbus) is what lets shell reach the machine’s systemd. Then:

sudo nspawn build -t team/app:1 ./app
sudo nspawn start team-app-1

push

sudo nspawn push IMAGE [--to REFERENCE]

IMAGE is a local image name or the reference it was pulled from or built as. By default the image is pushed under its own reference; --to pushes it under another one, for example to retag it or to send it to a different registry:

sudo nspawn push team/app:1
sudo nspawn push team-app-1 --to registry.example/team/app:2

push authenticates first, with the credentials stored for that registry, and tells you which nspawn login to run when the registry wants some it does not have. The blobs that are already on the registry are skipped and the manifest is uploaded last. A push needs a tag, not a digest, and the image’s manifest and blobs must still be in the store (images rm removes them). See Registries and credentials.

The hub images

The images the team publishes on hub.nspawn.org are built with mkosi from the definitions in nspawn/mkosi-definitions. Use them as a starting point for your own images, or open a pull request there to change what the hub ships.