Overview

What nspawn is

nspawn is a command line tool that manages systemd-nspawn machines:

• Images are OCI images that come from a registry. The default registry is the hub at hub.nspawn.org, where the nspawn.org team publishes images of common distributions built with mkosi. Docker Hub and any other registry work as well: search looks on the hub and on Docker Hub at once, and login keeps credentials per registry.
• An image of the hub is verified against its signature before anything comes down; layers are downloaded once, verified against their digests and shared between the machines that use them. A machine’s root file system is assembled from them by a backend: an overlayfs mount, a native systemd.mstack directory, or a flat copy. create makes more machines from an image that is already local, without touching the registry.
• Machines are started, inspected and stopped through the D-Bus APIs of systemd-machined and of systemd itself. machinectl and importctl are never called, and every machine is an ordinary systemd-nspawn@NAME.service unit that machinectl, systemctl and journalctl see like any other. A drop-in makes that unit call nspawn before the machine starts, once it runs and after it ends, so machinectl start, a unit enabled at boot or a crash get the same network setup and cleanup as nspawn start and nspawn stop.
• A machine’s settings are flags, remembered for it. -p publishes ports, -v mounts host directories or named volumes, -l labels a machine, --restart gives it a restart policy, and -m, --cpus and --pids-limit bound its resources. --health-cmd probes it, --secret hands it a secret, and --depends-on starts other machines before it, at boot as well. For an app, -e, --entrypoint, -u, -w and the arguments after -- change what it runs. --hostname, --cap-drop, --read-only, --tmpfs, --device, --gpus, --dns, --add-host, --ulimit and the rest shape the machine further. For scripts, ps --filter and ps --format select and print machines, inspect -f prints a template over a record, wait and port give a machine’s exit code and published ports, and system df and system prune count and free what nspawn keeps on disk.
• nspawn compose runs docker compose files: each service becomes a machine per replica, and its networks, volumes and secrets become nspawn’s. See Compose.
• build builds a Dockerfile with a builder of nspawn’s own, or runs mkosi on a directory with a mkosi.conf, and imports the result as a local image; push uploads an image to a registry, skipping the layers that are already there. See Building.
• The Docker Engine API, on a socket of the host (nspawn-api.socket) and in each machine that mounts /var/run/docker.sock: the docker command, GitLab Runner’s docker executor and other clients make, run, attach to, exec in and copy to and from containers, pull images and make volumes and networks, and reverse proxies that route by labels and dashboards find the machines as containers. The server is a client of the service, as the command line is. auto-update moves labelled machines to the image their tag names now. See The Docker API.
• generate prints the commands that make the same machines on another host, every flag they keep included.

The work is done by a service on the system bus, org.nspawn, and the command line is one of its clients, the way machinectl and systemctl are clients of machined and systemd. The service is started by the bus when a command arrives and exits when it has been idle for a minute. Between commands only these run: the healthcheck of a machine that has one (nspawn-health-NAME.service), the Docker API of a machine that asked for one and the host’s once you enable it, which systemd starts when a client connects, and the daily auto-update timer once you enable it. See The service.

It is one binary written in Rust. Its own state lives under /var/lib/nspawn, the assembled machines under /var/lib/machines; the units, drop-ins and .nspawn files it generates are plain files you can read, and its network rules live in one nftables table, ip nspawn.

The service

Every command is a method call on org.nspawn.Manager: images, machines, the network and credentials are methods. The long operations (pull, push, build, create, rm, images rm and prune, cp, volume rm and prune, secret rm, network rm, connect, disconnect and prune, builder prune, auto-update) come back as job objects that report their output and result, and exec, logs, events and an attached run hand their streams or terminal over the bus, with a process object that says how they ended. A method or job that fails says what failed with a name under org.nspawn.Error (NoSuchMachine, InUse, Invalid and the rest) and the message the command line prints. Scripts that would rather not speak D-Bus get the same dictionaries as JSON from inspect and --json on the listings.

Who may call what is polkit’s answer, the way it is for machined and systemd. The bus lets everyone in and the service asks polkit about the caller, under two actions:

Both are for administrators by default, so sudo nspawn ... works, and a desktop session (or a terminal where you started pkttyagent) is asked for a password instead. Root is never asked, which is also how the service keeps working where polkit is not installed: there, nobody but root can call it. The host’s socket of the Docker API asks polkit the same two actions about each of its clients, never letting it prompt. One interface asks nothing: org.nspawn.Names hands out the names of machines, images, networks and volumes, and nothing else about them, to any user.

To drive nspawn without a password, an administrator hands an action to a group in a rule of their own. The packages ship one as an example in their documentation directory; copy it to /etc/polkit-1/rules.d/50-nspawn.rules:

polkit.addRule(function (action, subject) {
    if (action.id.startsWith("org.nspawn.") && subject.isInGroup("wheel")) {
        return polkit.Result.YES;
    }
});

Everyone in that group can then run commands as root inside a machine and mount any path of the host into one, which is to say they are administrators of the host. Granting org.nspawn.inspect alone lets the group look without changing anything, though what it sees includes every machine’s -e variables and the build cache’s steps, --build-arg values among them.

A package installs the service; for a binary you built yourself, sudo nspawn daemon --install writes the bus policy, the polkit actions, the activation file and the unit, and tells systemd and the bus about them. Its journal is the usual one:

journalctl -u nspawn.service

Machines and apps

nspawn tells two kinds of images apart when it installs them:

The kind is detected at pull or build time and can be forced with --mode boot or --mode app. Both kinds join the bridge network, are listed by ps, stopped by stop and read by logs; the differences are described in Machines.

Networking

Every machine joins nspawn0, a bridge that nspawn manages itself: fixed addresses, NAT, published ports (-p 8080:80) and a generated /etc/hosts with the names of the other machines. network create makes more networks, each a bridge of its own, a machine may join several of them with names of its own on each, --network host shares the host’s network instead, --network none gives a machine no network, and --network veth gives booted machines the classic systemd-nspawn virtual ethernet pair configured by systemd-networkd. See Networking.

The pieces

What nspawn is not

• It is not a container runtime of its own: systemd-nspawn runs the machines, systemd supervises them, machined tracks them. nspawn only drives them, and its own service holds no machine open: it is started on demand and goes away again.
• It is not an orchestrator. Compose files run on one host; there is no service discovery beyond the names on the networks and no scheduling across hosts.
• It does not bring mkosi: mkosi configurations need mkosi installed on the host. Dockerfiles build with nspawn’s own builder.
• It does not sign images: the hub’s are signed by the workflow that builds them, and pull verifies one of those signatures before it downloads anything (see Signed images). Every blob is then checked against the sha256 digest in the manifest while it downloads, and registries are reached over HTTPS only.