This is the multi-page printable view of this section. Click here to print.
nspawn manages systemd-nspawn machines: images come from an OCI registry (the hub, Docker Hub or any other), are stored as shared layers and are started, inspected and stopped through the D-Bus APIs of systemd-machined and systemd. The work is done by a service on the system bus and the command line is its client. This documentation describes nspawn 3.1.0.
If you are new, start with the overview and then get started. The command reference lists every command and option.
nspawn is a command line tool that manages systemd-nspawn machines:
hub.nspawn.org,
where the nspawn.org team publishes images of common distributions built with
mkosi. Docker Hub and any other registry
work as well: search looks on the hub and on Docker Hub at once, and
login keeps credentials per registry.systemd.mstack directory, or a flat copy. create makes more machines from
an image that is already local, without touching the registry.machinectl and importctl are never called, and
every machine is an ordinary systemd-nspawn@NAME.service unit that
machinectl, systemctl and journalctl see like any other. A drop-in makes that unit call nspawn before
the machine starts, once it runs and after it ends, so machinectl start, a
unit enabled at boot or a crash get the same network setup and cleanup as
nspawn start and nspawn stop.-p publishes ports, -v
mounts host directories or named volumes, -l labels a machine, --restart
gives it a restart policy, and -m, --cpus and --pids-limit bound its
resources.
--health-cmd probes it, --secret hands it a secret, and --depends-on
starts other machines before it, at boot as well. For an app, -e,
--entrypoint, -u, -w and the arguments after -- change what it runs.
--hostname, --cap-drop, --read-only, --tmpfs, --device, --gpus,
--dns, --add-host, --ulimit and the rest shape the machine further. For
scripts, ps --filter and ps --format select and print machines,
inspect -f prints a template over a record, wait and port give a
machine’s exit code and published ports, and system df and system prune
count and free what nspawn keeps on disk.nspawn compose runs docker compose files: each service becomes a machine per
replica, and its networks, volumes and secrets become nspawn’s. See
Compose.build builds a Dockerfile with a builder of nspawn’s own, or runs mkosi on a
directory with a mkosi.conf, and imports the result
as a local image; push uploads an image to a registry, skipping the layers
that are already there. See Building.nspawn-api.socket) and in
each machine that mounts /var/run/docker.sock: the docker command, GitLab
Runner’s docker executor and other clients make, run, attach to, exec in and
copy to and from containers, pull images and make volumes and networks, and
reverse proxies that route by labels and dashboards find the machines as
containers. The server is a client of the service, as the command line is.
auto-update moves labelled machines to the image their tag names now. See
The Docker API.generate prints the commands that make the same machines on another host,
every flag they keep included.The work is done by a service on the system bus, org.nspawn, and the command
line is one of its clients, the way machinectl and systemctl are clients of
machined and systemd. The service is started by the bus when a command arrives
and exits when it has been idle for a minute. Between commands only these run:
the healthcheck of a machine that has one (nspawn-health-NAME.service), the
Docker API of a machine that asked for one and the host’s once you enable it,
which systemd starts when a client connects, and the daily auto-update timer
once you enable it. See
The service.
It is one binary written in Rust. Its own state lives under /var/lib/nspawn,
the assembled machines under /var/lib/machines; the units, drop-ins and
.nspawn files it generates are plain files you can read, and its network rules
live in one nftables table, ip nspawn.
Every command is a method call on org.nspawn.Manager: images, machines, the
network and credentials are methods. The long operations (pull, push, build,
create, rm, images rm and prune, cp, volume rm and prune, secret rm, network rm,
connect, disconnect and prune, builder prune, auto-update) come back as job
objects that report their output and result, and exec, logs, events and an
attached run hand their streams or terminal over the bus, with a process object
that says how they ended. A method or job that fails says what failed with a
name under org.nspawn.Error (NoSuchMachine, InUse, Invalid and the rest)
and the message the command line prints. Scripts that would rather not speak
D-Bus get the same dictionaries as JSON from inspect and --json on the
listings.
Who may call what is polkit’s answer, the way it is for machined and systemd. The bus lets everyone in and the service asks polkit about the caller, under two actions:
| Action | Methods |
|---|---|
org.nspawn.inspect | The ones that only read: images ls and images inspect, ps and machines ls, inspect, info, generate, top, stats, events, wait, port, system df, builder du, network ls and network inspect, volume ls and volume inspect, secret ls and secret inspect. |
org.nspawn.manage | Everything else: pulling, building, starting, stopping, restarting, pausing, removing, exec, shell, cp, logs, volumes, secrets, the registry commands and the credentials. |
Both are for administrators by default, so sudo nspawn ... works, and a
desktop session (or a terminal where you started pkttyagent) is
asked for a password instead. Root is never asked, which is also how the
service keeps working where polkit is not installed: there, nobody but root can
call it. The host’s socket of the Docker API asks polkit the
same two actions about each of its clients, never letting it prompt. One
interface asks nothing: org.nspawn.Names hands out the names of
machines, images, networks and volumes, and nothing else about them, to any
user.
To drive nspawn without a password, an administrator hands an action to a group
in a rule of their own. The packages ship one as an example in their
documentation directory; copy it to /etc/polkit-1/rules.d/50-nspawn.rules:
polkit.addRule(function (action, subject) {
if (action.id.startsWith("org.nspawn.") && subject.isInGroup("wheel")) {
return polkit.Result.YES;
}
});
Everyone in that group can then run commands as root inside a machine and mount
any path of the host into one, which is to say they are administrators of the
host. Granting org.nspawn.inspect alone lets the group look without changing
anything, though what it sees includes every machine’s -e variables and the
build cache’s steps, --build-arg values among them.
A package installs the service; for a binary you built yourself,
sudo nspawn daemon --install writes the bus policy, the polkit actions, the
activation file and the unit, and tells systemd and the bus about them. Its
journal is the usual one:
journalctl -u nspawn.service
nspawn tells two kinds of images apart when it installs them:
| Kind | What the image contains | How it runs |
|---|---|---|
| boot | An init system (systemd) and no other command, like the hub images | Booted with --boot, as machinectl start does. shell opens machined’s login session inside; exec enters the machine’s namespaces and needs nothing from it. |
| app | Anything else, for example an image from Docker Hub | Its entrypoint runs under systemd-nspawn’s stub init, with the environment, working directory, user and stop signal from the OCI config, or as PID 1 of the machine when the image carries s6-overlay (as linuxserver.io’s images do) or --init=false asks for it. Its network namespace is prepared by nspawn before the program starts, so the network is there when it starts. exec and shell enter the namespaces. |
The kind is detected at pull or build time and can be forced with
--mode boot or --mode app. Both kinds join the bridge network, are listed
by ps, stopped by stop and read by logs; the differences are described
in Machines.
Every machine joins nspawn0, a bridge that nspawn manages itself: fixed
addresses, NAT, published ports (-p 8080:80) and a generated
/etc/hosts with the names of the other machines. network create makes
more networks, each a bridge of its own, a machine may join several of
them with names of its own on each, --network host shares the host’s network
instead, --network none gives a machine no network, and --network veth
gives booted machines the classic systemd-nspawn virtual ethernet pair
configured by systemd-networkd. See Networking.
| Piece | Where | Role |
|---|---|---|
nspawn | github.com/nspawn/nspawn | The tool this documentation is about: the service and its command line. |
| The hub | hub.nspawn.org | An OCI registry with the images the team publishes. It is the default registry of the tool. |
| mkosi definitions | github.com/nspawn/mkosi-definitions | The public mkosi configuration the hub images are built from. |
| Blog | blog.nspawn.org | Release notes and news. |
pull verifies one of those signatures before it downloads
anything (see Signed images). Every blob is
then checked against the sha256 digest in the manifest while it downloads,
and registries are reached over HTTPS only./run). cgroup v2 is
required.overlay backend (the
layers are mounted with lowerdir+=). On a kernel without overlayfs, auto
extracts images as flat directories.ip and nft) for the bridge network. Nothing
else: the bridge does not need systemd-networkd or NetworkManager on the host.
Only --network veth needs systemd-networkd, and only --interface with a
wifi adapter needs iw (for an app on the bridge) or systemd 256 (for a
booted machine).sudo, which works everywhere, including hosts without polkit.shell, which uses machined’s login
session; the hub images have it. exec enters the machine’s namespaces and
needs nothing inside.Debian 13 or later and Ubuntu 24.04 or later, on amd64:
sudo curl -fsSLo /etc/apt/sources.list.d/nspawn.sources https://apt.nspawn.org/nspawn.sources
sudo apt update
sudo apt install nspawn
Fedora 44 or later, on x86_64:
sudo curl -fsSLo /etc/yum.repos.d/nspawn.repo https://rpm.nspawn.org/nspawn.repo
sudo dnf install nspawn
dnf asks to import the signing key: compare its fingerprint with the one
below. With SELinux enforcing, dnf also installs
nspawn-selinux, the policy exec, shell, logs and cp need.
The AUR has nspawn, from the
latest release, and nspawn-git,
from the latest commit on master. Chaotic-AUR
builds both.
apt.nspawn.org and rpm.nspawn.org keep the last five releases:
sudo apt install nspawn=VERSION-1 or sudo dnf downgrade nspawn-VERSION
installs an earlier one. Both are signed with this Ed25519 key:
1294612AB2DC4AE2F6118240FC5E133778B4B3B3
Every release also has the plain
binary, in nspawn-VERSION-x86_64-linux.tar.gz, with a SHA256SUMS. To build
it instead you need Rust 1.88 or newer:
git clone https://github.com/nspawn/nspawn.git
cd nspawn
cargo build --release
sudo install -Dm755 target/release/nspawn /usr/local/bin/nspawn
Then install the bus policy, the polkit actions and the units of the service:
sudo nspawn daemon --install
Keep the binary in /usr/local/bin or /usr/bin: the unit of every machine
calls it by that path, and SELinux refuses a system service a binary under
/home. With SELinux enforcing, the binary needs the label of the
nspawn-selinux policy for exec, shell, logs and cp:
sudo semanage fcontext -a -t nspawn_exec_t /usr/local/bin/nspawn
sudo restorecon -v /usr/local/bin/nspawn
Find an image. search asks the hub and Docker Hub and prints, for every hit,
the reference pull takes:
sudo nspawn search fedora
SOURCE NAME DESCRIPTION STARS OFFICIAL
hub.nspawn.org fedora tags: 43, 44, latest, rawhide - -
hub.nspawn.org fedora-devel tags: 44, latest - -
Docker Hub docker.io/library/fedora Official Docker builds of Fedora 1300 yes
...
Pull one. References without a registry part go to the hub, and the local name
is derived from the reference unless you pass --name:
sudo nspawn pull fedora:44
hub.nspawn.org/fedora:44: signature verified (key 6wiWMtJZCUkV, keyless https://github.com/nspawn/mkosi-definitions/.github/workflows/mkosi.yml@refs/heads/master)
hub.nspawn.org/fedora:44: manifest 9240778b2c77 with 1 layer(s), assembling as overlay
blob f0c1e88e32b3: downloading
blob 28edf9a59c17: downloading
blob f0c1e88e32b3: downloaded
blob 28edf9a59c17: downloaded
image fedora-44 (boot image) is ready: nspawn start fedora-44
The image’s signature was checked first (every image on the hub is signed by
the workflow that builds it; see Signed images),
then the blobs went to /var/lib/nspawn. A mount unit assembles the machine’s
root at /var/lib/machines/fedora-44 from the layers when it starts,
/etc/systemd/nspawn/fedora-44.nspawn holds the settings nspawn boots it with,
and a drop-in of systemd-nspawn@fedora-44.service makes the unit call nspawn
around its life.
sudo nspawn start fedora-44
sudo nspawn ps
MACHINE IMAGE MODE COMMAND STATE UP PID NETWORK OS
fedora-44 hub.nspawn.org/fedora:44 boot init running 12s 48213 10.99.0.2 Fedora Linux 44 (Forty Four)
start returns once the machine’s own systemd is up, so a command can follow
right away. exec runs it inside and brings back its exit code; shell opens
a login session as root:
sudo nspawn exec fedora-44 -- systemctl is-system-running
sudo nspawn shell fedora-44
The hub images log in as root with the password root on the console. What the
machine printed is in logs, and --inside reads the journal of the machine
itself:
sudo nspawn logs fedora-44
sudo nspawn logs fedora-44 --inside -n 50
Stop it and, when you no longer need it, remove it. Removing an image also frees the layers and blobs that no other image references:
sudo nspawn stop fedora-44
sudo nspawn images rm fedora-44
Images without an init system run as apps: the entrypoint from the image runs
under a stub init, on the bridge like any other machine, so -p publishes its
ports on the host:
sudo nspawn search nginx --source dockerhub
sudo nspawn pull docker.io/library/nginx:latest --name web
sudo nspawn start web -p 8080:80
curl -sI http://localhost:8080/ | head -1
sudo nspawn logs web -f
sudo nspawn stop web
sudo nspawn run -d docker.io/library/nginx:latest --name web -p 8080:80 does
the pull and the start in one step. Without -d, run stays attached:
sudo nspawn run -it --rm docker.io/library/alpine:3 sh gives you a shell in a machine that goes away
when you leave it. The port and any other flag given to start or run are
remembered, so the next sudo nspawn start web publishes it again. stop sends
the image’s stop signal (SIGQUIT for nginx) to the program and kills the machine
after ten seconds if it is still there; -t changes the grace period.
Docker Hub limits anonymous pulls per address. sudo nspawn login docker.io -u USER
keeps your credentials for that registry only; see
Registries and credentials.
create, where things are stored.start, run, stop, kill, exec,
shell, logs, ps, stats, events, entrypoints, environment and
volumes.nspawn compose.build, and
push.docker
command, CI runners, reverse proxies and dashboards, and auto-update./etc/nspawn/nspawn.toml, environment
variables and flags.An image reference has the form [registry/]repository[:tag|@digest]:
hub.nspawn.org; the configuration page
shows how to change it. The first path component counts as a registry when it
looks like a host, for example docker.io/library/nginx or
registry.example:5000/team/app, or when it is the configured registry’s own
name, so a registry of the local network without a dot or a port (myhub)
reads back as itself.latest.., _, -
and /. Official Docker Hub images live under library/, so nginx is
docker.io/library/nginx; search prints references in that form, and
Docker Hub’s short names are expanded: docker.io/alpine is
docker.io/library/alpine, and index.docker.io is docker.io.FROM too.run and create look for a local image of a name without a registry under
the configured registry first, then as Docker Hub’s (app as
docker.io/library/app, the name of an image compose pulled from Docker Hub),
and pull from the configured registry only when neither is there. Given by
reference, the source is the most recently pulled image of that reference that
never ran as a machine.
The local name of an image, the one start, stop, exec and friends
use, is derived from the reference (fedora:44 becomes fedora-44) unless
--name says otherwise. It has to be a valid machine name, which is a host
name: letters, digits, . and -, at most 64 characters, with no part between
dots empty or starting or ending with -.
The hub is a plain OCI registry. nspawn hub ls lists its repositories with
their tags, and nspawn hub tags REPOSITORY the tags of one of them:
sudo nspawn hub ls # everything
sudo nspawn hub ls deb # repositories whose name contains "deb"
sudo nspawn hub ls --no-tags # faster on big registries
sudo nspawn hub tags fedora
The same commands work against any registry that implements the catalog
endpoint; point them somewhere else with --registry, the NSPAWN_REGISTRY
variable or the configuration file. A private CA is trusted with --ca-cert
or NSPAWN_CA_CERT.
The images on the hub are built with mkosi from the public definitions in nspawn/mkosi-definitions, with the package manager of each distribution, and rebuilt every week so that they carry the current security updates. They come in three kinds:
| Kind | Images |
|---|---|
| Distributions | debian, ubuntu, fedora, centos, almalinux, rockylinux, opensuse, archlinux, kali, most of them in several releases |
| Services | nginx, apache, caddy, haproxy, postgresql, mariadb, redis, valkey, memcached, rabbitmq, mosquitto, unbound, dnsmasq, bind9, samba, openssh, prometheus, node-exporter, grafana, wordpress, syncthing, forgejo |
| Machines to work in | debian-devel, ubuntu-devel, fedora-devel, archlinux-devel, python, nodejs, golang, rust, openjdk |
A service image is a whole Debian machine with that service installed and
enabled, so shell gets you a root shell and systemctl status nginx inside
tells you what it is doing. Its tags carry the version of the service
(nginx:1.26.3, nginx:1.26, nginx:latest), and every build also gets a
dated tag (nginx:1.26.3-20260923) that never moves, so a machine can be
pinned to one build. https://hub.nspawn.org/ browses the lot.
sudo nspawn search TERM [--source hub|dockerhub] [-n LIMIT]
search looks in the sources nspawn knows: the configured
hub (its catalog, case-insensitive substring match, with the tags of every
match) and Docker Hub (its search API). Every hit names its source and the
reference pull takes:
SOURCE NAME DESCRIPTION STARS OFFICIAL
hub.nspawn.org debian tags: 12, 13, bookworm, latest, sid, trixie, unstable - -
Docker Hub docker.io/library/debian Debian is a Linux distribution that's composed entirely o... 5100 yes
Docker Hub docker.io/someone/debian-x Debian with extras 3 -
--source asks one side only and -n limits the results per source (25 by
default). A source that cannot be reached is reported as a warning, not as an
error, so the other one still answers.
Registries are used anonymously until you log in:
sudo nspawn login # the hub
sudo nspawn login docker.io -u USER # Docker Hub
echo "$TOKEN" | sudo nspawn login registry.example -u ci --password-stdin
sudo nspawn logout docker.io
login asks for the user name and, without echo, the password on the
terminal, or reads the password from standard input with --password-stdin
(which needs -u), and checks them against the registry: GET /v2/ on
it, then basic authentication or a token request at the realm the
registry announces. Bad credentials are refused. A registry that never asks for
credentials gets them stored anyway, with a note saying so.
Credentials are kept in /etc/nspawn/auth.json, mode 0600, in the format of
containers-auth.json(5). That file is the only one consulted: the work happens
in the service, which has no home directory of yours to look into, so
credentials other tools keep for your user do not carry over. Every
operation chooses the credentials of the registry it talks to, so the hub’s
never travel to Docker Hub; the many names of Docker Hub (docker.io,
index.docker.io, registry-1.docker.io) are one entry.
Docker Hub limits anonymous pulls per address; logging in raises the limit. push
authenticates before it uploads anything and, when the registry wants
credentials it does not have, says which login to run.
Every image on the hub is signed twice by the workflow that builds it
(nspawn/mkosi-definitions) with
cosign: with the project’s key, whose public half (cosign.pub in that
repository) is built into nspawn, and keyless, with the workflow’s own identity
through Sigstore
(https://github.com/nspawn/mkosi-definitions/.github/workflows/mkosi.yml@refs/heads/master,
issued by GitHub). The signatures are referrers of the image on the registry:
Sigstore bundles that carry the certificate or the key’s hint, the transparency
log entry and a timestamp, so pull verifies them offline, with nothing but the
registry consulted, against the Sigstore trusted root the binary embeds. One of
the two has to verify, and the check comes before a single layer is downloaded:
$ sudo nspawn pull fedora:44
hub.nspawn.org/fedora:44: signature verified (key 6wiWMtJZCUkV, keyless https://github.com/nspawn/mkosi-definitions/.github/workflows/mkosi.yml@refs/heads/master)
hub.nspawn.org/fedora:44: manifest 9240778b2c77 with 1 layer(s), assembling as overlay
...
An image without a signature (a tag dated before the signing began, or one
pushed by hand with nspawn push) is refused with carries no signature, and
so is one whose signatures do not verify, with the reason of each. --no-verify
on pull, run and create skips the check for that command, and on build
for the base images it pulls; create
checks only an image it has to pull, never a local source. inspect shows who signed (signed_by: key <hint>, keyless <identity>) and when (signed_at), which a machine made with create
inherits from its source.
Images of other registries are pulled unverified until the configuration file says what they must carry, one table per registry; the same table can require other keys of the hub, or turn its check off:
[registries."hub.nspawn.org"]
verify = false
The policies belong to the service: --registry on the command line chooses
the registry, never the policy.
sudo nspawn pull REFERENCE [--name NAME] [--backend BACKEND] [--mode MODE] [--force] [--no-verify]
pull resolves the reference to the manifest for the host’s platform and checks
the image’s signatures where its registry has a policy (the hub’s is built in;
see Signed images) before anything is downloaded. Of an image
index it takes the entry of the best platform the host runs: amd64 then 386,
arm64 from the CPU’s variant down to v8 and then 32-bit arm, arm from the CPU’s
variant down to v5. Without one, it tries the entries that name no platform, in
the index’s order, and takes the first whose config gives a platform the host
runs. It then downloads every layer and the config blob that
is not already in the store, three at a time, each checked
against its sha256 digest while it streams: a blob is written next to its final
name and renamed only once verified, so an interrupted download never passes for
a complete one, and a download cut short takes its part file with it. It says of
each blob when it is downloaded, and then:
..
stopping there, and each layer’s archive is read to its end and has to have
the digest the image’s config gives it (an image whose config names another
number of layers than its manifest is refused). A flat machine’s layers are
applied beside its directory and moved into place, so that a layer that
fails leaves the machine being replaced as it was;/etc/systemd/nspawn/NAME.nspawn, the settings the machine boots
with, and the drop-in that makes systemd-nspawn@NAME.service call nspawn
around its life;/var/lib/nspawn, and keeps the manifest and
blobs so that the image can be pushed or cloned later.An image with the same name is not replaced unless you pass --force, and
never while its machine is running, starting or being restarted by its policy.
The old image only goes once the new one is downloaded, and the store is locked
only for that last step, so a long download holds up neither other commands nor
the unit hooks.
The backend decides how the layers become a directory systemd-nspawn can boot:
| Backend | What it does | Needs |
|---|---|---|
overlay | Layers are extracted once under /var/lib/nspawn and shared. Each machine gets an overlayfs mount at /var/lib/machines/NAME, defined by a generated mount unit, with a private upper directory for its writes. | Linux 6.5 or newer with overlayfs (lowerdir+=). |
mstack (experimental) | A native systemd.mstack directory at /var/lib/machines/NAME.mstack that points at the shared layers; the machine boots with managed user namespaces (PrivateUsers=managed) and the layers are shifted into the foreign UID range. Only when asked for by name, with a note that it is experimental: systemd-nsresourced and systemd-mountfsd are still settling, and the behaviour depends on the host’s systemd version (the 262 release cannot boot a managed user namespace, systemd/systemd#43899). | systemd 261 or newer with the systemd-nsresourced and systemd-mountfsd sockets available (nspawn starts them). |
flat | The layers are extracted into a plain directory at /var/lib/machines/NAME. Nothing is shared, maximum compatibility. | Nothing. |
auto (default) | overlay when overlayfs is available, otherwise flat. Never mstack. |
Pick one per image with --backend on pull, run, build or create, or set a
default in the configuration file. nspawn images ls shows which backend each
image uses. On create, auto means the backend of the image it comes from,
not the chain above.
App images are always assembled as overlay (or flat), even when mstack is
asked for: a machine under managed user namespaces cannot join the network
namespace nspawn prepares for apps on the bridge. pull says so with a note
when it makes that choice.
After assembling, nspawn looks at the root file system and at the OCI config:
usr/lib/systemd/systemd,
lib/systemd/systemd, sbin/init or usr/sbin/init, as the topmost layer
leaves it) and whose command (the entrypoint, or the cmd without one) is empty,
init or systemd by name, one of those paths, or systemd wherever it is, is
a boot image. It is started with Boot=yes, as machinectl start would./init included: the
entrypoint and command from the config run under systemd-nspawn’s stub init
(Boot=no, ProcessTwo=yes), with the config’s environment, working
directory, user and stop signal. An app image that carries s6-overlay (v3
under /package/admin/s6-overlay, v2 with /etc/s6/init/init-stage1) runs
its program as PID 1 of the machine instead, as s6 insists on.--mode boot or --mode app overrides the detection. The mode is recorded with
the image, and an image keeps the mode it was installed with until it is
pulled again; Machines explains how it changes start, exec,
shell and stop.
sudo nspawn create SOURCE NAME [--backend BACKEND] [--network NETWORK]... [-p [IP:][HOST:]CONTAINER[/udp]]...
[--entrypoint PROGRAM] [-e VAR[=VALUE]]... [-v SOURCE:TARGET[:ro]]... [-l KEY=VALUE]...
[--restart POLICY] [-m SIZE] [--cpus N] [--pids-limit N] [OPTIONS OF start] [-f] [-- ARGUMENTS...]
create makes another machine from an image that is already
local, given by its name or by the reference it was pulled from, without
touching the registry. A reference that is not local is pulled first, under
the image’s own name, as run does. The machine shares the source’s layers
and gets a writable layer, an address, settings and a record of its own, with
create as its origin; images rm of one never affects the others.
The network kind is inherited from the source unless --network says
otherwise; published ports are not, since two machines cannot publish the same
one, and neither are the source’s labels, restart policy, limits, variables,
volumes, healthcheck, secrets or the other flags: a new machine only gets what
its own command line gives. Every option of start is accepted and means the
same; --entrypoint, -e and the arguments after -- apply to app images
only, -v works for both kinds. Everything on the command line is checked
before anything is made, so a refused create leaves nothing behind.
pull --name also makes another machine of an image, but resolves the manifest
through the registry first.
sudo nspawn images ls
NAME TYPE BACKEND ORIGIN SOURCE SIZE RO
fedora-44 directory overlay pull hub.nspawn.org/fedora:44 612 MiB no
db directory overlay create hub.nspawn.org/fedora:44 612 MiB no
web directory overlay pull docker.io/library/nginx:latest 190 MiB no
old-arch directory - - - 1.2 GiB no
The list comes from systemd-machined, so it includes images nspawn did not
create (old-arch above, with - in the nspawn columns). ORIGIN is pull,
build or create.
sudo nspawn images rm NAME...
images rm refuses to remove the image of a running machine, or of one its
restart policy is bringing back; nspawn rm -f stops it first. A pulled image
is a machine too, so nspawn rm NAME removes the same thing.
For an image nspawn manages it unmounts and deletes the assembled root, the
settings file, the generated units and drop-ins, the network namespace of an
app and the record; for any other image, or for the leftovers of a failed
install, it removes what it finds and asks machined to forget the rest.
Afterwards the layers and blobs that no remaining image references are
deleted, the /etc/hosts files of the machines on the bridge are regenerated
and the published ports are brought in line with the machines that still run.
Named volumes are not deleted: they may belong to another machine. nspawn volume ls shows them and volume rm or volume prune removes the unused ones.
images ls --json prints the list as the service returns it. nspawn images
without a subcommand prints the same list as images ls, --json included.
system df shows what images, machines, volumes and the build cache take on
disk (-v for each of them, --json):
$ sudo nspawn system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 4 2 368.8MB 295.4MB (80%)
Containers 2 2 1.093kB 0B (0%)
Local Volumes 0 0 0B 0B
Build Cache 182 4 1.141GB 1.11GB (97%)
Here an image is a record that was pulled or built and never started; every
other record is a machine. images prune -a removes the images no machine was
made from; without -a it removes the dangling ones, which nspawn’s store never
has, since a tag pulled or built again replaces its record. system prune
removes stopped machines made from an image (by create or run), networks no
machine uses and the build cache no image uses; with -a also the images no
machine was made from and the whole build cache, and with --volumes the volumes
no machine uses that were made without a name, which only the Docker API makes.
--filter label=... and
until=... narrow what goes, and -f skips the question. builder prune takes
the build cache alone.
| Path | Contents |
|---|---|
/var/lib/machines/NAME | The root of a machine with the overlay or flat backend (a mount point in the first case). |
/var/lib/machines/NAME.mstack | The systemd.mstack directory of a machine with the mstack backend. |
/var/lib/nspawn/ | nspawn’s own state: blobs as downloaded, extracted layers, image records, stored manifests, the private upper directories of overlay machines, the generated network files and units of each machine, mkosi build output and cache, and the store lock. The directory is 0711 and every directory under it 0700, except the per-machine files an mstack machine binds from inside its user namespace (0711). |
/var/lib/nspawn/volumes/NAME | A named volume (-v NAME:/inside). |
/etc/systemd/nspawn/NAME.nspawn | The settings nspawn generates for a machine; regenerated at every start, mode 0600 since it carries the -e variables. |
/etc/systemd/system/systemd-nspawn@NAME.service.d/ | The drop-in with the unit hooks (and the machine’s Restart=, MemoryMax=, MemorySwapMax=, CPUQuota= and TasksMax= when it has them) and, for an overlay machine, the one that requires its mount unit. The mount unit itself is next to them in /etc/systemd/system/. |
/etc/systemd/system/machines.target.wants/systemd-nspawn@NAME.service | The boot link of a machine with --restart always or unless-stopped; rm removes it. |
/etc/nspawn/nspawn.toml | The configuration file, optional. |
/usr/lib/systemd/system/nspawn.service, /usr/share/dbus-1/system.d/org.nspawn.conf, /usr/share/polkit-1/actions/org.nspawn.policy | The service, its bus policy and its polkit actions, as a package installs them. nspawn daemon --install writes the unit and the bus policy under /etc instead, where they take precedence, and the actions in the same place as the packages. |
/etc/nspawn/auth.json | The credentials login stored, mode 0600. |
/run/netns/nspawn-NAME | The network namespace of a running app machine on the bridge. |
The state is kept outside /var/lib/machines on purpose: machined would list
a directory there as an image, and machinectl clean would delete it.
Every one of these commands is a call to the
service on the system bus, which asks polkit
whether you may: sudo always works, and a rule can let your group through
without a password. The examples here use sudo.
Every machine nspawn starts is the systemd unit systemd-nspawn@NAME.service,
registered with systemd-machined under its name. machinectl list,
machinectl status NAME, systemctl status systemd-nspawn@NAME and
journalctl -u systemd-nspawn@NAME (with --namespace=nspawn for what the
machine prints, see logs) all work on it; nspawn adds its own
commands on top.
The unit also carries a drop-in, nspawn-hooks.conf, that calls nspawn around
the machine’s life: ExecStartPre prepares its network and settings,
ExecStartPost publishes its ports once it is registered, and ExecStopPost
releases everything however the machine ended; a machine with
dependencies first waits for them in another ExecStartPre.
So a start by machinectl start NAME or at boot (systemctl enable systemd-nspawn@NAME) prepares the machine as nspawn start does, and a program
that exits or crashes is cleaned up after as nspawn stop would. The drop-in
names the nspawn binary that wrote it, which is why nspawn belongs in
/usr/local/bin or /usr/bin. It also carries the machine’s restart policy and
resource limits, when it has them, and, for an app machine and for a booted one
whose output goes to nspawn’s journal (the default), the ExecStart= that runs
systemd-nspawn through nspawn attach-exec, which is how run -i and run -t
hand the program an input or a terminal and how the output reaches that
journal.
sudo nspawn start NAME [--network NETWORK]... [--network-alias [NETWORK=]NAME]...
[-p [IP:]HOST:CONTAINER[/udp]]... [--entrypoint PROGRAM] [-e VAR[=VALUE]]...
[-v SOURCE:TARGET[:ro]]... [-l KEY=VALUE]... [--restart POLICY] [-m SIZE]
[--cpus N] [--pids-limit N] [HEALTHCHECK OPTIONS] [OTHER OPTIONS]
[--secret NAME[:TARGET[:MODE[:UID:GID]]]]... [--depends-on NAME[:CONDITION][:optional]]...
[--env-file FILE]... [--label-file FILE]... [--mount type=TYPE,...]...
[--ip [NETWORK=]ADDRESS] [--gpus GPUS] [--image-command] [--no-wait] [-- ARGUMENTS...]
start regenerates /etc/systemd/nspawn/NAME.nspawn from the machine’s
record, sets up the network it is configured for, starts the
unit through systemd and waits up to 30 seconds for the machine to register
with machined. For a booted machine it then waits, up to 20 more seconds, until
the machine’s own systemd is listening, so that a command can follow right
away; --no-wait skips that last wait. A program that runs and returns at once
is reported as such, not as a failure.
Everything given to start is remembered for the machine, so a plain
nspawn start NAME next time reuses the network, the ports, the command, the
variables and the volumes of the last run:
--network switches the machine between the bridge, networks of its own
made with network create (several at once, --network-alias giving it
more names on them), a veth pair, the host’s network, another machine’s
(container:NAME) and none, and --ip
fixes its address on one made with network create; see
Networking.-p [IP:]HOST:CONTAINER[/udp] publishes a port on the host, on every
IPv4 and IPv6 address or on one. It needs a bridge network;
-p none forgets all published ports.--entrypoint, -e, --env-file and the arguments after -- change what
an app image runs; see Command, entrypoint and environment.
Boot images refuse them.-v mounts a host directory or a named volume into any kind of machine, and
--mount is its long form; see Volumes.-l/--label, --label-file, --restart, -m, --memory-swap, --cpus and
--pids-limit label the machine, give it a restart policy and bound its
resources; see
Labels and Restart policies and limits.--health-cmd and the other --health-* options give it a healthcheck, or
replace the image’s; see Healthchecks.--hostname, -u, -w, --cap-add, --cap-drop, --privileged,
--read-only, --tmpfs, --shm-size, --device, --device-cgroup-rule,
--dns, --dns-search, --add-host, --ulimit, --oom-score-adj, --stop-signal,
--stop-timeout, --init and --sysctl shape the machine further,
--log-driver says where the program’s output goes, and
--timezone how systemd-nspawn sets
/etc/localtime; see Other flags.--secret hands it a secret as a file; see Secrets.--gpus and --device VENDOR/CLASS=NAME hand it devices of the host’s CDI
specs; see GPUs and CDI devices.Machines are started by name. A reference (fedora:44, docker.io/x) is
refused with a hint to pull it or to create a machine from a local image.
Images that were not installed by nspawn, for example something created with
machinectl import-tar, can be started too: they get the stock template’s veth
networking, and nspawn starts systemd-networkd on the host when the host has no
.network files of its own (see veth); the flags above
are refused for them.
sudo nspawn run [-d] [--rm] [-i] [-t] [OPTIONS] REFERENCE [COMMAND [ARGUMENT...]]
run makes a machine from an image and starts it: pull,
or create from a local image with the same reference, then start, with the
options of both. A name without a registry is looked for here under the
configured registry, then as Docker Hub’s (app as docker.io/library/app,
the name of an image compose pulled from Docker Hub), and only then pulled from
the configured registry.
What follows the image replaces an app’s command; one that
starts with a dash comes after --. Without -d it stays with the machine:
--log-driver journal), so nspawn logs NAME shows it later, and the machine
goes on should run be interrupted. With --log-driver none it comes straight
from the program and nothing is kept.run exits with the program’s exit code, or, for a program under the stub
init, 128 plus the signal nspawn sent it: 130 after Ctrl-C, 137 after kill.
A signal from elsewhere (a crash, a kill from inside the machine, Ctrl-C with
-t) gives 255 there. A program that is PID 1 ends with 1 whatever signal
killed it but kill’s SIGKILL (137), which is how systemd-nspawn ends then.
Ctrl-C, SIGTERM, SIGHUP and SIGQUIT go to the program; a third Ctrl-C within a
second leaves the machine running and returns.-i gives the program this standard input, -t a terminal; closing that
terminal stops the machine, since nothing would read it any more.--rm removes the machine once it ends, and leaves nothing behind when the
start fails. Named volumes stay, and so does an image the run had to pull,
under the image’s local name; without --name the
machine gets a name of its own.sudo nspawn run --rm docker.io/library/busybox:latest sh -c 'echo hi; exit 3'; echo $?
echo abc | sudo nspawn run -i --rm docker.io/library/busybox:latest wc -c
sudo nspawn run -it --rm docker.io/library/alpine:3 sh
sudo nspawn run -d --name web -p 8080:80 docker.io/library/nginx:latest
A booted image shows its console until it powers off (Ctrl-C powers it off).
run -it on one waits for its boot, opens a root shell and powers the machine
off when the shell ends, with the shell’s exit code: sudo nspawn run -it --rm fedora:44 is a throwaway Fedora with its own systemd.
A boot image is started with Boot=yes: systemd-nspawn runs the image’s init
as PID 1, like machinectl start does, and the machine joins the bridge with
an address that its own systemd-networkd configures.
exec enters the machine’s namespaces directly, so
nothing is needed inside: no D-Bus, no PAM. The exit code comes back and the
program is found on the machine’s PATH.shell opens machined’s login session (OpenMachineShell), which needs
D-Bus inside the machine; the hub images have it. Right after start the
machine’s D-Bus may not be up yet, so shell retries for up to 20 seconds
instead of failing.stop asks machined to power the machine off and repeats the request every
two seconds until the machine is gone, for up to a minute; the repetition
covers the window right after boot in which the init has not installed its
signal handlers yet. --force kills every process at once.nspawn-volumes.service,
holds local-fs.target until all of them are mounted, so services find their
data in place whatever the backend.An app image is started with Boot=no and ProcessTwo=yes: systemd-nspawn’s
stub init is PID 1 and the image’s entrypoint runs under it, as its child, with
the environment, working directory, user and stop signal from the OCI config.
Its network namespace is built on the host before the program starts, so the
network is there when it starts.
systemd-nspawn hands its payload NOTIFY_SOCKET=/run/host/notify, a socket only
root may write to with recent systemd: a program that notifies systemd as another
user logs that it could not, and goes on. -e NOTIFY_SOCKET= (or
NOTIFY_SOCKET: "" in a compose file’s environment) quiets one that takes an
empty value for none, as Go’s and Erlang’s clients do; libsystemd’s and Rust’s
refuse it.
A user namespace cannot join a network namespace that belongs to the host, so an
app on a bridge network runs without one (PrivateUsers=no), and so does an app
given devices of the host on any network, since a device node cannot be idmapped.
Capabilities,
seccomp and the other namespaces still apply. Such an app gets the capabilities
images need to run as written (CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL,
MKNOD, NET_BIND_SERVICE, NET_RAW, SETFCAP, SETGID, SETPCAP, SETUID, SYS_CHROOT,
AUDIT_WRITE) plus SYS_BOOT, with the kexec system calls filtered out. It does
not get systemd-nspawn’s set: its
CAP_SYS_ADMIN outside a user namespace is root on the host. --cap-add puts
one back, --privileged all of them.
An image that carries s6-overlay (linuxserver.io’s images, Home Assistant’s and
others) runs with its /init as PID 1 of the machine, as s6 requires:
ProcessTwo= is left out and the stop signal becomes
systemd-nspawn’s KillSignal=.
--init=false asks the same of any app, and --init puts the program back
under the stub init. A program that is PID 1 ignores the signals it has no
handler for, so stop kills it after its timeout when it does
not handle its stop signal.
exec and shell enter the namespaces of the machine’s leader process, on a
terminal when standard input and output are one, with the image’s
environment. exec runs as the machine’s
user, shell runs /bin/sh as root, and -u USER switches user for both.stop sends the image’s stop signal (StopSignal in the config, SIGTERM
by default, --stop-signal over it) to the program itself, waits --timeout
seconds (the machine’s --stop-timeout, else 10) and kills the machine if it
is still there. A stop through systemd (systemctl stop,
shutdown) reaches the stub init as a poweroff request, which it answers with
SIGTERM and SIGHUP to the program.nspawn stop on it afterwards only clears
what it left behind and says NAME was not running.What an app runs comes from its image’s entrypoint and cmd, unless the flags say otherwise:
-- replace the image’s cmd and follow its
entrypoint. nspawn start web -- nginx -T still runs
/docker-entrypoint.sh first.sh -c does, from the command
line, compose or the image’s own config: the program is then started through
the image’s /bin/sh, which gets the arguments encoded and runs the program
with them as they were given. A variable’s value may hold line breaks the
same way (-e, compose’s environment): it goes in encoded and the image’s
/bin/sh turns it back before the program runs; its name is then one of
letters, digits and underscores. An image without /bin/sh cannot take such
an argument or variable.--entrypoint PROGRAM replaces the entrypoint with one program and drops the
image’s cmd, so only the arguments after -- follow it (--entrypoint /bin/sh alone is a shell). --entrypoint "" drops the entrypoint, so the
arguments run alone. An app left with nothing to run is refused with
no command specified.--image-command forgets them and runs the image’s own entrypoint and cmd
again.-e VAR=value adds a variable on top of the image’s, and -e VAR copies it
from the shell that runs nspawn, or is left out when the shell does not have
it, so -e DEBUG passes an optional setting on only where it is set; a
start whose variables are all left out keeps the ones the machine
remembers. --env-file FILE reads KEY=VALUE lines
(no quoting, no expansion, # comments), before the -e
that follow. The later value of a variable wins, -e none forgets them all,
and exec sees the same environment as the program. Any name but an empty
one is taken (discovery.type, as Elasticsearch reads it).-w, -u and --stop-signal say otherwise. The user and group are
resolved from the image’s passwd and group files.
With USER:GROUP, from the image’s config or from -u, the group (a name or a
number) is the program’s primary and only group. systemd-nspawn asks getent
inside the machine, which busybox lacks and musl cannot answer, so nspawn binds
a stand-in that answers these lookups and passes others to the image’s own
getent. An image without a shell (distroless, scratch) gets a small program
instead, so -u 1000:1000 works there too. A uid missing from passwd runs
with group 0 and / as home.ps shows the effective command; nspawn run --rm IMAGE true is a quick way to
check that an image runs at all.
sudo nspawn start web -v /srv/www:/usr/share/nginx/html:ro -v pgdata:/var/lib/postgresql
-v SOURCE:TARGET[:MODE] mounts a host directory or a named volume:
SOURCE is a host directory (or file) that has to exist already:
start refuses a path that is not there rather than making
one. A SOURCE starting with . is a directory relative to the one the
command runs in, and a host path is taken cleaned. A SOURCE without a
leading / is a named volume that nspawn
keeps under /var/lib/nspawn/volumes/NAME, created on first use with what
the image has at TARGET, owner and files included
(a program that runs as a user finds its data directory its own), and
never deleted by rm or images rm. Names may contain letters, digits,
_, . and -, and may not start with a dot.TARGET is an absolute path inside the machine, other than /. The same
target cannot be mounted twice.:ro mounts it read-only; :rw is the default. The other words a mode may
hold are taken and do nothing: z and Z (the policy labels what a machine
sees), a propagation (systemd-nspawn binds with its own), nocopy, and
cached, delegated or consistent. Paths with whitespace are not
supported, and the same -v given twice is one mount.-v is repeatable and remembered; -v none forgets every volume.--mount type=bind|volume|tmpfs,source=...,target=... is the long form of
-v and --tmpfs (readonly, tmpfs-size, tmpfs-mode); a volume needs
its name, as a mount makes no anonymous volume, and a source or target with
: is refused.In machines that run with private users (every machine but an app on a bridge
network, in another machine’s network or given devices of the host), the mount is
idmapped, so a file root writes inside belongs to root on the host, and the other
way round. A source on a file system that takes no idmapped mount (overlayfs,
devtmpfs, proc, sysfs, NFS, CIFS, FUSE, tmpfs before Linux 6.3) goes in as it is,
its owners unmapped inside. On mstack
machines nspawn attaches
the volumes from the host right after the machine’s init starts, since
systemd-nspawn cannot idmap binds under managed user namespaces; that is what
nspawn-volumes.service waits for, for up to two minutes, failing visibly
otherwise. On overlay and flat the volumes come from the settings file and are
there before the init even runs.
Named volumes outlive the machines that use them. nspawn volume ls lists them
with the machines whose records mount them (--filter label=KEY[=VALUE] only
those with the label), volume create NAME makes one ahead of its first use
(filled from the image of the first machine that takes it while it is empty;
--label KEY=VALUE labels it), volume inspect shows what nspawn keeps of them
as JSON, and volume rm and volume prune remove the ones no machine uses; a volume a machine still names is refused until that machine is
started with other volumes (or -v none) or removed.
sudo nspawn start web --label caddy=web.example --label tier=front
The image’s own labels (LABEL in a Dockerfile,
OciLabels= in a mkosi.conf) are read from its configuration when it is
pulled or built, and -l/--label KEY=VALUE (a KEY alone for an empty value) on start, run or create adds the
machine’s own on top, remembered like the ports; --label-file FILE reads
them from a file of KEY=VALUE lines, and --label none forgets them.
nspawn does nothing with them itself: nspawn inspect and ps --json show the
merged labels and the image’s image_labels, for tools that configure
themselves from them, a reverse proxy say.
sudo nspawn start web --restart unless-stopped -m 512m --cpus 1 --pids-limit 500
--restart takes four policies:
no, the default: a machine that ends stays down.on-failure: restarted when its program, or its init, dies with an error;
on-failure:N is accepted, but the count is ignored: systemd restarts on every
failure.always: restarted whenever it ends, and started at boot.unless-stopped: like always, until nspawn stop, which also takes it
off the boot list until the next nspawn start.A machine that ends is started again after a second, then later and later, up
to half a minute, for as long as it keeps failing; ps shows it as
restarting meanwhile, and nspawn stop ends that. The ports, the address and
the volumes come back with it, since the unit’s own hooks prepare every run.
always and unless-stopped enable the unit the way machinectl enable does,
and removing the machine takes that back. machinectl stop stops a machine for
good too, but does not take an unless-stopped one off the boot list. A
poweroff inside the machine is an ending like any other, which always and
unless-stopped restart.
-m/--memory (512m, 2g), --cpus (0.5, 2) and --pids-limit bound
the whole machine: they are the MemoryMax=, CPUQuota= and TasksMax= of its
unit, which is why nothing inside shows them. --memory also
lets the machine use as much swap again (MemorySwapMax=), and no more, unless
--memory-swap gives memory and swap together: equal to
--memory for no swap, larger for the difference, -1 for swap without a
bound. 0 removes a limit, and brings --memory-swap back to its default;
--pids-limit -1 removes it too.
The restart policy and the limits are remembered like the ports and apply at
the next start, and
nspawn update changes them without one: a running
machine gets the new limits in its cgroup at once.
sudo nspawn update web -m 1g --cpus 2 --restart always
sudo nspawn start web --health-cmd "curl -fsS http://localhost/ || exit 1" --health-interval 10s --health-retries 3
An image’s HEALTHCHECK applies as it is, and
--health-cmd with --health-interval, --health-timeout,
--health-retries, --health-start-period and --health-start-interval on
start, run, create and update each replace their part of it
(--health-interval 10s alone keeps the image’s test, and 0 for a timing or
the retries, like an empty --health-cmd, is no value given); --no-healthcheck
turns it off; timings given without a command on an image that has none are kept
and probe nothing. Durations are written 10s, 1m30s or 500ms, and the
defaults are 30 seconds between probes, 30 seconds for a probe, 3 failures in a
row before the machine is unhealthy, no start period, and 5 seconds between
probes during one.
The probe runs inside the machine as its user: a --health-cmd line through
the image’s shell (/bin/sh -c unless its SHELL says otherwise), an image’s
CMD form as it is (exit 0 is healthy). It runs from a unit of its own,
nspawn-health-NAME.service, which requires the machine’s and goes with it. The machine is starting until a probe succeeds or
the start period ends, healthy after a success and unhealthy after the
retries fail in a row; a success brings it back.
ps shows the health next to the state: running (healthy),
running (unhealthy), running (health: starting).
inspect shows health (the status), health_failing_streak and
health_log, the last five probes as END EXIT_CODE OUTPUT, and
healthcheck as it applies (durations in microseconds, 0 for the
default):
"health": "healthy",
"healthcheck": {
"interval": 2000000,
"retries": 2,
"start_interval": 0,
"start_period": 0,
"test": [
"CMD-SHELL",
"test -f /ok"
],
"timeout": 0
},
events reports health_status with the new status on every change.
update changes the healthcheck of a running machine at once, and the
probes start afresh.
A stopped machine has no health, and neither has one without a healthcheck.
sudo nspawn network create shop
printf 'example' | sudo nspawn secret create pgpass
sudo nspawn run -d docker.io/library/postgres:17 --name db --network shop -v pgdata:/var/lib/postgresql/data \
--secret pgpass -e POSTGRES_PASSWORD_FILE=/run/secrets/pgpass --health-cmd "pg_isready -U postgres" --health-interval 2s
sudo nspawn run -d docker.io/library/nginx:1.27 --name web --network shop -p 8080:80 --depends-on db:healthy
--depends-on NAME[:CONDITION][:optional] on run, create and start is
compose’s depends_on for one machine: the other machine is started first,
and this one waits for what the condition asks of it.
| Condition | Waits until |
|---|---|
started (the default) | the other machine’s start went through. |
healthy | its healthcheck says healthy; it needs one, and an unhealthy verdict fails the start. |
completed | its program ended with 0, for a machine that prepares something and exits, such as a migration. It runs until one run of it succeeds: it runs first only when no run of it ended with 0 yet (a run of an earlier boot counts by the exit code the journal kept), not at every start or restart of the machines that depend on it. |
With :optional a failure of the other machine is a warning and this one
starts anyway. The flag is repeatable and remembered like the rest,
--depends-on none forgets them, and inspect lists them as depends_on.
The machines named must be nspawn’s, a machine cannot depend on itself, and a
cycle is refused, --network container: counting as a dependency.
systemd does the work: the machine’s unit wants the other one’s and is ordered
after it (one that completes is only ordered after it, and the hook starts it
when it has to run), and its first hook, nspawn await-dependencies, waits for
the condition. So machinectl start, a start at boot and a restart policy follow
the same order as nspawn start, which says which machines it started or ran
first:
$ sudo nspawn stop web db
stopped web
stopped db
$ sudo nspawn start web
started web
db started first: web depends on it
When a dependency fails, the start fails with its reason
(web did not start: ...). A start that waits for healthy or completed,
directly or through the machines its dependencies depend on, has no time
limit, since it lasts as long as the other machine takes; nspawn stop ends
it. A machine on another one’s network (--network container:NAME) that also
depends on it has that machine started first. Stopping a machine leaves the ones
that depend on it running, and rm refuses a machine others
depend on unless they are removed in the same command.
sudo nspawn start web --hostname web-1 -u nobody -w /tmp --cap-drop ALL --cap-add NET_BIND_SERVICE \
--read-only --tmpfs /run:size=64m --tmpfs /tmp --shm-size 64m --device /dev/null:/dev/nullo \
--dns 10.99.0.1 --dns-search example.test --add-host db:10.99.1.3 --add-host gw:host-gateway \
--ulimit nofile=1024:4096 --oom-score-adj 500 --stop-signal SIGQUIT --stop-timeout 30 \
--sysctl net.ipv4.icmp_echo_ignore_all=1
They are remembered like the rest and shown by
inspect. Each becomes a line of the machine’s .nspawn settings file or of
its unit:
--hostname NAME: the hostname inside, the machine’s name by default. A
booted machine gets it as its /etc/hostname, over the image’s, since its
systemd sets the hostname from that file.
-u USER[:GROUP] and -w DIR: the user (a name or a uid, listed in the
image’s passwd or not, with a group of the image’s group file or a gid
after a colon; a group name the image lacks
is refused) and the working directory the program runs with, instead of the
image’s. App images only; exec takes the same user unless its own -u
says otherwise.
--cap-add, --cap-drop and --privileged: capabilities added to or
dropped from the default set (NET_ADMIN or CAP_NET_ADMIN; ALL). For an
app on a bridge network, which runs without a user namespace, the default set
is the one App machines lists, SYS_BOOT included, so that a
reboot from inside ends the machine; for
a machine in a user namespace it is systemd-nspawn’s. An interface given with
--interface keeps NET_ADMIN on its own; a name that is no capability is
refused. --cap-drop ALL --cap-add NET_BIND_SERVICE keeps NET_BIND_SERVICE alone, and
--privileged keeps every one.
--read-only: the root read-only. --tmpfs PATH[:OPTIONS]: an empty tmpfs
at a path (size=64m, mode=0700; 1777 when given without options),
which is how a read-only machine still writes /tmp or
/var/cache. The options are tmpfs’s own (size, mode, uid, gid,
nr_inodes, nr_blocks, mpol, huge, inode32, inode64, noswap):
systemd-nspawn mounts it read-write, nodev and strictatime, so ro,
noexec, nosuid and the other atime flags are refused rather than
dropped. /run is a tmpfs of every machine already, so a --tmpfs on or
below it (/var/run in most images) is
left out with a note. --shm-size: the size of /dev/shm.
--device HOST[:CONTAINER][:PERMISSIONS]: a device node of the host, bound
into the machine and allowed to its cgroup (r, w, m, each once; rwm by
default; /dev/sda:r is the node at the same place), or a directory such as /dev/dri, whose nodes are allowed one by
one. An app given devices of the host runs without a user namespace, so root
inside opens a node as its owner and mode allow; a booted machine keeps its
user namespace, where a node keeps the host’s ownership. A CDI name such as
nvidia.com/gpu=all takes the devices of a spec instead; see
GPUs and CDI devices.
--device-cgroup-rule 'TYPE MAJOR:MINOR PERMISSIONS' lets the machine’s cgroup
open nodes by their numbers, the nodes that appear while it runs included:
sudo nspawn run -d --name flasher -v /dev/bus/usb:/dev/bus/usb \
--device-cgroup-rule 'c 189:* rwm' --network none IMAGE
reaches a USB device that comes and goes, as a flasher’s does. A rule for
every minor of a major goes by the device groups the host’s /proc/devices
names after it at each start (char-usb_device for 189), and is refused then
for a major without one; one node goes by its numbers (c 189:3 rw).
--dns and --dns-search: the DNS servers and search domains the machine
gets instead of the host’s resolvers, in an app’s generated resolv.conf and
in a booted machine’s .network file. --add-host HOST:IP (or HOST=IP, for
an IPv6 address): lines for its /etc/hosts, host-gateway standing for the
host’s address on the machine’s network.
--ulimit NAME=SOFT[:HARD]: resource limits of the program (nofile,
nproc, core, …; unlimited is a value). --oom-score-adj: the
machine’s, -1000 to 1000.
--stop-signal and --stop-timeout: what stop and restart use for the
machine instead of the image’s stop signal and the default 10 seconds; -t on
stop still wins, and -1 waits for as long as the program takes. kill
reads the stop signal to decide whether the restart policy applies.
--timezone MODE: how systemd-nspawn sets the machine’s /etc/localtime at
each start. Its default, auto, points it at the host’s zone every time: a
link where the machine’s zone database has that zone, otherwise a copy of the
host’s file, which a package installed inside, such as tzdata, may replace (a
read-only root gets the host’s file mounted over it). A zone set inside with
timedatectl does not survive a restart; off leaves the machine’s own, and
copy, bind, symlink and delete are its other modes. An app also takes
-e TZ=.
--log-driver DRIVER: where the program’s output goes. local, the
default, sends it to journald’s nspawn namespace, a journal of nspawn’s
own apart from the system’s, so that a program that writes a lot never
floods the host’s logs; logs and an attached run read it, and
journalctl --namespace=nspawn shows every machine’s. It holds 1 GiB at
most, shared by all machines, the oldest lines going first
(/usr/lib/systemd/journald@nspawn.conf, which
/etc/systemd/journald@nspawn.conf overrides). journal keeps the output
in the system’s journal, and none drops it: logs then refuses the
machine, an attached run of an app still shows the output straight from
the program, and an attached run of a booted machine, whose console goes
nowhere, is refused: it runs with -d.
--init and --init=false: the program under systemd-nspawn’s stub init, which
reaps orphans and turns a poweroff into SIGTERM, or as PID 1; see
App machines. App images only.
--sysctl KEY=VALUE: net.* keys, set in the network namespace nspawn
makes for an app machine on a bridge network, after the two it gets by
default, net.ipv4.ping_group_range=0 2147483647 and
net.ipv4.ip_unprivileged_port_start=0, which let a user other than root
ping without raw sockets and bind a port below 1024. Nothing else is accepted,
and a booted machine sets its
own.
--interface IFACE: a network interface of the host, moved into the
machine while it runs and back on the host when it stops, a wifi adapter
with its whole phy included. See Physical
interfaces.
A list takes none to forget it (--cap-drop none, --tmpfs none);
--hostname, -u, -w, --stop-signal, --timezone and --log-driver go
back to the default with an empty string (--hostname ""), --shm-size with
0, and --privileged=false and --read-only=false take those back. A path inside the
machine, the target of a volume or a secret, a --tmpfs or a --device path,
must be plain: a . or .. component is refused, since it would land elsewhere
once mounted. A path the image
declares as a volume with nothing mounted over it gets a note at start: no
anonymous volume is made for it, so what is written there goes with the
machine.
GPUs reach a machine through the Container Device Interface. The vendor’s tool
writes a spec under /etc/cdi or /var/run/cdi
that lists the device nodes, libraries, variables and hooks each device needs
(nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml for NVIDIA,
amd-ctk cdi generate for AMD), and the machine names the devices:
sudo nspawn run -d --name app --gpus all IMAGE
sudo nspawn start app --device nvidia.com/gpu=0
--device VENDOR/CLASS=NAME takes a device by the spec’s name (=all, =0, a
UUID), and --gpus all, --gpus 2 or --gpus '"device=0,1"' stands
for the devices of the NVIDIA spec (the AMD one when only that is there).
Compose’s gpus:, the devices of deploy.resources.reservations and CDI names
in devices: do the same. The specs are read again at every start, so a driver
update that rewrites them takes effect at the next one: the device nodes are
bound into the machine and allowed in its cgroup, the libraries bound where the
spec says, the variables set after the machine’s own, and the hooks
(nvidia-cdi-hook update-ldcache, create-symlinks) run just before the
machine starts, on its root.
Without a spec of the vendor, the start fails:
$ sudo nspawn run --rm --gpus all docker.io/library/busybox:1.37 -- true
error: could not select device driver "" with capabilities: [[gpu]]
The nodes are the host’s, with the host’s owner and mode. mstack machines take no CDI devices. It is tested with specs written for the test suite, not with real GPUs.
printf '%s' 'hunter2' | sudo nspawn secret create db-password
sudo nspawn start db --secret db-password -e POSTGRES_PASSWORD_FILE=/run/secrets/db-password
secret create NAME reads the content
from standard input, or from -f/--file, and keeps it encrypted with
systemd-creds, bound to the host’s TPM2 where there is one and to its
credential key otherwise, under /var/lib/nspawn/secrets; -l KEY=VALUE
labels it. --secret NAME on start, run or create gives the machine the
plaintext at /run/secrets/NAME, a read-only file of root’s with mode 0444;
NAME:TARGET:MODE:UID:GID chooses the path, mode and owner. The file is
decrypted when the machine starts, into a tmpfs of
root’s alone, and gone when it stops.
secret ls lists the secrets with the machines that take them:
SECRET SIZE CREATED USED BY
db-password 7 B 9s ago db
secret inspect prints what is known about them, never the content, and secret rm
removes the ones no machine takes; one a machine’s record names is refused
until that machine is started with other secrets (or --secret none) or
removed. mstack machines take no secrets.
sudo nspawn ps [-a] [--json] # same as: nspawn machines ls [-a] [--json]
MACHINE IMAGE MODE COMMAND STATE UP PID NETWORK OS
api docker.io/library/busybox:latest app /bin/sh -c while true; do sleep 1; done running 6s 895101 back:10.99.2.4 -
db docker.io/library/busybox:latest app /bin/sh -c while true; do sleep 1; done running 8s 894865 back:10.99.2.2 -
web docker.io/library/busybox:latest app /bin/sh -c touch /ok; while true; do ... running (healthy) 7s 894994 front:10.99.1.2 back:10.99.2.3 127.0.0.1:8080->80/tcp -
ps lists every container machined knows about (the virtual machines it also
registers, libvirt’s among them, are left out); machines that nspawn did not
install show - in the image columns. -a adds the nspawn machines that are
not running. COMMAND is the effective entrypoint and arguments of an app,
and NETWORK the machine’s address on the default network, its other networks
as NAME:ADDRESS, and the published ports, or host, veth, none or
container:NAME. A
machine between two runs of its restart policy is listed as restarting even
without -a, a frozen one as paused, and a healthcheck follows the state:
running (healthy).
ps --json prints the same machines as the service returns them, each with its
whole record, and nspawn inspect NAME... prints one or more machines, running
or not, as a JSON array, for scripts.
ps also takes flags for scripts. -q prints the names alone,
--filter KEY=VALUE keeps the machines that match (name
as a regular expression, label, status, network, ancestor, health,
exited, volume, publish, expose, id, before, since; the same key
twice matches either value, different keys all), and --format takes a Go
template over each machine’s fields (.Names, .Image, .Command,
.CreatedAt, .RunningFor, .Ports, .State, .Status, .Labels,
.Label "KEY", .Mounts, .Networks, .Mode, .Address, .OS, .Pid),
table TEMPLATE for aligned columns with headers, or json for an object per
line. inspect -f TEMPLATE prints a template over each object, a line each; a
key the object lacks prints <no value>, an object the template fails on is said
after the others are printed, and -f json prints the objects on one line:
$ sudo nspawn ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
NAMES IMAGE STATUS
web nginx:alpine Up 1 second
worker busybox:1.37 Up Less than a second
$ sudo nspawn ps -q --filter label=tier=front
web
$ sudo nspawn inspect -f '{{.state}} {{index .labels "tier"}}' web worker
running front
running back
sudo nspawn exec MACHINE [-u USER] [-e VAR[=VALUE]]... [-w DIR] [-T] [-t] [-i] [-d] COMMAND...
sudo nspawn shell MACHINE [-u USER]
exec runs one command inside a running machine of either kind and exits with
the command’s status, so it works in scripts and pipelines. The program is looked
up on the machine’s PATH, the image’s environment and the -e variables of the
machine apply, and the working directory is the machine’s. Neither D-Bus nor
anything else is needed inside:
-u the command runs as the machine’s user, its --user or else its
image’s USER, and as root on a booted machine.
-u USER[:GROUP] takes names or numbers of the image’s passwd and group
files; with a group, that one is the only group.-e adds variables for this command, -w a working directory.-T refuses a terminal even from one and -t
asks for one even without. -i is accepted and ignored.-d leaves the command running in the background and returns at once.The command runs with the machine’s capabilities and resource limits, like the
machine’s own processes, with one exception: in an app that shares the host’s
user namespace and is not --privileged, it goes without CAP_SYS_BOOT, which
the machine keeps only beside a seccomp filter against kexec that the command
does not carry. Right after a start, exec waits up to five seconds until
systemd-nspawn has finished confining the machine.
$ sudo nspawn run -d --name worker -u 65534 docker.io/library/busybox:1.37 -- sleep 600
started worker
$ sudo nspawn exec worker -- id
uid=65534(nobody) gid=65534(nobody)
$ sudo nspawn exec -u root worker -- id
uid=0(root) gid=0(root)
exec and a healthcheck probe into a paused machine fail at once. shell
opens an interactive shell as root (or -u USER): machined’s login session
for booted machines, /bin/sh in the machine’s namespaces for apps.
sudo nspawn cp ./nginx.conf web:/etc/nginx/
sudo nspawn cp web:/var/log/nginx ./nginx-logs
sudo nspawn cp ./site/. web:/usr/share/nginx/html
cp copies files and directories between the host and a machine, running or
not:
/
has to be a directory.DIR/. copies the contents of DIR rather than DIR.cp. Modes and modification
times are kept./ or /. names the directory a link leads to); devices, sockets
and fifos are left out, and so are the kernel’s file systems mounted inside
a running machine (/proc, /sys and the like).A stopped overlay or flat machine can be copied into and out of; a stopped
mstack machine cannot, since its tree only exists while it runs. A stopped
machine’s volumes are mounted only while it runs, so a path at or below one of
them, or a source that holds one, is refused until it starts. Where
SELinux enforces, a host directory mounted with -v keeps its own label, which
the service may not be allowed to write to;
named volumes are nspawn’s own and always work.
sudo nspawn logs MACHINE... [-f] [-n N] [--since WHEN] [--until WHEN] [-t] [--all] [--inside]
What the machine writes to its console goes, by default, to journald’s
nspawn namespace under systemd-nspawn@MACHINE.service, apart from the
system’s journal (see --log-driver above), and logs reads it with
journalctl, together with the system’s journal, which holds systemd’s lines
and the output of a machine with --log-driver journal. By default only the
machine’s own output is shown, from its creation on, every run of it included;
a removed machine of the same name does not show:
-f keeps printing new output, starting from the last 10 lines unless -n
says otherwise. -n N (or --tail N) shows the last N lines, all every
one. --since and --until take a duration ago (42m, 1h30m), Unix
seconds, an RFC 3339 time or date, or a time as journalctl reads it (-1h,
"10 min ago", "2026-09-24 10:00"), and nothing is followed past
--until; -t prefixes each line with its timestamp.web | ...).--all also shows what systemd logged about the unit: start, stop,
failures.--inside reads the journal of a booted machine itself
(journalctl --machine), which is where the services running inside log.logs works for stopped machines too, since the journal keeps what they wrote.
sudo nspawn stats [NAME...] [--no-stream] [--json]
stats shows what each running machine uses, drawn again
every second: CPU (100% is one CPU busy), memory in use against the limit (the
host’s memory without one), network and disk traffic, and processes. The
numbers are those of the cgroup of the machine’s unit, which holds the whole
machine, and of its interfaces as the machine sees them. --no-stream prints
one table, --json one object per machine and reading.
NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
api 0.00% 2.4 MiB / 3.8 GiB 0.06% 372 B / 0 B 0 B / 4.0 KiB 3
db 0.00% 2.4 MiB / 3.8 GiB 0.06% 892 B / 0 B 0 B / 4.0 KiB 3
web 0.00% 2.4 MiB / 3.8 GiB 0.06% 522 B / 0 B 0 B / 4.0 KiB 3
sudo nspawn events [--since WHEN] [--until WHEN] [-f KEY=VALUE]... [--json]
events reports what happens: machines that start, die
(with their exit code), stop, are restarted, run out of memory, fail or change
health (health_status), and what nspawn does: pulls, builds, creations,
pushes, kills, updates, pauses, auto-updates and removals, of networks, volumes
and secrets too. It
reads the journal, where systemd logs every start and end of a machine’s unit
however it was started, and nspawn logs what it does, so --since reads past
events back; --since and --until take the forms logs takes.
Filters take name=, type= (machine, network, volume,
secret), event= and label=KEY or label=KEY=VALUE: the same key given
twice matches either value, different keys must all match. --json prints one
object per event. What events printed while a network backend was made, db
ran on it with run -d -l tier=data, run --rm --name job ... -- sh -c 'exit 3' ran and ended, and kill -s HUP db ended db:
2026-10-21T06:12:16.543884Z network create backend (created=1792563136, interface=nsbr-backend, internal=false, subnet=10.99.1.0/24)
2026-10-21T06:12:18.490159Z machine pull db (created=1792563138, image=docker.io/library/busybox:1.37, reference=docker.io/library/busybox:1.37, tier=data)
2026-10-21T06:12:18.843934Z machine start db (created=1792563138, image=docker.io/library/busybox:1.37, tier=data)
2026-10-21T06:12:20.792735Z machine pull busybox-1.37 (created=1792563140, image=docker.io/library/busybox:1.37, reference=docker.io/library/busybox:1.37)
2026-10-21T06:12:21.920339Z machine create job (created=1792563141, from=busybox-1.37, image=docker.io/library/busybox:1.37, reference=docker.io/library/busybox:1.37)
2026-10-21T06:12:22.569822Z machine start job (created=1792563141, image=docker.io/library/busybox:1.37)
2026-10-21T06:12:22.605119Z machine die job (code=exited, created=1792563141, exit_code=3, image=docker.io/library/busybox:1.37)
2026-10-21T06:12:22.641937Z machine fail job (created=1792563141, image=docker.io/library/busybox:1.37, result=exit-code)
2026-10-21T06:12:23.051001Z machine remove job (created=1792563141, image=docker.io/library/busybox:1.37)
2026-10-21T06:12:23.142907Z machine kill db (created=1792563138, image=docker.io/library/busybox:1.37, signal=1, tier=data)
2026-10-21T06:12:23.153501Z machine die db (code=exited, created=1792563138, exit_code=129, image=docker.io/library/busybox:1.37, tier=data)
Times are in UTC. A machine’s die carries the exit code of that run:
128 plus the signal nspawn sent it (143 after a stop), or the status systemd
logged once the machine’s record is gone (255 for an app killed by a signal).
nspawn’s own events of a type carry the same attributes whatever the action, so
a remove says as much as a create: image= and created= (which machine of
the name it was) on a machine’s, subnet=, interface=, internal= and
created= on a network’s, path= on a volume’s, size= on a secret’s.
systemd’s take image=, created= and the labels from the machine’s record,
and lack them once it is removed.
sudo nspawn stop NAME... [-f] [-t SECONDS] [--no-wait]
Stops the machines as described above for boot and app machines, one after
the other, waits until each is gone (a minute at most for a booted machine),
stops the unit so that the image can be removed right away, clears the failure
a signal-killed program leaves on the unit, releases the firewall exceptions
of a veth machine and removes the machine’s published ports and network
namespace. --no-wait returns right after the request, without the kill after
--timeout (the machine’s --stop-timeout, else 10 seconds); the unit hooks
release the network when the machine ends. With a restart policy, --no-wait
on an app also lets the stub init send the program SIGTERM and SIGHUP, since
nothing stays behind to stop the unit later. -t -1 waits for as long as the
program takes. -f kills every process at once. Stopping a machine that already
ended is not an error, and a
paused machine is thawed first.
A machine with a restart policy stays stopped until it is started again: an
unless-stopped machine is also taken off the boot list until the next
start, while an always one still starts at the next boot.
sudo nspawn restart NAME... [-t SECONDS]
restart is a stop followed by a start with everything the machine
remembers; a machine that is not running is started.
-t is stop’s.
sudo nspawn pause NAME...
sudo nspawn unpause NAME...
pause freezes every process of a machine through the cgroup freezer of its
unit, and unpause thaws them. ps shows a frozen
machine as paused; stop thaws it first, and kill thaws it for a signal
that stops it.
sudo nspawn top MACHINE
PID USER TIME COMMAND
894994 root 00:00:00 (sd-stubinit)
895009 root 00:00:00 /bin/sh -c touch /ok; while true; do sleep 1; done
895198 root 00:00:00 sleep 1
top lists the processes of a running machine: the PID as
the host sees it, the user as the machine sees it, CPU time and command, read
from the machine’s cgroup and PID namespace.
sudo nspawn kill web # SIGKILL: stopped for good
sudo nspawn kill -s HUP web # a signal for the program
kill sends a signal to machines. SIGKILL, the default,
stops the machine for good, as stop --force does. Any other signal goes to
the program of an app, or to the init of a booted machine, and counts as a
stop that keeps the restart policy from bringing the
machine back, unless the machine has a stop signal of its own (its image’s
STOPSIGNAL or --stop-signal) and this is another one: then the policy
decides, should the signal end it. A booted machine’s halt and poweroff
requests count as a stop; other signals to a booted machine leave it to the
policy. Right after a start, kill waits a few seconds at most for the program
to be running, so that the signal reaches it.
$ sudo nspawn run --name job docker.io/library/busybox:1.37 -- sh -c 'exit 3'
$ sudo nspawn wait job
3
$ sudo nspawn port web
80/tcp -> 0.0.0.0:8080
80/tcp -> [::]:8080
wait NAME... waits for machines to stop and prints each one’s exit code, and
port NAME [PORT[/PROTO]] the ports a running machine publishes.
sudo nspawn rm web2
sudo nspawn rm -f web # stop it first
rm removes a machine: its record, its tree, its unit files, the boot link a
restart policy made, and the layers nobody else uses. A pulled image is a
machine too, so rm and images rm remove the same thing; rm -f stops a
running machine first where both refuse otherwise. Named volumes are kept, and
rm says which.
A machine that others depend on (--depends-on) or take their network from
(--network container:NAME) stays while they are there. Named in the same
command, they are removed first and the machine after them; one of them that
stays (running, without -f) keeps it too.
nspawn generate [NAME...] prints the commands that make machines again, on
another host or on this one after a reinstall: a shell script with
network create for the networks they join, then run -d for each machine
with every flag it keeps, after the machines it depends on or takes its network
from, which come along when they are not named. Without names it covers every
machine nspawn has. For the two machines of Dependencies:
$ sudo nspawn generate web
# secret pgpass is encrypted for this host: nspawn secret create pgpass where this script runs, before it
nspawn network create shop --subnet 10.99.1.0/24
nspawn run -d --name db --pull always docker.io/library/postgres:17 \
--network shop \
--env POSTGRES_PASSWORD_FILE=/run/secrets/pgpass \
--volume pgdata:/var/lib/postgresql/data \
--health-cmd 'pg_isready -U postgres' \
--health-interval 2s \
--secret pgpass:/run/secrets/pgpass:0444:0:0
nspawn run -d --name web --pull always docker.io/library/nginx:1.27 \
--network shop \
--publish 8080:80 \
--depends-on db:healthy
Each run -d pulls its machine on its own (--pull always; blobs already
there are not downloaded again), so that a machine never takes the network
kind, mode or backend of another one made from the same image. An image
without flags of its own is pulled again instead of run, unless another
machine takes its network, which needs it running. What a flag cannot carry is
said in comments at the top: a secret is encrypted for this host, so
nspawn secret create it there first; a host directory a machine mounts has
to exist there; an image built here has to be pushed first; a healthcheck in
exec form comes back as --health-cmd, which runs it through /bin/sh, so the
image needs one. The contents of named volumes and of a machine’s writable layer
stay behind, and so do the addresses nspawn handed out: volumes are seeded again
from the image, and only addresses fixed with --ip come along.
Every machine records which networks it uses. start --network changes them,
and the choice sticks for the next start.
| Network | Default for | What the machine gets |
|---|---|---|
bridge | Every image nspawn installs | A fixed address on the nspawn0 bridge, NAT to the outside, published ports, the names of the other machines and of the host. Needs nothing from the host’s own network manager. |
| a network’s name | The same on a bridge of its own, made with network create; several at once, with --network repeated: see Networks of your own. | |
host | The host’s network namespace: the machine sees the host’s interfaces and binds to the host’s ports. Works for both kinds of image. | |
none | No network at all: lo and nothing else. Works for both kinds of image. | |
container:NAME | The network namespace of the running machine NAME: its interfaces, address, hosts file and resolv.conf, nothing of the machine’s own. App images only; see container:NAME. | |
veth | Images not installed by nspawn | The classic systemd-nspawn setup: a virtual ethernet pair whose host end is configured by systemd-networkd through the stock 80-container-ve.network. Booted images only. |
On the first start of a bridged machine (or with nspawn network up, which
is handy at boot) nspawn creates the bridge with the first address of the
subnet, enables IPv4 forwarding and installs the nftables table ip nspawn
with masquerading for the subnet. The defaults are the bridge nspawn0 and the
subnet 10.99.0.0/24; both, and the DNS servers, can be changed in
the configuration file. An interface that already has
that name is only taken over when it is a bridge nspawn made, or an empty one;
bridge = "docker0" is refused rather than acted on. Nothing else on the host
is touched, so it works the same with systemd-networkd, NetworkManager or no
network manager at all.
Each machine gets a fixed address from the subnet, remembered with its record.
A booted machine receives it through a .network file that nspawn
generates and bind-mounts at /run/systemd/network/10-host0.network, for the
systemd-networkd inside to apply, together with the DNS servers: the host’s
upstream resolvers by default, dns from the configuration if set, --dns
and --dns-search of the machine over both, and public resolvers as a last
resort, with a warning, when none can be determined.
An app machine has nothing inside to configure an interface, so nspawn
builds its network namespace before the program starts: ip netns, a veth pair
on the bridge, the address and the default route, handed to systemd-nspawn with
NamespacePath= (--network-namespace-path= on its command line before systemd
259), plus a generated /etc/resolv.conf. The namespace lives at
/run/netns/nspawn-NAME while the machine runs and goes away with it. It
gets net.ipv4.ping_group_range=0 2147483647 (ICMP echo
sockets for every group, what a ping without raw sockets uses) and
net.ipv4.ip_unprivileged_port_start=0 (ports below 1024 for every user), and
then the machine’s --sysctl values, which win.
A generated /etc/hosts, mounted into every bridged machine, resolves the names
of the other machines on the bridge and host.nspawn.internal for the host;
--add-host HOST:IP adds lines of your own, host-gateway standing for the
host’s address on the machine’s network. On hosts with systemd 258 or newer,
machined also lets the host resolve machine names by itself. The bridge
carries IPv4 only, so neither the machines nor the bridge get an IPv6
link-local address, and a machine’s name leads to its bridge address:
ping web from the host answers from 10.99.0.x. Published ports answer
over IPv6 all the same; see IPv6.
network create makes a bridge of its own for a group of machines:
sudo nspawn network create front --label tier=edge
sudo nspawn network create back --internal
sudo nspawn run -d --name db --network back --network-alias postgres -e POSTGRES_PASSWORD=example docker.io/library/postgres:17
sudo nspawn run -d --name web --network front --network back --network-alias front=www -p 127.0.0.1:8080:80 docker.io/library/nginx:latest
front is up: nsbr-front on 10.99.1.0/24
back is up: nsbr-back on 10.99.2.0/24
The bridge is nsbr-NAME (a hash of the name when it is too long for an
interface name), and its subnet the next /24 of network_pool (10.99.0.0/16
unless set in the configuration file) that overlaps no
other network and nothing the host routes already; --subnet chooses one.
Machines of one network reach each other by name, through the same generated
/etc/hosts, and host.nspawn.internal is that network’s gateway. Nothing
of another network reaches them, the default one included: the forward chain
of the ip nspawn table drops what crosses from one bridge to another, and that
drop stands whatever firewalld or iptables accept. Ports a machine
publishes are the exception, reachable from every network through the host,
as from the LAN. --internal makes a network with no way out: its machines
reach each other and the host, nothing beyond, and cannot publish ports.
A machine may join several networks: --network front --network back puts it
on both, with an address on each, the first one primary. The primary network
is where its published ports lead and, unless it is internal, where its default
route goes; an internal primary leaves the route to the first network that is
not. A booted machine gets one interface per network (host0, host1, …),
each configured by the systemd-networkd inside through a .network file of
its own; an app machine gets them in the namespace nspawn prepares. Its
/etc/hosts lists, for every network it is on, the members of that network
with their addresses there, so a proxy on front and back reaches both
sides while front and back still do not reach each other. veth, host,
none and container:NAME go alone.
--network-alias NAME gives a machine another name on its primary network,
--network-alias NETWORK=NAME on that network: every member of the network
resolves the alias as well,
so postgres reaches db now and can be given to another machine later. An
alias several machines share leads to the one whose name sorts first.
--network-alias none forgets them.
--ip ADDRESS fixes the machine’s address on its primary network, --ip NETWORK=ADDRESS on another one: on a network made with network create, not
the default one, inside its subnet, not its gateway, and nobody else’s address
there, or the start or create is refused. The address is kept across restarts,
and a machine made
from this one with create gets an address of its own.
$ sudo nspawn network create lab --subnet 10.99.5.0/24
lab is up: nsbr-lab on 10.99.5.0/24
$ sudo nspawn run -d --name fixed --network lab --ip 10.99.5.30 docker.io/library/busybox:1.37 -- sleep 600
started fixed
$ sudo nspawn exec fixed -- ip -4 -o addr show host0
2: host0 inet 10.99.5.30/24 scope global host0\ valid_lft forever preferred_lft forever
network connect [--alias NAME] [--ip ADDRESS] NETWORK MACHINE and network disconnect NETWORK MACHINE change the networks a machine joins.
A running app or booted machine gets an interface on the network at once
(hostN, with its address; its default route stays where it was, unless it had
none and the network leads out) and loses it at once, and from its next start
it joins the network as if --network had named it:
$ sudo nspawn network connect --alias api --ip 10.99.5.20 lab worker
worker: host1 on lab, 10.99.5.20
$ sudo nspawn exec worker -- ip -4 -o addr show
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: host0 inet 10.99.0.3/24 scope global host0\ valid_lft forever preferred_lft forever
3: host1 inet 10.99.5.20/24 scope global host1\ valid_lft forever preferred_lft forever
$ sudo nspawn network disconnect lab worker
worker: left lab
A machine on host networking, on veth, sharing another’s or without a network,
and an mstack machine, cannot; nor can a machine leave the only network it has
(connect another one first, or start it with --network none). Leaving the
primary network makes the next one primary, where its published ports go at
once while it runs, from its next start otherwise.
network ls lists the networks with the machines whose records name them,
network inspect NAME shows one with its machines, their addresses, aliases
and ports, and network rm and network prune remove the ones no machine
uses, bridge, rules and firewall exceptions included; network prune --filter label=KEY[=VALUE] (or until=) removes only the unused ones it selects.
network create --label KEY=VALUE tags a network for whoever reads network inspect:
sudo nspawn network ls
NETWORK INTERFACE SUBNET INTERNAL MACHINES
bridge nspawn0 10.99.0.0/24 no bb
back nsbr-back 10.99.2.0/24 yes api db web
front nsbr-front 10.99.1.0/24 no web
The /etc/hosts of web, on front and back with www as its alias on
front, lists the members of both networks with their addresses there:
# Generated by nspawn; do not edit.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
10.99.1.2 web www
10.99.2.3 web
10.99.1.1 host.nspawn.internal
10.99.2.4 api
10.99.2.2 db postgres
sudo nspawn network inspect front
[
{
"created": 1790331611,
"gateway": "10.99.1.1",
"interface": "nsbr-front",
"internal": false,
"labels": {
"tier": "edge"
},
"machines": [
{
"address": "10.99.1.2",
"aliases": [
"www"
],
"name": "web",
"ports": [
"127.0.0.1:8080->80/tcp"
],
"running": true
}
],
"name": "front",
"subnet": "10.99.1.0/24"
}
]
sudo nspawn start web -p 8080:80 -p 5353:53/udp -p 127.0.0.1:9090:9090 -p 8000-8010:8000-8010
On IPv4 each -p [IP:][HOST:]CONTAINER[/udp] becomes a DNAT entry in the ip nspawn table. Without an address the port is reachable from other hosts, from
the host’s own addresses and from 127.0.0.1 (through route_localnet); with
one, 127.0.0.1:9090:9090 say, on
that address of the host alone, for a reverse proxy in front. A range,
8000-8010:8000-8010, is one mapping per port, which inspect shows one by
one. The entries are installed once the machine is registered and removed when
it ends, by the unit hooks, so they also go away after a crash or when the
program exits on its own. A port another running machine publishes, on every
address or on that one, or one a service of the host already listens on, is
refused before the machine starts, and a refused port is not remembered. On a
machine with several networks the port leads to its primary address.
Without a host port, -p 80 or -p :80, or with 0, the port gets a free one of
the host, and nspawn port web shows which: the kernel hands one out, passing
over the ports other running machines publish and those a service of the host
holds. A host range with one port of the machine, -p 8000-8010:80, gets the
first free port of the range, and -p 80-82 a free host port for each port.
create picks them, or the start that gave them. The long form,
published=8080,target=80,protocol=udp,host_ip=127.0.0.1, has the same fields,
target the only one required. -P/--publish-all publishes each TCP and UDP
port the image exposes (EXPOSE) that no -p publishes to a free port of the
host.
The list is remembered for the machine, the host ports it got included:
nspawn start web next time publishes the same ports, and -p none forgets
them all. Ports need a bridge network (not an internal one); a machine on the
host’s network listens on the host’s ports directly.
The machines have IPv4 addresses alone, and nftables cannot translate IPv6 into
IPv4, so a port answers on the host’s IPv6 addresses through a proxy in user
space. nspawn-proxy-NAME.service, a transient
unit the machine’s hooks start once it runs and stop with it, listens there and
relays TCP connections and UDP datagrams to the machine’s address; UDP replies
leave from the address the client wrote to, so HTTP/3 works too. The IP of -p
says where a port is published:
-p | Where |
|---|---|
8080:80 | every IPv4 address and, on a host with IPv6, every IPv6 address |
0.0.0.0:8080:80 | every IPv4 address |
127.0.0.1:8080:80 | that IPv4 address |
[::]:8080:80 | every IPv6 address |
[2001:db8::1]:8080:80, [::1]:8080:80 | that IPv6 address |
$ sudo nspawn port web
80/tcp -> 0.0.0.0:8080
80/tcp -> [::]:8080
A link-local address (fe80::) cannot be named, as it needs an interface;
[::] takes it with the others. The port checks cover the IPv6 side as well,
and when the proxy cannot listen, start says so and the IPv4 side works all
the same (journalctl -u nspawn-proxy-web says why). A machine sees an IPv6
client as the bridge’s gateway address, not the client’s own. The client comes
in through the host’s input path, not its forwarding,
so a host firewall has to let the port in, as for any service of the host;
nspawn changes no rule for it.
trusted zone at runtime, which
also lets published ports through on IPv4. The binding does not survive
firewall-cmd --reload; the next start or nspawn network up puts it
back. The IPv6 side of a port needs the port open in the zone of the host’s
interface, as a service of the host does.FORWARD
policy to DROP, which would silence every machine on the bridge. start
then adds two rules to the DOCKER-USER chain, which docker reserves for
that, or to the top of FORWARD itself: anything out of the bridge, and into
the bridge only what was published or belongs to a connection a machine
opened. Where docker or ufw is installed and has not set the policy yet, the
rules go to FORWARD all the same, so that they are there when it changes:
ufw sets it when it is enabled, machines running or not, and docker when it
is the one that turns IP forwarding on. Neither empties FORWARD. That needs
the iptables command, which those tools bring with them.iptables command to edit them,
start says so and the exception has to be made by hand. Without it the
machines reach nothing beyond the bridge and published ports answer on this
host alone.--network veth keeps the classic systemd-nspawn behaviour for booted
machines: systemd-nspawn creates a virtual ethernet pair and systemd-networkd on
the host brings up the host end (ve-NAME), gives it an address, serves DHCP
to the machine and masquerades its traffic, all through the stock
80-container-ve.network. App images refuse it, since nothing inside would
configure the pair.
Because that needs systemd-networkd, start starts it, when it is not running,
if the host has no .network files of its own in /etc/systemd/network or
/run/systemd/network, and refuses with an explanation if it has, so that
nspawn never takes over
interfaces another network manager is handling. A masked systemd-networkd is an
error. With firewalld, ve-NAME is bound to the trusted zone while the
machine runs; otherwise the default zone would drop the machine’s DHCP
requests.
--network host sets VirtualEthernet=no: the machine shares the host’s
network namespace, sees the host’s interfaces and binds to the host’s ports.
Published ports do not apply, and an app that runs this way keeps its user
namespace.
--network none sets Private=yes: the machine has lo and nothing else.
Published ports do not apply, and an app that
runs this way keeps its user namespace too, unless it is given devices of the
host.
--network container:NAME puts an app machine in the network namespace of the
machine NAME, the sidecar pattern (a VPN client with the programs that
must go through it, for one): the same interfaces and address, NAME’s hosts
and resolv.conf files, and nothing of the machine’s own. A port the program
serves is published with -p on NAME, since that is where the address is.
NAME can be an app or a booted machine on any bridge network, has to be
running when the machine starts (--depends-on NAME has it started first, at
boot as well), and cannot be removed while a machine names it, unless that
machine goes in the same rm. When NAME stops or restarts, the machine keeps
the namespace it joined, which then leads nowhere, and has to be restarted to
join the new one.
# the provider's settings (VPN_SERVICE_PROVIDER and the rest) go in vpn.env
sudo nspawn run -d docker.io/qmcgaw/gluetun --name vpn --cap-add NET_ADMIN --device /dev/net/tun --env-file vpn.env -p 8080:80
sudo nspawn run -d docker.io/library/nginx:1.27 --name web --network container:vpn
For web, the service binds the name systemd-nspawn looks for
(/run/netns/nspawn-web) to the network namespace of vpn’s leader process,
the mount ip netns attach would make, so the settings are the ones of any app
on a bridge, and drops the name when web stops while the namespace stays
vpn’s. -p, --ip, --network-alias, --dns, --dns-search, --add-host
and --sysctl are refused on such a machine, since they shape a network of its
own, and so is a booted image, whose systemd would configure the shared
interfaces again. ps and inspect show the network as container:NAME;
network inspect does not list the machine among the network’s members, as it
has no address there.
--interface IFACE on run, create and start gives a machine a network
interface of the host, whole: a second ethernet port, or a wifi adapter for a
machine doing wireless work. --device cannot do it, since a network interface
is not a node under /dev, and --network host cannot either: the machines
run in a user namespace, which may not configure the host’s interfaces. The
interface is moved into the machine’s network namespace before its program or
init runs and is back on the host when the machine stops or is removed, with
its name kept on both sides, as systemd-nspawn’s Interface= does it. It
arrives down and unconfigured, as the kernel moves it: the
machine brings it up and configures it, with ip in an app or its
systemd-networkd in a booted machine. The flag is repeatable and remembered
like the rest, --interface none forgets them, and inspect lists them as
interfaces.
sudo nspawn run -d kali:latest --name kali --interface wlp11s0f3u2u3
sudo nspawn exec kali -- iw dev
sudo nspawn stop kali # the adapter is back on the host
Where systemd-nspawn makes the machine’s network namespace (a booted machine,
on the bridge or with veth, and any machine with --network none) the
interface goes into the machine’s settings file as Interface= and
systemd-nspawn moves it at start and back at exit. An app machine on a bridge
network gets its namespace from nspawn itself, by its path, which takes no
Interface= beside it, so the unit’s hooks move the interface in with ip
before systemd-nspawn runs and give it back after; such an app runs without a
user namespace, so iw reg set and monitor mode work inside as on the host.
Either way the unit waits for the interface’s device unit, so that a machine
started at boot waits for a USB adapter udev has not seen yet.
A wifi adapter cannot leave its phy, so the phy moves whole, every interface it
carries with it (nspawn says which), and a driver without namespace support
(ath6kl, wilc1000) refuses the move naming the phy. An app on the bridge
takes a wireless interface on any systemd, through iw, which the host
needs; Interface= moves a phy since systemd 256, so on an
older host a wireless interface is refused for a booted machine or one with
--network none, with a message that says so. The regulatory domain (iw reg set) and rfkill are host-wide: they are set on the host, or from an app on
the bridge; --device /dev/rfkill hands the switch to such an app.
One machine at a time takes an interface: a second one naming it is refused
while the first runs (stop it, or start it with --interface none). Refused
as well: lo, the bridges and veth ends of nspawn’s own, a port of a bridge or
bond (ip link set IFACE nomaster first), a name that is not on the host (a
machine that ended a moment ago may still hold it: the kernel gives interfaces
back a little after their namespace dies), a machine with --network host, one
with --network container:NAME (give the interface to NAME) and the mstack
backend. The interface of the host’s default route is allowed with a warning,
since the host loses its route while the machine runs. When the interface
comes back, NetworkManager or systemd-networkd manage it again as before, and
a name taken on the host in the meantime makes the kernel rename it (dev0,
wlan0). stop --force kills the machine: an app on the bridge still gets
every interface back from its release hook, while a machine that took them
through Interface= may not get to give them back, and the kernel then
returns a physical interface or a wifi phy a moment later and destroys a
virtual one (a VLAN, a macvlan, a dummy) with the namespace.
nspawn compose runs docker compose files on nspawn’s machines, with the
commands, flags and output of docker compose 5.6.0: in a directory with a
compose.yaml, sudo nspawn compose up -d brings the services up and
sudo nspawn compose down takes them away. It is a client of the
service like every other
command, so it takes the same permissions.
services:
web:
image: nginx:alpine
ports:
- "8080:80"
depends_on:
db:
condition: service_healthy
db:
image: postgres:18-alpine
environment:
POSTGRES_PASSWORD: example
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 2s
volumes:
- db:/var/lib/postgresql
volumes:
db:
In a directory shop with that file:
$ sudo nspawn compose up -d
Image postgres:18-alpine Pulling
Image postgres:18-alpine Pulled
Image nginx:alpine Pulling
Image nginx:alpine Pulled
Network shop_default Creating
Network shop_default Created
Volume shop_db Creating
Volume shop_db Created
Container shop-db-1 Creating
Container shop-db-1 Created
Container shop-web-1 Creating
Container shop-web-1 Created
Container shop-db-1 Starting
Container shop-db-1 Started
Container shop-db-1 Waiting
Container shop-db-1 Healthy
Container shop-web-1 Starting
Container shop-web-1 Started
$ sudo nspawn compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
shop-db-1 postgres:18-alpine "docker-entrypoint.s…" db 4 seconds ago Up 3 seconds (healthy)
shop-web-1 nginx:alpine "/docker-entrypoint.…" web 3 seconds ago Up Less than a second 0.0.0.0:8080->80/tcp, [::]:8080->80/tcp
$ sudo nspawn compose exec db psql -U postgres -c 'select 1'
?column?
----------
1
(1 row)
The services are nspawn’s machines, and every other command sees them:
$ sudo nspawn ps
MACHINE IMAGE MODE COMMAND STATE UP PID NETWORK OS
shop-db-1 docker.io/library/postgres:18-alpine app docker-entrypoint.sh postgres running (healthy) 3s 19093 shop_default:10.99.1.2 Alpine Linux v3.24
shop-web-1 docker.io/library/nginx:alpine app /docker-entrypoint.sh nginx -g daemon... running 0s 19262 shop_default:10.99.1.3 8080->80/tcp Alpine Linux v3.24
sudo nspawn compose down -v stops and removes the machines, the network and,
with -v, the volume.
PROJECT-SERVICE-N with _
written -. The machines carry the com.docker.compose.* labels, which is
how ps, logs,
stop, down and the rest find a project again, from its files or from
-p NAME alone.postgres:18-alpine is postgres-18-alpine); every machine of
it is made from that image.PROJECT_KEY, and a volume an nspawn volume, also
PROJECT_KEY. A secret or config from a file is bound read-only where the
service reads it; one from the environment or written in the file is an
nspawn secret, removed by down.VOLUME of the image nothing of the service
mounts over, is an nspawn volume named after the machine, MACHINE_anon_HEX.
A machine made anew finds its own again; up -V makes new ones, and down -v
and rm -v remove them.up made it is made anew, with the
project’s machines on it.depends_on orders what compose does: up, start
and run bring the services up in order and wait for their conditions, and
restart also restarts the services whose depends_on sets restart: true. The machines carry no
--depends-on, so nspawn does not keep a service that others depend on from
being made anew, scaled down or removed. With -p NAME and no -f, down,
stop, logs and the other commands that act on what runs take the project
from the labels, a compose file in the directory or not: the labels say what
depends on what, so down SERVICE takes the services that depend on it too.
With the files it takes that service alone.com.docker.compose.config-hash label records, or its image did./etc/letsencrypt/live for anyone
but root) is left to the service, which runs as root and mounts it, or
refuses it when it is not there either.A service with a build section is built by nspawn’s
builder: when its
image is not here (an image it names is pulled first, and built when the pull
fails), every time with pull_policy: build, and always with up --build or
compose build; --no-build builds nothing. Its image is the one it names or
PROJECT-SERVICE. A name without a registry or a namespace is nspawn’s own,
under the configured registry as nspawn build -t puts it, so
sudo nspawn run NAME takes it; one like user/app stays Docker Hub’s, where
compose build --push sends it. down --rmi local removes the images the
project built.
config, up, down, create, start, stop, restart, kill, pause,
unpause, rm, run, exec, ps, ls, logs, top, port, images,
pull, build, scale, wait, cp, stats, events, volumes and
version, each with its flags. alpha, attach, bridge, commit,
export, publish, push and watch are refused with an error that names an
alternative where there is one; --dry-run is refused by the commands that
change something, as nothing simulates their changes.
The files load with interpolation, .env, profiles, include, extends,
overrides, !reset and !override, and the schema’s checks, with docker
compose’s messages; sudo nspawn compose config prints the project as
loaded. What nspawn cannot do (remote build
contexts, SSH forwarding into a build, Swarm-only attributes) config shows as
warnings, and up, create, run and scale refuse it for the services they
start.
COMPOSE.md lists every attribute and option with what nspawn compose does with it: yes, partly (with the difference), ignored with a warning, or refused with the reason.
build builds a Dockerfile with a builder of nspawn’s own, or runs mkosi on an
mkosi configuration. A directory with a Dockerfile (or dockerfile,
Containerfile), -f naming one, - on standard input or any of the
Dockerfile flags below make it a Dockerfile build; a directory with
mkosi.conf or mkosi.conf.d and no Dockerfile, or the mkosi flags, an mkosi
build. Either result is stored like a pulled image and can be started right
away or pushed. A build of a tag replaces the image of that reference under the
tag’s local name as long as nothing ran from it, whatever it was pulled or built
as: the tag moves to the new build, so a script that builds and runs its image
builds it again. A machine that has the name (one that ran, starts or runs), or
an image of another reference, keeps it, and the build goes beside it with the
same reference, into the image an earlier build of the tag put there or else
under a free name (app-2), which a note says; run, create, compose and the
Docker API find it by the reference, the image before the machine. An mkosi
build given --name is refused instead when a machine or an image of another
reference has that name. --force replaces whatever has the name, a machine
that ran included, and refuses one that runs. -f names the Dockerfile, so
--force has no short form.
sudo nspawn build [DIRECTORY|-] [-t TAG]... [-f DOCKERFILE] [--build-arg KEY[=VALUE]]...
[--target STAGE] [--no-cache] [--no-cache-filter STAGE]... [--pull]
[--label KEY=VALUE]... [--network default|none|host] [--add-host HOST:IP]...
[--allow network.host] [--secret id=ID[,src=PATH|,env=VAR]]...
[--shm-size SIZE] [--ulimit NAME=SOFT[:HARD]]... [-q] [--progress TYPE]
[--iidfile FILE] [--push] [--no-verify] [--platform PLATFORM] [--load]
[--backend BACKEND] [--mode MODE] [--force]
The Dockerfile is read as BuildKit’s frontend reads it (a # syntax= of
docker/dockerfile:1, 1.N, 1.N.M or labs is taken, another frontend image is
refused), and run by a builder of nspawn’s own: every stage the target needs
(--target, the last one otherwise), multi-stage builds, COPY --from, ARG
and --build-arg, ONBUILD, heredocs, COPY --chown --chmod --parents --exclude --link, ADD of local archives (gzip, bzip2, xz, zstd or plain tar,
unpacked) and of http(s) URLs (--checksum), and RUN --mount of the cache,
bind, tmpfs and secret kinds, and --build-arg SOURCE_DATE_EPOCH=SECONDS for the
time of the image and of the history entries the build adds. The context is the
directory given. The command line packs it as the calling user, everything owned
by root, times to the nanosecond and file capabilities kept, and leaves out what
.dockerignore excludes (Dockerfile.dockerignore next to the Dockerfile wins
over it); - takes a context tar, unpacked with its owners, or a Dockerfile with
no context, from standard input.
An archive that ADD unpacks goes into the destination entry by entry: an entry
replaces what is at its path unless both are directories, a symlink on the way
is followed inside the destination, a parent the archive does not list is made
0755, and its owners (unless --chown), modes, times, FIFOs, device nodes and
extended attributes are kept. A file ADD fetches from a URL has the time of
its Last-Modified, the epoch without one.
A static program built in one stage and copied into an empty image:
FROM alpine:3.22 AS build
RUN apk add --no-cache gcc musl-dev
COPY hello.c /src/
RUN gcc -static -o /hello /src/hello.c
FROM scratch
COPY --from=build /hello /hello
ENTRYPOINT ["/hello"]
$ sudo nspawn build -t hello:1 .
docker.io/library/alpine:3.22: manifest 3e9b4b680bfc with 1 layer(s)
#1 [build 1/3] RUN apk add --no-cache gcc musl-dev
#1 0.070 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/APKINDEX.tar.gz
...
#1 1.387 OK: 172 MiB in 29 packages
#1 DONE 8.5s
#2 [build 2/3] COPY hello.c /src/
#2 DONE 0.2s
#3 [build 3/3] RUN gcc -static -o /hello /src/hello.c
#3 DONE 0.2s
#4 [stage-1 1/2] COPY --from=build /hello /hello
#4 DONE 0.7s
#5 [stage-1 2/2] ENTRYPOINT [/hello]
#5 DONE 0.0s
image hello-1 is ready: nspawn run hello-1, nspawn push hello-1
$ sudo nspawn run --rm hello:1
hello from a scratch image
Every step runs in a root of its own, an overlay of the layers so far, and what
it changes becomes a layer. A RUN runs under systemd-nspawn with the
confinement of an app machine (the default capabilities, the system call
filter, no user namespace so that files keep their owners), as PID 1 with the
hostname buildkitsandbox, in a network namespace the build sets up on the
default bridge; RUN --network=none has none, and --network=host needs
--allow network.host. A RUN --mount=type=tmpfs is mode 1777.
Steps that ran before with the same inputs come from the build cache, so the same build again takes a moment and gives the same image ID:
$ sudo nspawn build -t hello:1 .
#1 [build 1/3] RUN apk add --no-cache gcc musl-dev
#1 CACHED
#2 [build 2/3] COPY hello.c /src/
#2 CACHED
#3 [build 3/3] RUN gcc -static -o /hello /src/hello.c
#3 CACHED
#4 [stage-1 1/2] COPY --from=build /hello /hello
#4 CACHED
#5 [stage-1 2/2] ENTRYPOINT [/hello]
#5 DONE 0.0s
image hello-1 is ready: nspawn run hello-1, nspawn push hello-1
A cache entry keeps the layers its step ran on, so a --no-cache build of
another Dockerfile with the same first steps does not invalidate this one’s
later steps.
--no-cache builds every step again, --no-cache-filter STAGE the steps of a
stage, and --pull asks the registry for the base images even when they are
here.
| Option | Meaning |
|---|---|
-t, --tag TAG | Reference for the result; repeatable. A name without a registry is under the configured one, as pull reads it. Without -t the build fills the cache and prints the image ID. |
-f, --file PATH | The Dockerfile; default Dockerfile, dockerfile or Containerfile in the context. - reads it from standard input. |
--build-arg KEY[=VALUE] | A build argument; KEY alone takes it from the environment. Repeatable. |
--target STAGE | The stage to build. |
--no-cache, --no-cache-filter STAGE | Build every step, or a stage’s steps, again. |
--pull | Ask the registry for the base images again. |
--label KEY=VALUE | A label for the image. Repeatable. |
--network MODE | The network of RUN steps: default, none, or host with --allow network.host. |
--add-host HOST:IP | A line for the hosts file of RUN steps; host-gateway is the bridge’s address. Repeatable. |
--secret id=ID[,src=PATH|,env=VAR] | A secret for RUN --mount=type=secret, read by the command line as you. Repeatable. |
--shm-size SIZE, --ulimit NAME=SOFT[:HARD] | /dev/shm and resource limits of RUN steps. |
-q, --quiet | Print only the image ID. |
--progress auto|plain|tty|quiet | How to show the steps; every one prints every line. |
--iidfile FILE | Write the image ID to a file. |
--push | Push the tags once built. |
--no-verify | Pull the base images without verifying their signatures. |
--platform, --load | This host’s platform is the only one; the image is local already. |
--rm, --force-rm, -m, --memory-swap, -c, --cpu-period,
--cpu-quota, --cpuset-cpus, --cpuset-mems, --compress, --squash,
--isolation and --security-opt, which Makefiles still give, are taken and
ignored, three of them (--isolation, --security-opt, --squash) with a
warning.
A failing step ends the build with
process "/bin/sh -c false" did not complete successfully: exit code: 1, and
leaves nothing behind; a step whose process a signal ends exits with 128 plus
the signal’s number, and a build whose command line goes away (Ctrl-C) stops the
same way.
$ sudo nspawn builder du
Reclaimable: 1.112GB
Total: 1.141GB
builder du -v lists every entry, builder prune removes the cache entries no
image shares (-a all of them, --filter until=24h, --keep-storage 10g),
system df counts the cache and system prune removes it.
No SSH forwarding (--ssh), no output other than a local image (--output,
--cache-to), no named, remote or git contexts, no ADD of a git repository, no
attestations (--provenance=false and --sbom=false are taken), only this
host’s platform, no RUN --security=insecure or RUN --device, no --check or
--call, and no linter: a # check= directive or BUILDKIT_DOCKERFILE_CHECK is
read and its words checked, and no rule runs.
Stages run one after another; COPY --link copies as a plain COPY; /run is
a tmpfs during a RUN, so what a step writes there is not kept, and
systemd-nspawn adds container=systemd-nspawn, container_uuid, USER and
LOGNAME to a step’s environment. Each step starts from the layers as the image
keeps them, so what a layer does not carry (ACLs and extended attributes other
than file capabilities) is gone for the next step. FIFOs and device nodes of a
context directory are left out of it; those of a context tar are kept. A RUN of
USER 0:GID or USER root:GROUP runs with group 0, and a RUN as root without
root’s supplementary groups: systemd-nspawn sets no group of its own for root.
sudo nspawn build [DIRECTORY] -t TAG [--name NAME]
[-d DISTRIBUTION] [-r RELEASE] [--profile PROFILE]...
[--backend BACKEND] [--mode MODE] [--force] [--keep-output]
[-- MKOSI_ARGUMENTS...]
build runs mkosi on a directory that has
a mkosi.conf or a mkosi.conf.d (the current directory by default), with
--format=oci, and imports the result exactly as pull would import an image
from a registry. The service runs mkosi, so mkosi has to be installed where the
service finds it (/usr/bin). The same configuration
tree that works with mkosi on its own works here; --distribution,
--release, --profile and anything after -- are passed through.
The command line that mkosi receives is:
mkosi --directory=DIRECTORY --format=oci --compress-output=zstd
--output-directory=/var/lib/nspawn/builds/mkosi-NAME-TIMESTAMP-SUFFIX
--cache-directory=/var/lib/nspawn/cache/mkosi
--workspace-directory=/var/lib/nspawn/cache/mkosi-workspace/mkosi-NAME-TIMESTAMP-SUFFIX
--image-id=REPOSITORY [--distribution=...] [--release=...] [--profile=...]...
[MKOSI_ARGUMENTS...] --force build
The / characters of the repository are replaced by - in the image id. The
package cache is kept between builds; the workspace is deleted at the end, and
the output directory after the import unless --keep-output is given.
When mkosi is done, nspawn finds the OCI layout it produced, adds a few
annotations to the manifest (org.opencontainers.image.version with the tag
when mkosi wrote none from ImageVersion=,
org.opencontainers.image.ref.name with the full reference, and
org.nspawn.builder), verifies every blob against its digest, copies the blobs
into the store and installs the image with the chosen backend, with build as
its origin. The image can be started right away, cloned with create or
pushed. The labels of its OCI configuration, which mkosi writes from
OciLabels=, are kept with it: nspawn inspect shows them as image_labels,
and they travel with the image when it is pushed.
A minimal configuration for a Fedora machine that boots systemd, taken from the test suite of nspawn:
[Distribution]
Distribution=fedora
Release=44
[Output]
ImageId=e2e-built
OciLabels=org.nspawn.e2e=built
[Content]
Bootable=no
SELinuxRelabel=no
RootPassword=root
Packages=
systemd
systemd-networkd
systemd-resolved
dbus-broker
passwd
util-linux
systemd-networkd is what configures host0 on the bridge, and dbus-broker
(or dbus) is what lets shell reach the machine’s systemd. Then:
sudo nspawn build -t team/app:1 ./app
sudo nspawn start team-app-1
sudo nspawn push IMAGE [--to REFERENCE]
IMAGE is a local image name or the reference it was pulled from or built as.
By default the image is pushed under its own reference; --to pushes it under
another one, for example to retag it or to send it to a different registry:
sudo nspawn push team/app:1
sudo nspawn push team-app-1 --to registry.example/team/app:2
push authenticates first, with the credentials stored for that registry, and
tells you which nspawn login to run when the registry wants some it does not
have. The blobs that are already on the registry are skipped and the manifest
is uploaded last. A push needs a tag, not a digest, and the image’s manifest
and blobs must still be in the store (images rm removes them). See
Registries and credentials.
The images the team publishes on hub.nspawn.org are built with mkosi from the
definitions in
nspawn/mkosi-definitions. Use
them as a starting point for your own images, or open a pull request there to
change what the hub ships.
nspawn serves the Docker Engine API, 1.51 and any version down to 1.24 a client
asks for, on a socket of the host and in each machine that mounts
/var/run/docker.sock. Clients make, start, attach to, exec in, copy files to and from and
remove containers, pull and remove images, and make, join and remove volumes and
networks. The server is a client of the service,
as the command line is: every change it makes is a method of org.nspawn, under
the same polkit actions, and what it makes is a machine like any other, which
nspawn ps shows.
The packages ship it turned off:
sudo systemctl enable --now nspawn-api.socket
export DOCKER_HOST=unix:///run/nspawn/api.sock
Every user may connect to /run/nspawn/api.sock, and polkit decides what each
one may do, with the service’s actions: reads and waits ask for
org.nspawn.inspect, logs and changes for org.nspawn.manage, _ping and
version for nothing. polkit is never asked to prompt, since nothing on a socket
answers an authentication agent: a user other than root needs a rule, such as
nspawn-wheel.rules, which the packages ship as an example. Root may do
everything.
Then the docker command works with it:
$ docker pull busybox:1.37
1.37: Pulling from library/busybox
docker.io/library/busybox:1.37: manifest 66a6306db78b with 1 layer(s), assembling as overlay
68fe9bff2ad4: Already exists
30ecbe150909: Already exists
Digest: sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec
Status: Downloaded newer image for busybox:1.37
docker.io/library/busybox:1.37
$ docker run --rm busybox:1.37 echo hello
hello
$ docker run -d --name web -p 8080:80 busybox:1.37 httpd -f -p 80 -h /tmp
c7dac8ca487657695c14e21240f15fd99e7eac48d6c13e770febe7b0ac74d31e
$ echo hello > index.html && docker cp index.html web:/tmp/index.html
$ curl -s http://127.0.0.1:8080/
hello
$ docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
NAMES IMAGE STATUS PORTS
web busybox:1.37 Up Less than a second 0.0.0.0:8080->80/tcp, [::]:8080->80/tcp
$ docker exec web sh -c 'id; pwd'
uid=0(root) gid=0(root)
/
$ sudo nspawn ps
MACHINE IMAGE MODE COMMAND STATE UP PID NETWORK OS
web docker.io/library/busybox:1.37 app httpd -f -p 80 -h /tmp running 0s 2814430 10.99.0.3 8080->80/tcp -
Volumes and networks are nspawn’s:
$ docker volume create data
data
$ docker network create back
68735561ab7cd7e4da9b248b9c68ff65d4518c05d1730d79056e8dd3c2dc8eda
$ docker network connect back web
$ docker run --rm --network back -v data:/data busybox:1.37 sh -c 'wget -qO- http://web/ | tee /data/page'
hello
$ docker run --rm -v data:/data busybox:1.37 cat /data/page
hello
Where nothing else on the host has /run/docker.sock, clients that only look
there or at /var/run/docker.sock find the socket with a drop-in:
sudo mkdir -p /etc/systemd/system/nspawn-api.socket.d
printf '[Socket]\nSymlinks=/run/docker.sock\n' | sudo tee /etc/systemd/system/nspawn-api.socket.d/symlink.conf
sudo systemctl daemon-reload && sudo systemctl restart nspawn-api.socket
GitLab Runner’s docker executor runs its jobs through the socket: the clone, the
job’s image, services with their aliases, artifacts, cache volumes and a network
per build (FF_NETWORK_PER_BUILD). In its config.toml:
[[runners]]
executor = "docker"
[runners.docker]
host = "unix:///run/nspawn/api.sock"
image = "alpine:3.22"
The runner runs as root, or as a user that a polkit rule lets through.
_ping, version, info and system df.Warnings (CpuShares, SecurityOpt, a logging driver other than
json-file, local, journald or none, among others); what would change what
the container does is refused with 400 (PidMode, IpcMode or UTSMode of the
host, a user namespace mode, another runtime, VolumesFrom, an IPv6 address).docker run -i, docker run -it and docker start -ai
give the program their input, and an attach that keeps the input gets the output
byte for byte.docker cp both ways, on running containers and on stopped ones, owners
and modes kept. A volume of a stopped container is mounted only while it runs, so
a copy to or from it is refused until it starts.POST /images/create), remove and prune.
A record pulled or built that never ran is an image, not a container; a machine
made from it is a container. An image’s ID is its config’s digest. An image built
here with a name without a registry (nspawn build -t somos/app:1, which nspawn
files as hub.nspawn.org/somos/app:1) goes by that short name too while Docker
Hub’s image of it is not here, so docker run somos/app:1 finds it.bridge, host and none are the
pre-defined networks.Builds, loads, imports, tags, pushes and commits of images, container renames and exports, swarm and plugins answer 501 with “not supported”.
A name the API takes that is no machine name, such as reaper_1, stays the
container’s name in the API, and its machine gets one made of it (reaper-1- and
eight hex digits), which nspawn ps shows. A network keeps the name the API gave
it the same way.
A machine that mounts /var/run/docker.sock, /run/docker.sock or
/run/nspawn/api.sock from the host gets nspawn’s Docker API there, whatever the
host has at that path, with -v /var/run/docker.sock:/var/run/docker.sock or
compose’s use_api_socket. Mounted read-write, it takes every request the API serves:
$ sudo nspawn run --rm -v /var/run/docker.sock:/var/run/docker.sock \
docker.io/library/docker:28.5.1-cli -- docker run --rm busybox:1.37 echo from-a-machine
docker-28.5.1-cli-6fd52594: 10 layer(s) shared with docker-28.5.1-cli, assembling as overlay
from-a-machine
Mounted with :ro, it reads, follows logs and kills, and refuses any other change
with 403:
$ sudo nspawn run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro \
docker.io/library/docker:28.5.1-cli -- docker run --rm busybox:1.37 true
docker-28.5.1-cli-c46d6cef: 10 layer(s) shared with docker-28.5.1-cli, assembling as overlay
docker: Error response from daemon: this machine binds nspawn's Docker API read-only
Run 'docker run --help' for more information
Tools that follow containers through the socket find nspawn’s machines the same
way: reverse proxies that route by labels (caddy-docker-proxy, Traefik,
nginx-proxy with docker-gen and acme-companion), dashboards (homepage) and log
viewers (Dozzle). They only read, and kill to have a server read its
configuration again, so :ro is all they need. caddy-docker-proxy in front of a
service, from a compose.yaml in a directory proxy:
services:
caddy:
image: lucaslorentz/caddy-docker-proxy:2.13-alpine
ports:
- "80:80"
environment:
- CADDY_INGRESS_NETWORKS=proxy_default
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
whoami:
image: traefik/whoami
labels:
caddy: http://whoami.example.test
caddy.reverse_proxy: "{{upstreams 80}}"
$ sudo nspawn compose up -d
$ curl -s -H 'Host: whoami.example.test' http://127.0.0.1/ | head -2
Hostname: proxy-whoami-1
IP: 127.0.0.1
A machine that has the socket sees every machine and their -e variables
(--secret values never show). Only root inside the machine can connect.
Each machine that takes the socket has its own, /run/nspawn/api/NAME.sock,
opened by systemd (nspawn-api@NAME.socket) before the machine starts and closed
after it stops; the host’s is nspawn-api.socket. The server behind each,
nspawn-api@NAME.service or nspawn-api.service, starts when something connects
and leaves a minute after the last connection, the attached sessions and exec
commands of its clients included. A container ID comes from the machine’s name and
creation time, so it stays across restarts and updates. nspawn binds the machine’s
/etc/hostname from a directory named after that ID, so tools that look for their
own container in /proc/self/mountinfo find it.
What one socket’s clients hold at once is bounded: 64 connections (one more is
closed at once), 16 of them logs, events or attach streams, each running a
journalctl, 16 waits and attaches, and 64 exec commands that run; one more of
any of these gets 503. On the host’s socket a user other than root holds a quarter
of the connections and of the exec commands and half of the streams and of the
waits at most, so that no user keeps the others or root out, and root has 16
streams of its own. mstack machines cannot take the socket.
sudo nspawn auto-update moves the machines labelled
org.nspawn.auto-update=registry to the image their tag points to now.
io.containers.autoupdate=registry is taken too, on the machine, never from
its image, with image meaning registry and disabled or an empty value no
updates.
$ sudo nspawn run -d --name tagged -l org.nspawn.auto-update=registry \
docker.io/library/busybox:latest -- sleep 600
$ sudo nspawn auto-update --dry-run
UNIT CONTAINER IMAGE POLICY UPDATED
systemd-nspawn@tagged.service tagged docker.io/library/busybox:latest registry false
For each machine the registry is asked for the tag’s digest, with a HEAD that
registries do not count against their limits. A newer image is pulled, its
signature policy included, and the machine is made anew from it with every
option it had: networks and address, ports, volumes, variables, labels,
limits, restart policy and the rest (generate prints the same command before
and after), and its creation time, so its Docker API ID stays. A running
machine is stopped and started again and has to come up: healthy when it has a
healthcheck, still running ten seconds after its start otherwise. When it does
not, the old image and what the machine had written are put back and it starts
again, which the table reports as rolled back.
A machine on the new image starts with a fresh writable layer: what it keeps
belongs in volumes. Machines that share its
network (--network container:) start again with it, and so do compose
services whose depends_on asks for restart: true. Built images, references
pinned by digest, run --rm machines, booted machines (their state is their
writable layer), flat and mstack ones, and machines whose image declares a
volume (VOLUME) none of their own volumes covers, whose data would stay in the
old writable layer, are left alone and show skipped, as the local policy
does; the reason goes to standard error for a machine named on the command
line, and is in --json always. A policy that is none of these (a misspelt
one) shows failed, so that the run and its timer show it.
--dry-run only says which machines have a newer image waiting (pending).
The UPDATED column takes false, pending, true, failed, rolled back
(the old image back and running) or skipped; --json prints a JSON object per machine, and the command fails
when a machine failed or went back.
The packages ship a daily timer with up to fifteen minutes of random delay, disabled:
sudo systemctl enable --now nspawn-auto-update.timer
nspawn needs no configuration to work: the defaults point at the hub, use
/var/lib/machines and /var/lib/nspawn, pick the best backend for the host
and run the bridge on 10.99.0.0/24. Everything below is optional.
--registry or
NSPAWN_REGISTRY, --ca-cert or NSPAWN_CA_CERT).The service reads /etc/nspawn/nspawn.toml when it exists; nspawn --config FILE daemon --install puts it on another file, which then must exist. On any
other command, --config (or NSPAWN_CONFIG) only changes the registry and CA
certificate the command line passes to the service. Every key is optional, and
unknown keys are an error so that a typo does not silently fall back to a
default.
| Key | Type | Default | Meaning |
|---|---|---|---|
registry | string | hub.nspawn.org | Registry for references without a host part. |
ca_cert | absolute path | none | Extra CA certificate (PEM) trusted for the configured registry only. |
backend | auto, overlay, flat or mstack | auto | Backend for pull and build when they are not given --backend: auto is overlay, or flat without overlayfs; mstack is experimental. |
machines_dir | /var/lib/machines, /etc/machines, /run/machines, /usr/local/lib/machines or /usr/lib/machines | /var/lib/machines | Where machines are assembled: one of the directories systemd-nspawn finds machines in. |
state_dir | absolute path | /var/lib/nspawn | Blobs, layers, records, volumes and everything else nspawn keeps. It may neither be inside a directory machined lists images from nor hold one, where machinectl clean or remove would delete what nspawn keeps. |
bridge | string, 1 to 15 letters, digits, - or _ | nspawn0 | Name of the bridge the machines join. It must be free, empty, or a bridge nspawn made. |
subnet | IPv4 CIDR, prefix 8 to 30 | 10.99.0.0/24 | Subnet of the bridge; its first address is the bridge’s own. |
network_pool | IPv4 CIDR | 10.99.0.0/16 | Where network create takes the /24 of a network given no --subnet: the first one that overlaps no other network and nothing the host routes. |
dns | list of IPv4 addresses | the host’s upstream servers | DNS servers handed to bridged machines. They must be reachable from the bridge: no loopback, no IPv6. |
An example:
# /etc/nspawn/nspawn.toml
registry = "registry.example:5000"
ca_cert = "/etc/pki/tls/certs/example-ca.pem"
backend = "overlay"
bridge = "br-lab"
subnet = "172.30.5.0/24"
dns = ["172.30.5.1", "9.9.9.9"]
Changing bridge or subnet affects machines started afterwards; the address
recorded for a machine is reassigned from the new subnet on its next start.
What the images of a registry must carry, one table per registry (host, or
host:port). The hub’s policy is built in (the project’s key or the build
workflow’s keyless identity, one of them required, its transparency log entry
checked): a table for hub.nspawn.org replaces it, and verify = false turns
the check off. Every other registry is verified only when it has a table.
[registries."registry.example.com"]
key = "/etc/nspawn/keys/example.pub" # a cosign public key (PEM); keys = [...] for several
identity = "https://github.com/org/repo/.github/workflows/build.yml@refs/heads/main"
issuer = "https://token.actions.githubusercontent.com" # goes with identity
required = true # default: no verifying signature fails the pull
rekor = true # default: the signature's transparency log entry must verify
trusted_root = "/etc/nspawn/trusted_root.json" # another Sigstore deployment; the public one otherwise
[registries."hub.nspawn.org"]
verify = false # nothing is checked for this registry
| Key | Type | Default | Meaning |
|---|---|---|---|
verify | boolean | true | false turns verification off for the registry, the hub’s built-in policy included. |
key, keys | absolute path, list of absolute paths | none | Cosign public keys (PEM) the images may be signed with; key and keys add up. |
identity, issuer | strings, both or neither | none | A keyless signer: the certificate’s identity (the SAN, a URI or an email) and the issuer of the token behind it. A table names at least one key or an identity, unless verify = false. |
required | boolean | true | false turns a missing signature into a note and lets the pull go on; a signature that is there but does not verify still fails it. |
rekor | boolean | true | false trusts the key or certificate alone, without the transparency log entry (a private deployment that keeps no log). A bundle still has to carry a log entry or a timestamp, which cosign always adds. |
trusted_root | absolute path | the public Sigstore root the binary embeds | The trusted_root.json of another Sigstore deployment. |
The keys are read by the service when a pull needs them, and checked when it
starts: a key or a trusted root it cannot read is reported in its journal and
fails the pulls of that registry. What the command line was given with
--registry chooses the registry, never the policy.
The file is read by the service, so a change takes effect on its next start:
sudo systemctl restart nspawn.service (or waiting for it to go idle)
picks it up. --config on the command line does not reach it: the command line
is a client and only carries the registry and its CA certificate. To put the
service on another file, name it when installing the service,
sudo nspawn --config /etc/nspawn/lab.toml daemon --install; the unit then
carries it, and so do the hooks the service writes for every machine.
| Variable | Same as |
|---|---|
NSPAWN_REGISTRY | --registry |
NSPAWN_CA_CERT | --ca-cert |
NSPAWN_CONFIG | --config |
They are convenient for scripts:
sudo NSPAWN_REGISTRY=registry.example:5000 NSPAWN_CA_CERT=/etc/pki/tls/certs/example-ca.pem nspawn hub ls
nspawn login keeps registry credentials in /etc/nspawn/auth.json, and that
file is the only one consulted. See
Registries and credentials.
Some settings belong to a machine rather than to the host, and are given when it is created or started; every one of them is remembered until it is changed:
--backend and --mode on pull and build; --backend on run and
create.--name on pull, run and an mkosi build (a Dockerfile build is named by
-t), to choose the local name.--network (bridge, veth, host, none or networks made with
network create, several of them), --network-alias, --ip, -p, -e,
-v, --label, --entrypoint and the arguments after -- on start, run
and create. -p none, -e none, -v none, --label none,
--network-alias none and --image-command forget what was remembered.--restart, -m/--memory, --memory-swap, --cpus and --pids-limit on
start, run and create, applied at the next start, and on update,
applied at once.
--restart no and a limit of 0 remove them.--health-cmd and the other --health-* flags,
--no-healthcheck) on start, run, create and update, which changes
it on a running machine at once.--hostname, -u, -w, --cap-add, --cap-drop, --privileged,
--read-only, --tmpfs, --shm-size, --device, --device-cgroup-rule,
--gpus, --dns,
--dns-search, --add-host, --ulimit, --oom-score-adj, --stop-signal,
--stop-timeout, --timezone, --log-driver, --init, --sysctl,
--secret, --interface and --depends-on on start, run and create,
applied at the next start. A list takes none to forget it; --hostname,
-u, -w, --stop-signal, --timezone and --log-driver go back to the
default with an empty string, --shm-size with 0, and --privileged=false
and --read-only=false take those back.--rm on run: the machine is removed when that run ends.nspawn --help and nspawn COMMAND --help list the options; this page
explains them in full. Errors are printed as error: ... on standard error and
the exit status is 1 (2 for a command line clap refuses); exec and an attached
run exit with the status of the program they ran, and nspawn compose prints
its errors as they are, without error:.
Every command but daemon, completions and the unit hooks is a call to the
service on the system bus, which asks polkit
whether the caller may take the action: the listings (images ls, ps,
machines ls, inspect, images inspect, info, generate, top, stats,
events, wait, port, system df, builder du, network ls, network inspect,
volume ls, secret ls, secret inspect) ask for
org.nspawn.inspect; everything else, search and hub included, asks for
org.nspawn.manage. Both are for administrators by default, and root is never
asked.
These are accepted by every command and can also come from the environment;
nspawn compose ignores them, with a warning, as its pulls take the service’s
configuration:
| Option | Environment | Meaning |
|---|---|---|
--registry REGISTRY | NSPAWN_REGISTRY | Registry (hub) for image references without a host part. |
--ca-cert FILE | NSPAWN_CA_CERT | Extra CA certificate (PEM) to trust when talking to the registry. |
--config FILE | NSPAWN_CONFIG | Configuration file; see Configuration. |
-h, --help | Help. |
login and logout take the registry as their argument, so there --registry
has to come before the subcommand: nspawn --registry hub.example login -u me.
-V, --version belongs to nspawn itself, not to the subcommands.
Query the hub (an OCI registry).
nspawn hub ls [FILTER] [--no-tags]
Lists the repositories of the registry with their tags. FILTER keeps the
repositories whose name contains that text; --no-tags skips the tag query of
every repository, which is faster on big registries. hub list is an alias.
nspawn hub tags REPOSITORY
Prints the tags of one repository, one per line.
nspawn search TERM [--source hub|dockerhub] [-n LIMIT]
Finds images on the hub and on Docker Hub and prints
every hit with its source and the reference pull takes.
| Option | Meaning |
|---|---|
TERM | Text to look for in image names. |
--source hub|dockerhub | Only one source instead of both. |
-n, --limit LIMIT | Results per source. Default: 25. |
nspawn login [REGISTRY] [-u USERNAME] [--password-stdin]
Checks credentials against a registry (GET /v2/, then basic authentication
or a token from the realm the registry names) and keeps them in
/etc/nspawn/auth.json for pull, push, search and hub.
| Option | Meaning |
|---|---|
REGISTRY | Registry host, for example docker.io or hub.nspawn.org. Default: the hub. |
-u, --username USERNAME | User name; asked for on the terminal when missing. |
--password-stdin | Read the password from standard input instead of the terminal. |
nspawn logout [REGISTRY]
Forgets the credentials stored for a registry (the hub by default).
nspawn pull REFERENCE [-n NAME] [--backend BACKEND] [--mode MODE] [-f] [--no-verify]
Downloads an image from the hub or another registry, checking its signature first where the registry has a policy (the hub’s is built in; see Signed images), and makes it available to systemd-machined. On a terminal a bar shows how far each blob got, with its size, speed and time left; in a pipe or a log only the lines are written.
| Option | Meaning |
|---|---|
REFERENCE | [registry/]repository[:tag|@digest], for example fedora:44 or docker.io/library/nginx. |
-n, --name NAME | Local image name. Default: derived from the reference, for example fedora-44. |
--backend auto|overlay|flat|mstack | How to assemble the image on this host. Default: auto, or the backend of the configuration file: overlay, or flat without overlayfs. mstack (systemd 261 or newer, managed user namespaces) is experimental. App images are assembled as overlay even when mstack is chosen. |
--mode auto|boot|app | Whether the image boots an init system or runs a single program. Default: auto. |
-f, --force | Replace an existing image with the same name. |
--no-verify | Skip the signature check: the hub’s images are signed and refused without a valid signature, and the configuration can ask the same of other registries. |
nspawn create SOURCE NAME [--backend BACKEND] [--network NETWORK]... [--network-alias [NETWORK=]NAME]...
[-p [IP:][HOST:]CONTAINER[/udp]]... [-P] [--entrypoint PROGRAM] [-e VAR[=VALUE]]...
[-v SOURCE:TARGET[:ro]]... [-l KEY=VALUE]... [--restart POLICY] [-m SIZE] [--memory-swap SIZE]
[--cpus N] [--pids-limit N] [HEALTHCHECK OPTIONS] [OTHER OPTIONS] [--interface IFACE]... [--secret SECRET]...
[--depends-on NAME[:CONDITION][:optional]]... [--env-file FILE]... [--label-file FILE]...
[--mount type=TYPE,source=SRC,target=DST]... [--ip [NETWORK=]IP]... [--gpus REQUEST]... [-f] [--no-verify] [-- ARGUMENTS...]
Makes another machine from a local image without
touching the registry; a reference that is not local is pulled first, under
the image’s own name, as run does. The layers are shared with the source.
| Option | Meaning |
|---|---|
SOURCE | A local image name, used as it is. Otherwise a reference: the image of that reference that never ran, the last one pulled when there are several; a name without a registry is looked for under the configured one, then as Docker Hub’s. A reference that is not local is pulled. |
NAME | Name of the new machine. |
--backend BACKEND | How to assemble it. Default: like the source. |
--network NETWORK | Networks of the new machine, as for start. Default: the source’s kind; a network made with network create is not inherited, as ports and labels are not. |
--network-alias [NETWORK=]NAME | Other names of the machine on its networks, as for start. |
-p, --publish [IP:][HOST:]CONTAINER[/udp] | Ports to publish on the host, like start -p. Not inherited from the source. |
-P, --publish-all | Publish each port the image exposes, like start -P. |
--entrypoint PROGRAM | Replace the image’s entrypoint and drop its cmd, so only the arguments given follow it; an empty string runs the arguments alone. An app left with nothing to run is refused with no command specified. App images only. |
-e, --env VAR[=VALUE] | Environment for the program, VAR=value or VAR copied from the calling shell, left out when the shell does not have it. App images only. |
-v, --volume SOURCE:TARGET[:ro] | Mount a host directory or a named volume; z, Z, a propagation, nocopy and cached are taken and do nothing. |
-l, --label KEY=VALUE | Label the machine, on top of the image’s own labels. Not inherited from the source. |
--restart, -m, --memory-swap, --cpus, --pids-limit | Restart policy and limits, as for start. Not inherited from the source. |
the healthcheck options, the other options, --env-file, --label-file, --mount, --ip, --gpus, --interface, --secret, --depends-on | As for start. Not inherited from the source; a fixed address cannot be shared, so a copy gets one of its own. |
-f, --force | Replace an existing machine with the same name. |
--no-verify | Skip the signature check of an image that has to be pulled (a local source is not checked). |
-- ARGUMENTS... | App images: replace the image’s cmd; they follow its entrypoint. |
nspawn build [DIRECTORY|-] [-t TAG]... [-f DOCKERFILE] [--build-arg KEY[=VALUE]]... [--target STAGE]
[--no-cache] [--no-cache-filter STAGE]... [--pull] [--label KEY=VALUE]...
[--network MODE] [--add-host HOST:IP]... [--allow ENTITLEMENT]... [--secret id=ID[,src=PATH]]...
[--shm-size SIZE] [--ulimit NAME=SOFT[:HARD]]... [-q] [--progress TYPE] [--iidfile FILE]
[--push] [--no-verify] [--backend BACKEND] [--mode MODE] [--force]
nspawn build [DIRECTORY] -t TAG [-n NAME] [-d DISTRIBUTION] [-r RELEASE]
[--profile PROFILE]... [--backend BACKEND] [--mode MODE] [--force]
[--keep-output] [-- MKOSI_ARGS...]
Builds an image from a Dockerfile, or with mkosi from an
mkosi configuration, and makes it available locally, ready to push. A
directory with a Dockerfile (or dockerfile, Containerfile), -f, - or
any Dockerfile option makes it a Dockerfile build, run by nspawn’s own
builder; a directory with mkosi.conf or mkosi.conf.d and no Dockerfile, or
an mkosi option, an mkosi build, which needs mkosi. See
Building images.
| Option | Meaning |
|---|---|
DIRECTORY | The build context: a directory with a Dockerfile, or an mkosi configuration. - reads a context tar, or a Dockerfile with no context, from standard input. Default: .. |
-t, --tag TAG | Reference for the result, for example myapp:1 or hub.example/team/app:2. Repeatable for a Dockerfile; an mkosi build takes one, and needs it. A Dockerfile build without one fills the cache and prints the image ID. |
--force | Replace whatever has the image’s name, a machine that ran included. Without it, an image of the tag’s reference that nothing ran from gives way to the build; a machine, or an image of another reference, keeps the name, and the build goes beside it with the same reference (into the image an earlier build of the tag put there, or else a free name such as app-2), which a note says. |
--backend, --mode | As for pull. |
-f, --file PATH | The Dockerfile. Default: Dockerfile, dockerfile or Containerfile in the context; - reads it from standard input. |
--build-arg KEY[=VALUE] | A build argument; KEY alone takes it from the environment. Repeatable. |
--target STAGE | The stage to build. Default: the last one. |
--no-cache | Build every step again instead of taking it from the cache. |
--no-cache-filter STAGE | Build these stages again. Repeatable. |
--pull | Ask the registry for the base images again even when they are here. |
--label KEY=VALUE | A label for the image. Repeatable. |
--network MODE | The network of RUN steps: default (the bridge), none, or host (with --allow network.host). |
--add-host HOST:IP | A line for the hosts file of RUN steps; host-gateway is the host’s address on the bridge. Repeatable. |
--allow ENTITLEMENT | network.host lets a step use --network=host. Repeatable. |
--secret id=ID[,src=PATH|,env=VAR] | A secret for RUN --mount=type=secret, read by the command line as the caller; env= wins over a file, and with neither the variable named ID when it is set, else the file. Repeatable. |
--shm-size SIZE | Size of /dev/shm in RUN steps: 64m, 1g. |
--ulimit NAME=SOFT[:HARD] | A resource limit of RUN steps. Repeatable. |
--platform PLATFORM | The platform to build for; this host’s is the only one. |
-q, --quiet | Print only the image ID. |
--progress auto|plain|tty|quiet | How to show the steps; all print every line. |
--iidfile FILE | Write the image ID to this file. |
--push | Push the tags once built. |
--load | Taken: the image is local already. |
--no-verify | Pull the base images without verifying their signatures. |
-n, --name NAME | mkosi: local image name. Default: derived from the tag. A machine or an image of another reference that has it is refused without --force. |
-d, --distribution DISTRIBUTION | mkosi: distribution to build (mkosi --distribution). |
-r, --release RELEASE | mkosi: release to build (mkosi --release). |
--profile PROFILE | mkosi: profile to enable; repeatable. |
--keep-output | mkosi: keep the output directory instead of deleting it after the import. |
-- MKOSI_ARGS... | mkosi: extra arguments passed verbatim. |
--rm, --force-rm, -m, --memory-swap, -c, --cpu-period,
--cpu-quota, --cpuset-cpus, --cpuset-mems, --compress, --squash,
--isolation and --security-opt, which Makefiles still give, are taken and
ignored, three of them (--isolation, --security-opt, --squash) with a
warning; --cache-from is taken and ignored with a
note. --ssh, -o/--output, --cache-to, --build-context,
--check, --call and --attest are refused, and so are --sbom and
--provenance unless they ask for none (=false).
nspawn push IMAGE [--to REFERENCE]
Uploads a local image to the hub or another registry, with the credentials
stored for it. IMAGE is a local image name, or the reference it was pulled
from or built as. --to pushes it under a different reference than the one
recorded for the image. The destination needs a tag, not a digest. On a
terminal a bar shows how far each blob got, as with pull.
Manage local images; without a subcommand, list them. image is the same
group: nspawn image ls, image rm, image prune and image inspect.
nspawn images ls [--json]
nspawn images [--json]
Lists the local images known to systemd-machined: name, type, and for the ones
nspawn installed the backend, origin (pull, build or create), source
reference, size and whether the image is read-only. --json prints the list as
the service returns it. images list is an alias, and images without a
subcommand prints the same list.
nspawn images inspect [-f FORMAT] NAME|REFERENCE...
Prints the records of images as a JSON array, the keys of
inspect without the ones it adds for a machine (state, start
time, leader, health, published ports, exit code). A record’s name is that
record; anything else is a reference, and the image it means is the one the
service resolves it to, as run and create do: busybox:1.37 is the
configured registry’s busybox:1.37 when there is one here,
docker.io/library/busybox:1.37 otherwise; an image goes before a machine made
from it, and a machine is shown when it alone holds the image. One that is not
here is said as no local image of NAME after the others are printed, and the
command exits with 1. -f, --format FORMAT prints a Go template over each
record, as for inspect.
nspawn images prune [-a] [-f] [--filter KEY=VALUE]...
Removes images no machine uses: with -a every
image no machine was made from; without, the dangling ones, which nspawn’s
store has none of, since a tag pulled or built again replaces its record. Here
an image is a record that was pulled or built and never started. --filter takes
label=KEY, label=KEY=VALUE and until=DURATION or unix seconds,
repeatable; -f skips the question.
nspawn images rm NAME...
Removes local images and the layers and blobs nobody uses any more. Refuses
the image of a running machine, or of one its restart policy is bringing back,
and a name nothing is behind. Named volumes are kept. A pulled image is a machine too, so this is the same as
rm without --force.
nspawn rm [-f] NAME...
Removes machines: the record, the tree, the unit files, the
boot link a restart policy made, and the layers and blobs nobody else uses.
Every name is tried; one that cannot be removed, or that nothing is behind
(no machine or image named NAME), is reported at the end. Named volumes are
kept, and each one kept is mentioned.
A machine that others depend on (--depends-on) or take their network from
(--network container:NAME) is refused, unless they are named in the same
command: they are removed first and the machine after them; if one of them
cannot go (it runs and -f is not given), the machine stays too.
| Option | Meaning |
|---|---|
-f, --force | Stop a running (or restarting) machine first, with SIGKILL, instead of refusing it. |
nspawn ps [-a] [-q] [-f KEY=VALUE]... [--format FORMAT] [--no-trunc] [--json]
nspawn machines ls [-a] [-q] [-f KEY=VALUE]... [--format FORMAT] [--no-trunc] [--json]
Lists the running machines: name, image, mode, command,
state, uptime, leader PID, network and OS. A machine whose restart policy is
bringing it back shows as restarting, even without -a. -a, --all also
lists the nspawn machines that are not running: stopped, or starting and
closing while their unit comes up or goes down. --json prints what the service returns for
each machine, its whole record included. machines list is an alias of
machines ls.
| Option | Meaning |
|---|---|
-q, --quiet | The names alone, one per line. |
-f, --filter KEY=VALUE | Only the machines that match: name (a regular expression), label, status, network, ancestor, health, exited, volume, publish, expose, id, before, since. The same key twice matches either value, different keys all. Repeatable. |
--format FORMAT | A Go template over each machine’s fields (.Names, .Image, .Command, .CreatedAt, .RunningFor, .Ports, .State, .Status, .Labels, .Label "KEY", .Mounts, .Networks, .Mode, .Address, .OS, .Pid); table TEMPLATE for aligned columns with headers, json for an object per line. |
--no-trunc | Show .Command, .Mounts and .Image whole; otherwise the command is cut to 20 characters, each mount to 15, and the image’s digest is left out. |
nspawn inspect [-f FORMAT] NAME...
Prints everything nspawn knows about machines or images, running or not, as a
JSON array with one object per name: the record (image
reference, digest, backend, mode, networks, addresses, aliases, ports, volumes,
environment, command, labels, restart policy, limits, healthcheck, secrets,
the machines it depends on (depends_on), the other flags, and who signed the
image and when, as the pull verified it:
signed_by, signed_at), for a running machine its state, start time, leader
PID, OS and health, and for a stopped one the exit code of its last run. The keys are
those of the
D-Bus interface.
-f, --format FORMAT prints a Go template over each object instead, a
line each ({{.state}}, {{json .labels}}, {{index .addresses "front"}}),
with the functions json, join, split, lower, upper, title, pad
and truncate; an empty one prints the JSON.
nspawn info [-f FORMAT]
Prints the command’s version, then what the service answers: its version, its
machines by state (a record pulled or built that never ran is an image, not a
machine), its images, the registry, the default storage backend
(auto (overlay) says what auto picks on this host), the default log driver,
where machines and state are kept, systemd’s version and the host’s kernel, OS,
architecture, CPUs, memory, name and machine ID. Exits with 1 and the error when
the service does not answer, which is how a script checks that nspawn is
usable. -f json (--format json) prints the answer on one line, and
-f FORMAT a Go template over its keys ({{.version}},
{{.machines.running}}).
nspawn generate [NAME...]
Prints the nspawn commands that make machines again, on another host or on
this one, as a shell script: network create for the networks they join, then
run -d --pull always for each machine with every flag it keeps, after the
machines it depends on or takes its network from, which come along when they
are not named. An image without flags of its own is pulled again instead,
unless another machine takes its network. Without names it covers every
machine nspawn has. What a flag cannot carry is said in comments at the top:
secrets, encrypted for this host, which nspawn secret create makes again on
the other one; host directories a machine mounts; images built on this host,
which have to be pushed first; a healthcheck in exec form, which comes back as
--health-cmd. The contents of volumes and of a machine’s writable layer, and
the addresses the bridge handed out, stay behind; an address fixed with --ip
comes along. See
The same machines on another host.
| Option | Meaning |
|---|---|
NAME... | Machines to make again. Default: every machine of nspawn’s. |
nspawn start NAME [--network NETWORK]... [--network-alias [NETWORK=]NAME]... [-p [IP:][HOST:]CONTAINER[/udp]]... [-P]
[--entrypoint PROGRAM] [-e VAR[=VALUE]]... [-v SOURCE:TARGET[:ro]]... [-l KEY=VALUE]...
[--restart POLICY] [-m SIZE] [--memory-swap SIZE] [--cpus N] [--pids-limit N]
[--health-cmd COMMAND] [--health-interval DURATION] [--health-timeout DURATION] [--health-retries N]
[--health-start-period DURATION] [--health-start-interval DURATION] [--no-healthcheck]
[--hostname NAME] [-u USER[:GROUP]] [-w DIR] [--cap-add CAP]... [--cap-drop CAP]... [--privileged]
[--read-only] [--tmpfs PATH[:OPTIONS]]... [--shm-size SIZE]
[--device HOST[:CONTAINER[:rwm]]]... [--device-cgroup-rule RULE]... [--dns IP]... [--dns-search DOMAIN]...
[--add-host HOST:IP]... [--ulimit NAME=SOFT[:HARD]]... [--oom-score-adj N]
[--stop-signal SIGNAL] [--stop-timeout SECONDS] [--timezone MODE] [--log-driver DRIVER] [--init] [--sysctl KEY=VALUE]...
[--interface IFACE]... [--secret NAME[:TARGET[:MODE[:UID:GID]]]]...
[--depends-on NAME[:CONDITION][:optional]]... [--env-file FILE]... [--label-file FILE]...
[--mount type=TYPE,source=SRC,target=DST]... [--ip [NETWORK=]IP]... [--gpus REQUEST]... [--image-command] [--no-wait]
[-- ARGUMENTS...]
Boots an image as a machine. Every option is remembered for the next start.
| Option | Meaning |
|---|---|
--network NETWORK | Network of the machine: bridge, the default network (the default); the name of a network made with network create; veth, a veth pair configured by systemd-networkd on the host (booted images only); host, the host’s own network; none, no interface but lo; or container:NAME, the network namespace of that running machine (app images only; see container:NAME). Repeatable for several bridge networks, the first one primary: its address is where published ports lead and its gateway the default route, unless it is internal, in which case the first network that is not has the route. veth, host, none and container:NAME go alone. |
--network-alias [NETWORK=]NAME | Another name for the machine on its primary network, or on NETWORK: every member of that network resolves it. Repeatable; none forgets them. |
-p, --publish [IP:][HOST:]CONTAINER[/udp] | Publish a port on the host: on every IPv4 and IPv6 address of the host, or on IP alone (127.0.0.1:8080:80, [::1]:8080:80); 0.0.0.0 is every IPv4 address, [::] every IPv6 one. No host port (80, :80) or 0 takes a free one of the host, kept by the machine; a host range with one port of the machine (8000-8010:80) the first free one in it. 8000-8010:8000-8010 publishes a range, one mapping per port. Also published=HOST,target=CONTAINER[,protocol=udp][,host_ip=IP], target the only one required. Repeatable; none forgets them all. |
-P, --publish-all | Publish each TCP and UDP port the image exposes (EXPOSE) that no -p publishes to a free port of the host. --dns-option and --expose are taken with a warning that they are ignored. |
--ip [NETWORK=]IP | A fixed IPv4 address on the primary network, or on NETWORK: one made with network create, inside its subnet and nobody else’s there. Repeatable, once per network. |
--entrypoint PROGRAM | Replace the image’s entrypoint and drop its cmd, so only the arguments given follow it; an empty string runs the arguments alone. An app left with nothing to run is refused with no command specified. App images only. |
-e, --env VAR[=VALUE] | Environment for the program, VAR=value or VAR copied from the calling shell, left out when the shell does not have it, so -e DEBUG passes an optional setting on only where it is set. Repeatable; none forgets them, and variables that are all left out keep the remembered ones. App images only. |
--env-file FILE | Variables from a file of KEY=VALUE lines (no quoting, no expansion, # comments, a KEY alone copied from the calling shell or left out when it is not set there), before the -e that follow and win. Repeatable. |
-v, --volume SOURCE:TARGET[:ro] | Mount a host directory or a named volume (z, Z, a propagation, nocopy and cached are taken and do nothing), made on first use (or made ahead and still empty) with what the image has at TARGET, owner and files included. Repeatable; none forgets them. |
--mount type=bind|volume|tmpfs,source=...,target=... | The long form of -v and --tmpfs: readonly, tmpfs-size, tmpfs-mode; a volume needs its name, and image, cluster or npipe mounts, volume-subpath and driver options are refused. Repeatable. |
-l, --label KEY=VALUE | Label the machine, on top of the image’s own labels. Repeatable; none forgets them. |
--label-file FILE | Labels from a file of KEY=VALUE lines, before the -l that follow and win. Repeatable. |
--restart no|on-failure[:N]|always|unless-stopped | Restart policy; the count of on-failure is taken and not kept. always and unless-stopped also start the machine at boot; nspawn stop takes an unless-stopped machine off the boot list until the next start. Applied at the next start; update changes it at once. |
-m, --memory SIZE | Memory limit of the whole machine: 512m, 2g (at least 4m), and as much swap again unless --memory-swap says otherwise; 0 removes it. Applied at the next start; update changes it at once. |
--memory-swap SIZE | Memory and swap together: equal to --memory for no swap, larger for the difference, -1 for swap without a bound; 0 goes back to as much swap again as memory. Needs --memory. Applied at the next start; update changes it at once. |
--cpus N | CPU limit of the whole machine: 0.5, 2; 0 removes it. Applied at the next start; update changes it at once. |
--pids-limit N | Most processes and threads the machine may have (at least 16 for a booted machine); 0 or -1 removes the limit. Applied at the next start; update changes it at once. |
--health-cmd COMMAND | Command that says whether the machine is healthy, run inside it as its user through /bin/sh -c at every interval: exit 0 is healthy. Replaces the image’s HEALTHCHECK. update changes it at once. |
--health-interval DURATION | Time between probes: 10s, 1m30s, 500ms. Default: 30s. |
--health-timeout DURATION | Time a probe may take before it counts as failed. Default: 30s. |
--health-retries N | Consecutive failed probes that make the machine unhealthy. Default: 3. |
--health-start-period DURATION | Time after the start during which failed probes do not count. Default: 0s. |
--health-start-interval DURATION | Time between probes during the start period. Default: 5s. |
--no-healthcheck | No probes, whatever the image says. |
--hostname NAME | Hostname inside the machine. Default: its name. A booted machine gets it as its /etc/hostname. |
-u, --user USER[:GROUP] | User the program runs as, a name or a uid (listed in the image’s passwd or not), instead of the image’s, with a group after a colon: a name of the image’s group file or a number, which becomes the primary and only group of the program; a name the image lacks is refused. nspawn resolves both from the image’s passwd and group files through a stand-in for getent; a uid the passwd lacks runs with group 0 and / as home. exec takes it too unless its own -u says otherwise. App images only. |
-w, --workdir DIR | Working directory of the program, instead of the image’s. App images only. |
--cap-add CAP | Capability to keep on top of the default set (for an app on a bridge network, the set App machines lists, SYS_BOOT included; systemd-nspawn’s for a machine in a user namespace): NET_ADMIN, CAP_NET_ADMIN, ALL. Repeatable; none forgets them. |
--cap-drop CAP | Capability to drop from the default set. --cap-drop ALL --cap-add X keeps X. Repeatable; none forgets them. |
--privileged | Every capability; --privileged=false takes it back. |
--read-only | Mount the machine’s root read-only; --read-only=false takes it back. |
--tmpfs PATH[:OPTIONS] | An empty tmpfs at a path inside (/tmp:size=64m,mode=0700), 1777 when given without options. One on or below /run, a tmpfs of every machine already, is left out with a note. Repeatable; none forgets them. |
--shm-size SIZE | Size of /dev/shm: 64m, 1g; 0 for the default. |
--device HOST[:CONTAINER[:rwm]] | A device node of the host for the machine (/dev/ttyUSB0:/dev/ttyUSB0:rw), or a directory whose nodes are allowed one by one (/dev/dri), or a CDI name (nvidia.com/gpu=all) for the devices of the host’s specs. An app given devices of the host runs without a user namespace. Repeatable; none forgets them. |
--device-cgroup-rule RULE | A rule of the machine’s device cgroup, TYPE MAJOR:MINOR PERMISSIONS (c, b or a for both; a number or *; r, w, m): every node of a major (c 189:* rwm), those that appear while it runs included, by the device groups the host’s /proc/devices names after it at each start, refused then for a major without one; or one node (c 189:3 rw). For a bind of the nodes’ directory under /dev. Repeatable; none forgets them. |
--gpus REQUEST | GPUs of the host’s CDI specs: all, a count, or '"device=0,1"', for the NVIDIA spec’s devices (the AMD one when only that is there); driver= says which. See GPUs and CDI devices. Repeatable; none forgets them. |
--dns IP | DNS server for the machine, instead of the host’s. Repeatable; none forgets them. |
--dns-search DOMAIN | DNS search domain, in an app’s resolv.conf and a booted machine’s .network file; 254 characters at most, with a letter among its labels. A trailing dot is dropped and . stands for none. Repeatable; none forgets them. |
--add-host HOST:IP | A line for the machine’s /etc/hosts, HOST:IP or HOST=IP (the second for an IPv6 address), the IP in brackets or not; the name is anything without colons or whitespace. host-gateway is the host’s address on the machine’s network. Repeatable; none forgets them. |
--ulimit NAME=SOFT[:HARD] | A resource limit of the program (nofile=1024:4096, core=unlimited). Repeatable; none forgets them. |
--oom-score-adj N | OOM score adjustment of the machine, -1000 to 1000. |
--stop-signal SIGNAL | Signal stop sends the program, instead of the image’s (SIGTERM). |
--stop-timeout SECONDS | Seconds stop waits after the signal before SIGKILL, unless -t says otherwise; -1 waits for as long as the program takes. Default: 10. |
--timezone MODE | How systemd-nspawn sets the machine’s /etc/localtime at each start, its Timezone=: auto (the default, the host’s zone: a link where the machine’s zone database has it, otherwise a copy of the host’s file, which a package installed inside may replace; a read-only root gets the host’s file mounted over it), off (left alone, so a zone set inside stays), copy, bind, symlink or delete. auto forgets the setting. |
--log-driver DRIVER | Where the program’s output goes: local (the default: journald’s nspawn namespace, a journal of nspawn’s own apart from the system’s, capped at 1 GiB, which logs and an attached run read), journal (the system’s journal) or none (dropped; logs refuses the machine, an attached run of an app still shows the output straight from the program, and a booted machine runs with -d). |
--init, --init=false | The program under systemd-nspawn’s stub init, which reaps orphans and turns a poweroff into SIGTERM, or as PID 1 of the machine. Without either, PID 1 for an image that carries s6-overlay, the stub init otherwise. App images only. |
--sysctl KEY=VALUE | A net.* sysctl for an app machine’s network namespace, after the two it gets by default, net.ipv4.ping_group_range=0 2147483647 and net.ipv4.ip_unprivileged_port_start=0, so that a user other than root can ping without raw sockets and bind a port below 1024. Repeatable; none forgets them. |
--interface IFACE | A network interface of the host, moved into the machine while it runs and given back when it stops: an ethernet one, or a wifi adapter with its whole phy (iw on the host for an app on the bridge, systemd 256 for a booted machine); the name is kept inside. Not with --network host or container:NAME; one machine at a time. See Physical interfaces. Repeatable; none forgets them. |
--secret NAME[:TARGET[:MODE[:UID:GID]]] | A secret made with secret create as a read-only file inside the machine: NAME alone is /run/secrets/NAME with mode 0444, root’s. Repeatable; none forgets them. Not on mstack machines. |
--depends-on NAME[:CONDITION][:optional] | Another machine of nspawn’s to start first, like compose’s depends_on, and what to wait for: started (the default: its start went through), healthy (its healthcheck says healthy; it needs one) or completed (its program ended with 0; it runs until one run of it succeeds, a run of an earlier boot counting by the exit code the journal kept, and is ordered before the machine, not wanted by it). With :optional a failure of it is a warning and the machine starts anyway. systemd keeps the order, at boot and for a restart policy as well. See Dependencies. Repeatable; none forgets them. |
--image-command | Forget the remembered entrypoint and arguments and run the image’s own again. |
--no-wait | Do not wait for a booted machine’s init to be up before returning. Its registration is still awaited, so that ports and firewall rules can be applied. |
-- ARGUMENTS... | App images: replace the image’s cmd; they follow its entrypoint. |
A path inside the machine, the target of -v or --secret, a --tmpfs or a
--device path, must be plain: a . or .. component is refused.
nspawn run [-d] [--rm] [-i] [-t] [-n NAME] [--pull missing|always|never]
[--backend BACKEND] [--mode MODE] [-f] [--no-verify] [OPTIONS OF start] [--no-wait]
REFERENCE [COMMAND [ARGUMENT...]]
Makes a machine from an image and starts it: pull (or
create from a local image with the same reference) followed by start, whose
options are remembered as after start. A name without a registry is looked
for here under the configured registry, then as Docker Hub’s
(docker.io/library/NAME), and pulled from the configured registry only when
neither is there. What pull or create report goes to
standard error, so that standard output carries the program’s output alone.
Without -d, run stays attached. The machine’s output
follows until the machine ends, a line at a time, stdout and stderr together:
it is read from nspawn’s journal, so nspawn logs NAME shows it later too, and
the machine goes on should run be interrupted. With --log-driver none it
comes straight from the program instead, and nothing is kept. run exits with the program’s
exit code, or 128 plus the signal nspawn sent it (130 after Ctrl-C, 137 after
kill); a signal from elsewhere (a crash, a kill from inside, Ctrl-C with -t)
gives 255, or 1 for a program that is PID 1, as systemd-nspawn ends then. Ctrl-C, SIGTERM, SIGHUP and SIGQUIT are passed on to the program; a
third Ctrl-C within a second leaves the machine running and returns. A booted
image shows its console until it powers off, and Ctrl-C powers it off.
| Option | Meaning |
|---|---|
REFERENCE | [registry/]repository[:tag|@digest], for example nginx:1.27. |
COMMAND [ARGUMENT...] | App images: replace the image’s cmd and follow its entrypoint; everything after the reference that is not an option of run, or everything after --, which a command that starts with a dash needs. |
-d, --detach | Start the machine in the background and return; nspawn logs -f NAME follows its output. |
--rm | Remove the machine once it ends, with or without -d; named volumes stay. An image the run had to pull is kept under the image’s local name when that name is free, and without --name the machine gets a name of its own (alpine-3-1f0c9a2e). Refused with a restart policy. A run that fails to start leaves nothing behind. |
-i, --interactive | App images: give the program this standard input. Not with -d. |
-t, --tty | App images: give the program a terminal; Ctrl-C and resizes reach it through the terminal. Closing the terminal stops the machine. With -i on a booted image: wait for it to boot, open a root shell, and power it off when the shell ends, with the shell’s exit code. Not with -d. |
-n, --name NAME | Name of the machine. Default: derived from the reference, for example nginx-1.27. A name that is taken is refused, with a pointer to start. |
--pull missing|always|never | When to ask the registry. missing, the default: a local image with the same reference is made into the machine, as create does, and the registry is asked only when there is none. always: every time, for the image the tag names now. never: a local image or nothing. |
--backend auto|overlay|flat|mstack | How to assemble the machine, as for pull. |
--mode auto|boot|app | As for pull; a mode other than auto always pulls. |
-f, --force | Make the machine anew when one of that name exists; it must be stopped. |
--no-verify | Skip the signature check of the image, as for pull. |
the options of start | --network, --network-alias, --ip, -p, --entrypoint, -e, --env-file, -v, --mount, -l, --label-file, --restart, -m, --memory-swap, --cpus, --pids-limit, the healthcheck options, the other options, --gpus, --interface, --secret, --depends-on and --no-wait (with -d only), as for start. Options may come before or after the reference, as long as they come before the command. |
nspawn stop NAME... [-f] [-t SECONDS] [--no-wait]
Powers off running machines, one after the other: the machine’s stop signal
(--stop-signal, else the image’s) to the program of an app, a poweroff
request to a booted machine. Stopping a machine that already ended only cleans
up after it, and a paused one is thawed first. A machine with a restart policy
stays stopped, and one that was waiting to be restarted is stopped too. An
unless-stopped machine is also taken off the boot list until the next
start; an always one stays enabled and starts again at the next boot (rm
takes that away). Every name is tried; one that could not be stopped is
reported at the end.
| Option | Meaning |
|---|---|
-f, --force | Kill every process at once instead of asking the machine to stop. |
-t, --timeout SECONDS | App images: seconds to wait after the stop signal before killing the machine; a negative number waits for as long as the program takes. Default: the machine’s --stop-timeout, else 10. |
--no-wait | Return right after the stop request, without waiting for the machine to be gone and without the kill after the timeout. |
nspawn restart NAME... [-t SECONDS]
Stops and starts machines: a stop followed by a start with everything the machine remembers, and prints each name once it runs again. A machine that is not running is started. Every name is tried; one that could not be restarted is reported at the end.
| Option | Meaning |
|---|---|
-t, --timeout SECONDS | As for stop. |
nspawn kill [-s SIGNAL] NAME...
Sends a signal to running machines and prints each name
once it is sent. SIGKILL, the default, stops the machine for good, as
stop --force does. Any other signal goes to the program of an app or the
init of a booted machine and counts as a stop that keeps the
restart policy from bringing the machine back, unless the machine has a stop
signal of its own (its image’s STOPSIGNAL or --stop-signal) and this is
another one: then the policy decides, should the signal end it. For a booted
machine, SIGRTMIN+3 and SIGRTMIN+4 (halt and poweroff) count as a stop. Right
after a start it waits a few seconds at most for the program to be running.
| Option | Meaning |
|---|---|
-s, --signal SIGNAL | A name (KILL, SIGHUP, RTMIN+3) or a number. Default: KILL. |
nspawn pause NAME...
nspawn unpause NAME...
pause freezes every process of running machines through the cgroup freezer
of their units, and unpause thaws them; each name is
printed once done. ps and inspect show a frozen machine as paused, and
stop and kill thaw it first.
nspawn update NAME... [--restart POLICY] [-m SIZE] [--memory-swap SIZE] [--cpus N] [--pids-limit N]
[--health-cmd COMMAND] [--health-interval DURATION] [--health-timeout DURATION] [--health-retries N]
[--health-start-period DURATION] [--health-start-interval DURATION] [--no-healthcheck]
Changes the restart policy, the limits and the healthcheck of machines, and prints each name once changed. A running machine gets the new limits and healthcheck at once (its probes start afresh) and the policy for its next ending; a stopped one at its next start. The options are those of start; at least one is needed.
nspawn exec MACHINE [-u USER] [-e VAR[=VALUE]]... [-w DIR] [-T] [-t] [-i] [-d] COMMAND...
Runs a command inside a running machine of either kind, attached to the
terminal when standard input and output are one, in the machine’s namespaces,
with the image’s environment and the machine’s -e variables. The program is found on the machine’s PATH and
runs with the machine’s capabilities and resource limits, like its own
processes, except CAP_SYS_BOOT in an app that shares the host’s user namespace
(unless --privileged), since the command does not carry the seccomp filter that
keeps kexec out of the machine; right after a start it waits, a few seconds at most, until
systemd-nspawn has finished confining the machine. Exits with the command’s
status. A paused machine is refused at once.
| Option | Meaning |
|---|---|
-u, --user USER[:GROUP] | User inside the machine, a name or a number of its passwd file, with a group of its group file after a colon (then the only group; otherwise the user’s supplementary groups come along) and the home of the passwd entry. Default: the machine’s user, its --user or else its image’s USER; root on a booted machine. |
-e, --env VAR[=VALUE] | A variable for the command, VAR=value or VAR copied from the calling shell, left out when the shell does not have it. Repeatable. |
-w, --workdir DIR | Working directory of the command, instead of the machine’s. |
-T, --no-tty | No terminal, even from one: pipes, as in a script. Without -t or -T, a terminal only when standard input and output are one, so redirected output is byte-exact. |
-t, --tty | A terminal for the command, even without one here. |
-i, --interactive | Accepted and ignored: the command’s input is always this one. |
-d, --detach | Leave the command running in the background and return at once. |
COMMAND... | Command and arguments. |
nspawn shell MACHINE [-u USER]
Opens an interactive shell inside a running machine as USER (default
root): machined’s login session for booted machines, /bin/sh in the
machine’s namespaces for app machines.
nspawn top MACHINE
Lists the processes of a running machine: PID as the
host sees it, USER as the machine sees it, TIME of CPU and COMMAND,
read from the machine’s cgroup and PID namespace.
nspawn wait NAME...
Waits for machines to stop and prints each one’s exit code, a line per name; a machine already stopped gives the code of its last run.
nspawn port NAME [PRIVATE_PORT[/PROTO]]
Prints the ports a running machine publishes
(80/tcp -> 0.0.0.0:8080, and 80/tcp -> [::]:8080 on a host with IPv6), or
only those of one port of the machine; one it does not publish is an error.
nspawn cp MACHINE:PATH DESTINATION
nspawn cp SOURCE MACHINE:PATH
Copies files and directories between the host and a machine. The machine may be running, or stopped if it is an overlay or flat machine; a stopped mstack machine has no tree on the host until it runs.
/ must be a directory.DIR/. as the source copies the contents of DIR instead of DIR itself.cp. Modes and
modification times are kept./proc, /sys and the like).MACHINE: starts at the machine’s root. A local path
with a colon is written ./a:b.nspawn logs MACHINE... [-f] [-n N] [--since WHEN] [--until WHEN] [-t] [--all] [--inside]
Shows what machines printed. With several machines every
line carries its machine’s name (web | ...). The lines come from nspawn’s
journal namespace (--log-driver local, the
default) and from the system’s journal, which holds systemd’s lines and what a
machine with --log-driver journal writes. A machine started with --log-driver none
keeps nothing, and logs says so (a booted one’s own journal is still there
with --inside).
| Option | Meaning |
|---|---|
-f, --follow | Keep printing new output; starts from the last 10 lines unless --lines says otherwise. |
-n, --lines N, --tail N | Only the last N lines, or all of them with all. |
--since WHEN | Only output newer than this: a duration ago (42m, 1h30m), unix seconds, an RFC 3339 time or date, or a time as journalctl reads it (-1h, "10 min ago", "2026-09-24 10:00"). |
--until WHEN | Only output older than this, in the forms --since takes; nothing is followed then. |
-t, --timestamps | Prefix every line with its timestamp. |
--all | Also show what systemd says about the machine’s service: start, stop, failures. |
--inside | Booted machines only: read the machine’s own journal instead of its console output. |
nspawn stats [NAME...] [--no-stream] [--json]
Shows what running machines use: CPU (100% is one CPU busy), memory in use against the limit (the host’s memory without one), network and disk traffic, and processes, read from the cgroup of each machine’s unit, which holds the whole machine. Without names it shows every running machine. The table is drawn again every second on a terminal.
| Option | Meaning |
|---|---|
--no-stream | Print one table, from two readings a second apart, and return. |
--json | One JSON object per machine and reading instead of the table. |
nspawn auto-update [--dry-run] [--json] [NAME...]
Moves the machines labelled org.nspawn.auto-update=registry (or
io.containers.autoupdate=registry), all of them or the ones named, to the
image their tag points to now: each is made anew with
every option it had, and put back on the old image when it does not come up
(healthy, with a healthcheck). Prints a table, UNIT CONTAINER IMAGE POLICY UPDATED, with false, pending, true, failed, rolled back or
skipped, and fails when a machine failed or went back. Built images,
references pinned by digest, run --rm machines, booted, flat and mstack
machines, and machines whose image declares a volume none of their own volumes
covers are skipped; the reason goes to standard error for a machine named on
the command line, and is in --json always. See
auto-update.
| Option | Meaning |
|---|---|
--dry-run | Only say which machines have a newer image waiting. |
--json | One JSON object per machine. |
nspawn events [--since WHEN] [--until WHEN] [-f KEY=VALUE]... [--json]
Reports what happens to machines, networks, volumes and secrets:
for machines start, die (with its exit code), stop,
restart, oom and fail, which systemd logs for every machine however it
was started, health_status (with the new status) and what nspawn does,
pull, build, create, push, kill, update, pause, unpause,
auto-update and remove; for networks create, remove, connect and
disconnect; for volumes and secrets create and remove. It reads the journal, so
past events can be read back.
| Option | Meaning |
|---|---|
--since WHEN | Events since this time: a duration ago (42m, 1h30m), unix seconds, an RFC 3339 time or date, or a time as journalctl reads it ("2026-09-24 10:00", -1h, today). Without it, only new ones. |
--until WHEN | Stop at this time instead of waiting for new events, in the forms --since takes; needs --since. |
-f, --filter KEY=VALUE | Only matching events: name=NAME, type=machine|network|volume|secret, event=ACTION, label=KEY or label=KEY=VALUE. The same key given twice matches either value, different keys must all match. |
--json | One JSON object per event: time, time_usec, type, action, name, attributes, labels. |
The default network (bridge) and the ones made with network create.
nspawn network up
Brings up every network’s bridge with its NAT rules and firewall exceptions,
and brings the published ports in line with the machines that run. start
does it for its machine’s network too; this is useful at boot and for
troubleshooting.
nspawn network ls [--json]
Lists the networks, the default one first: name, bridge interface, subnet,
whether it is internal, and the machines whose records name it. --json
prints them as the service returns them. network list is an alias.
nspawn network create NAME [--subnet CIDR] [--internal] [--label KEY=VALUE]...
Makes a network of its own: a bridge nsbr-NAME
(hashed when the name is long) with its NAT and firewall rules, brought up at
once. Machines of one network reach each other by name, and nothing of another
network, the default one included, reaches them; ports they publish are
reachable from everywhere through the host, as from the LAN.
| Option | Meaning |
|---|---|
NAME | 1 to 64 letters, digits, _ and -, not starting with -; not bridge, host, veth, none or default. |
--subnet CIDR | The network’s IPv4 subnet. Default: the next free /24 of network_pool (see Configuration), never one the host already routes. |
--internal | No way out: its machines reach each other and the host, nothing beyond, and cannot publish ports. |
--label KEY=VALUE | Label the network. Repeatable. |
nspawn network inspect NAME...
Prints networks as a JSON array: name,
interface, subnet, gateway, whether it is internal, its labels, and its
machines with their addresses, aliases, ports and whether they run. bridge
is the default network.
nspawn network rm NAME...
Removes networks no machine uses, with their bridges and rules. A network a
machine’s record names is refused, with the machines that use it; the default
network cannot be removed. network remove is an alias.
nspawn network connect [--alias NAME]... [--ip ADDRESS] NETWORK MACHINE
Joins a machine to a network: a running app or
booted machine gets an interface on it at once (hostN, with its address; the
default route stays, unless the machine had none and the network leads out),
and from its next start it joins the network as if --network had named it. --alias gives it another name there, repeatable; --ip a fixed
address, on a network made with network create. Not for a machine on host
networking, veth, another’s network or none, nor an mstack one.
nspawn network disconnect [-f] NETWORK MACHINE
Takes a machine off a network: a running one
loses the interface at once. Leaving the primary network makes the next one
primary; the only network a machine has cannot be left. -f makes it no error
when the machine or its membership is not there.
nspawn network prune [-f] [--filter FILTER]...
Removes every network no machine uses, after asking on standard input unless
-f is given. --filter label=KEY[=VALUE] or until=TIME (a duration ago, or
unix seconds) keeps it to the networks it selects; repeatable, every one must
hold.
Named volumes, the directories -v NAME:/path makes under
/var/lib/nspawn/volumes. Removing a machine keeps them.
nspawn volume ls [--json] [--filter label=KEY[=VALUE]]...
Lists the named volumes with the machines whose records mount them, when each
was made and its path. --filter label=KEY or label=KEY=VALUE lists only
those with the label, every filter holding. --json prints the list as the
service returns it. volume list is an alias.
nspawn volume create NAME [--label KEY=VALUE]...
Makes a volume ahead of its first use (start makes it otherwise): a
directory owned by root, mode 0755. One that exists already is fine, and keeps
its own labels. Names have letters, digits, _, . and -, not starting with
a dot. --label labels it; labels and the creation time are kept beside the
volume’s directory, never inside it.
nspawn volume inspect NAME...
What nspawn keeps about the volumes, as JSON: their path, the machines that mount them, when each was made and their labels.
nspawn volume rm NAME...
Removes volumes no machine uses. A volume a machine still names is refused
until that machine is started with other volumes (or -v none) or removed.
nspawn volume prune [-f]
Removes every volume no machine uses once a yes comes on standard input:
from a script, whose input ends without one, nothing is removed.
-f removes them without asking.
Secrets, kept encrypted on the host with
systemd-creds, handed to machines as files by --secret on start, run
and create.
nspawn secret ls [--json]
Lists the secrets: name, when it was made, the size of its plaintext, its
labels and the machines that take it; never the content. --json prints the
list as the service returns it. secret list is an alias.
nspawn secret create NAME [-f FILE] [-l KEY=VALUE]...
Keeps a secret read from standard input, or from -f/--file, encrypted for this
host: bound to its TPM2 where there is one, to its credential key otherwise,
under /var/lib/nspawn/secrets. A name that exists already is refused.
| Option | Meaning |
|---|---|
NAME | Letters, digits, _, . and -, at most 64, not starting with a dot. |
-f, --file FILE | Read the content from this file instead of standard input. |
-l, --label KEY=VALUE | Label the secret. Repeatable. |
nspawn secret inspect NAME...
Prints what nspawn keeps about secrets as a JSON array: name, creation time, size, labels and the machines that take them; never the content.
nspawn secret rm NAME...
Removes secrets no machine takes. One a machine’s record names is refused,
with the machines that take it, until they are started with other secrets (or
--secret none) or removed. Every name is tried; one that cannot be removed is
reported at the end. secret remove is an alias.
nspawn system df [-v] [--json]
What images, machines, volumes and the build cache take on disk, and what a
prune would give back. -v lists each of them with
its size, --json prints the service’s answer.
nspawn system prune [-a] [--volumes] [-f] [--filter KEY=VALUE]...
Removes stopped machines made from an image (by create or run), networks
no machine uses and the build cache no image shares, after asking
unless -f is given; -a also removes
every image no machine was made from and the whole build cache. A label=
filter leaves the build cache alone.
--volumes also removes the volumes no machine uses that were made without a
name, which only the Docker API makes, and counts what they held in the space
reclaimed; it takes no until= filter. --filter takes label=KEY, label=KEY=VALUE and
until=DURATION or unix seconds.
nspawn builder du [-v] [--json]
What the build cache holds, most recently used first: the total and what is
reclaimable, -v every row.
nspawn builder prune [-a] [-f] [--filter until=DURATION]... [--keep-storage SIZE]
Removes build cache: without -a the cache
entries no image shares, with -a all of them. --filter until= keeps what was used since, --keep-storage SIZE
keeps the most recently used cache up to that size (64m, 2g), and -f
skips the question.
nspawn compose [OPTIONS] COMMAND [ARGS...]
Runs docker compose files on nspawn’s machines, with the commands config,
up, down, create, start, stop, restart, kill, pause, unpause,
rm, run, exec, ps, ls, logs, top, port, images, pull,
build, scale, wait, cp, stats, events, volumes and version,
each with docker compose’s flags. nspawn compose COMMAND --help prints each one’s options. See Compose.
nspawn daemon [--install] [--idle-exit SECONDS]
Serves org.nspawn on the system bus. The bus starts it on demand, so this is
not a command to type; --install is.
| Option | Meaning |
|---|---|
--install | Write the bus policy, the polkit actions, the activation file and the unit that let the bus start this binary on demand, the units of the Docker API sockets, a machine’s and the host’s (the host’s not enabled), and of auto-update (its timer disabled), and the journald configuration of nspawn’s log namespace, then return. A package installs the same files. |
--idle-exit SECONDS | Exit after this long without a call or a running job. Default: 60; 0 keeps serving. |
nspawn completions bash|elvish|fish|powershell|zsh
Writes the completions for that shell on standard output. Commands and flags come from the same definition the command line itself is built from; besides them, TAB completes names, asked from the service as it goes:
| After | TAB offers |
|---|---|
stop, exec, kill, pause, unpause, top, shell, stats, port | the running machines |
start | the images that do not run |
rm, inspect, generate, logs, restart, update, wait, auto-update, images rm, --depends-on, the machine of network connect and network disconnect | every machine and image |
create, push | the local images |
run | the references of the local images |
--network | the networks, and host, none and veth |
network rm, network inspect, network connect, network disconnect | the networks |
volume rm | the volumes |
The service hands those names to any user without asking polkit, so this works
under sudo and without a password; secrets are not completed. The flags come
once the word starts with - (- brings the short forms, -- the long ones),
or where there is no name to offer. The packages install the completions, so
this is for a binary you built yourself:
nspawn completions bash > ~/.local/share/bash-completion/completions/nspawn
man nspawn is generated from the same definitions as the completions, with
the --help texts.
nspawn network prepare NAME, nspawn network publish NAME and
nspawn network release NAME are what the drop-in of
systemd-nspawn@NAME.service runs as ExecStartPre, ExecStartPost and
ExecStopPost, preceded by nspawn await-dependencies NAME for a machine
with --depends-on, and nspawn health-run NAME is what
nspawn-health-NAME.service runs to probe a machine’s healthcheck. nspawn attach-exec NAME -- ... is the drop-in’s ExecStart=, nspawn api-serve NAME
is what nspawn-api@NAME.service runs and nspawn api-serve what
nspawn-api.service runs, and nspawn remove-after-exit removes a
run --rm machine. They are not meant to be typed and are hidden from --help.
A hub for systemd-nspawn images and the tool that uses them. The team builds
images of common Linux distributions with mkosi,
publishes them on an OCI registry at hub.nspawn.org, and maintains nspawn,
the command line tool documented on this site. The
mkosi definitions of the images
are public.
Yes, version 255 or newer. nspawn drives systemd-nspawn through systemd-machined and the systemd service manager, so the host needs both, with cgroup v2. The images on the hub contain systemd as well; images without an init system run as apps.
No. nspawn’s machines are systemd-nspawn containers managed by systemd: they are
systemd-nspawn@NAME.service units,
appear in machinectl list, log to journald and boot a full init when the
image has one. nspawn’s own service holds none of them open. docker compose
files run with nspawn compose, Dockerfiles build with nspawn build, and the
Docker Engine API, on a socket of the host and in machines that mount
/var/run/docker.sock, serves the docker command, CI runners, reverse proxies
and dashboards, but there is no orchestration across hosts.
Yes. sudo nspawn search nginx finds them, and
sudo nspawn pull docker.io/library/nginx:latest --name web fetches one like
any other registry image. Since it has no init system, it is installed as an
app image: its entrypoint runs under systemd-nspawn’s stub init, on the bridge,
with -p for its ports. An image that carries s6-overlay, as linuxserver.io’s
do, runs its /init as PID 1 instead. Docker Hub limits anonymous pulls per
address; sudo nspawn login docker.io -u USER raises the limit. See
Getting started.
Not necessarily. Every command goes through the service on the system bus, and
the service asks polkit about the caller: root is never asked, an administrator
is asked for a password (a desktop session prompts; in a plain terminal, start
pkttyagent first), and a rule of your own can hand the actions to a group so
that nobody is asked at all. The work does need privileges, since the machines
live below /var/lib/machines and the bridge is the host’s, so the service,
which runs as root, does it.
On a host without polkit the service answers root alone. See The service for the two actions and the rule.
Where the work happens. The command line is its client, the way machinectl
and systemctl are clients of machined and systemd: every command is a method
call, and pulls, pushes, builds, creates, removals and copies come back as job
objects that report their output. The bus starts the service when a command
arrives and it exits after a minute without work. Between commands only a
machine’s healthcheck, the Docker API of a machine that asked for one and the
auto-update timer, once enabled, run. A package installs it; for a binary you
built yourself,
sudo nspawn daemon --install. Its journal is journalctl -u nspawn.service,
and the interface is described in docs/DBUS.md of the repository. After an
upgrade the service keeps running the binary it started with until it goes
idle or is restarted; a command of another version says so and asks for
sudo systemctl restart nspawn.service (the rpm and the deb restart it
themselves).
Yes. The workflow that builds the images of the hub, in
nspawn/mkosi-definitions, signs
each one twice with cosign after pushing
it: keyless, with the identity of that workflow on master (a short-lived
certificate from Fulcio, recorded in the Rekor transparency log), and with the
project’s key, whose public half is cosign.pub in that repository. The
signatures are stored on the hub next to the image, as OCI referrers of its
digest, so they cover every tag that points to it, and the hub itself verifies
the key signature and shows the image as signed.
nspawn pull verifies them too: before a single layer is
downloaded, one of the two signatures has to verify offline against the
project’s key and the workflow’s identity, both built into nspawn, and an
image without a valid signature is refused (--no-verify skips the check for
one command; see Signed images). Other
registries get a policy in the configuration.
To verify an image yourself, with cosign:
cosign verify \
--certificate-identity https://github.com/nspawn/mkosi-definitions/.github/workflows/mkosi.yml@refs/heads/master \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
hub.nspawn.org/fedora:44
or, with the key from the repository, cosign verify --key cosign.pub hub.nspawn.org/fedora:44. Every blob is also checked against the sha256
digest in the manifest while it downloads, and registries are reached over
HTTPS only.
The first generation of nspawn was a shell wrapper around machinectl that
downloaded tar.xz and raw.xz images from hub.nspawn.org/storage, signed
with the nspawn.org master key and verified by systemd-importd. Its last
release was 0.6; version 1.0 replaced it with the OCI based client described
here: images are OCI images on
a registry, layers are shared, and machinectl is no longer involved. The
history of the old script is still in the
nspawn repository.
Yes. nspawn creates the bridge, its addresses and its nftables rules itself,
and hands each machine its address through a generated file or a prepared
network namespace. Neither systemd-networkd nor NetworkManager on the host is
involved; only --network veth depends on systemd-networkd. With firewalld the
bridge is put in the trusted zone, and on hosts where docker or ufw sets the
forward policy to drop, or is installed and may set it later, nspawn adds the
exception the bridge needs. See
Firewalls.
exec enters the machine’s namespaces: nothing is needed inside, the exit code
comes back and the program is found on the machine’s
PATH. shell opens an interactive shell: machined’s login session for a
booted machine, which needs D-Bus inside, or /bin/sh in the namespaces for an
app.
With volumes: -v /srv/data:/data mounts a host directory,
-v pgdata:/var/lib/postgresql a named volume that nspawn keeps under
/var/lib/nspawn/volumes/pgdata. Named volumes survive rm and images rm. See
Volumes.
Yes: sudo nspawn create fedora-44 db makes another machine that shares the
layers of fedora-44 and has a writable layer, an address, ports and settings
of its own. See More machines from one image.
Yes. sudo nspawn start NAME --restart always (or unless-stopped) enables the
unit and also restarts the machine when it ends; systemctl enable systemd-nspawn@NAME.service (or machinectl enable NAME) works too. Either way
the drop-in nspawn installs on the unit prepares the network and publishes the
ports whoever starts the machine.
sudo nspawn ps shows it, and sudo nspawn network inspect bridge (or the
name of another network) lists the machines of a network with their addresses.
Inside the machines, the other
machines are reachable by name and the host as host.nspawn.internal; on hosts
with systemd 258 or newer, machined resolves the machine names on the host as
well.
sudo nspawn rm NAME (or images rm) removes a machine or image, and
afterwards every layer and blob that no remaining image references. Overlay
machines keep their writes in a private directory under /var/lib/nspawn,
which goes away with the image; named volumes stay until sudo nspawn volume rm NAME or sudo nspawn volume prune. sudo nspawn system df shows what images,
machines, volumes and the build cache take. sudo nspawn system prune removes
stopped machines made from an image (by create or run), unused networks and
the build cache no image uses, -a unused images and the whole build cache too;
named volumes go with volume prune, and sudo nspawn builder prune takes the
build cache alone.
For questions, #nspawn-org on Matrix
or on Libera.Chat. For bugs, the issues
of github.com/nspawn/nspawn for the
tool, mkosi-definitions
for the images, and nspawn.org for
this site.
nspawn is developed on GitHub under the nspawn organization:
| Repository | What it holds |
|---|---|
| nspawn | The tool (Rust). |
| mkosi-definitions | The mkosi configuration of the images on the hub. |
| nspawn.org | This site. |
| nspawn.github.io | The blog. |
Issues and pull requests are welcome in all of them. The code of conduct applies everywhere.
A Rust toolchain of version 1.85 or newer is enough for the unit tests and clippy, which need no root and touch nothing outside temporary directories:
cargo fmt
cargo clippy --all-targets -- -D warnings
cargo test
cargo build --release
The repository documents itself: docs/HACKING.md for building and testing,
docs/ARCHITECTURE.md for how the pieces fit, docs/DBUS.md for the interface
the service exposes, and packaging/ for the Fedora, Arch and Debian packages
and the SELinux policy. A pull request runs the checks and builds those
packages.
A binary you built has to be installed before it can serve:
sudo install -Dm755 target/release/nspawn /usr/local/bin/nspawn followed by
sudo nspawn daemon --install. Replacing the binary of a running service is
fine; the next command starts the new one.
The end-to-end test runs the built binary against a real registry and systemd-machined, as root, on a host with systemd-nspawn:
NSPAWN=./target/release/nspawn sudo -E tests/e2e.sh
It exercises the whole tool: hub ls, search on the hub and on Docker Hub,
login and logout, a pull with both the overlay and the flat backend,
booting, exec through the PTY, volumes, the bridge network with names and
published ports between two machines, create from a local image, layer
sharing and garbage collection, an app from Docker Hub with entrypoint,
environment and volumes, machinectl start through the unit hooks, programs
that exit on their own or ignore their stop signal, and (when mkosi is
installed) a build, push and pull round trip. NSPAWN_REGISTRY (and
NSPAWN_CA_CERT for a private CA) must point at a registry that serves the
image named in IMAGE, fedora:44 by default; the Docker Hub steps need
internet access.
Keep pull requests focused, add a unit test when the change has logic that can be tested without a host, and run clippy before sending.
The images are plain mkosi configuration trees. Changes to what the hub ships,
new distributions or releases go to
mkosi-definitions. A definition
can be tried locally with nspawn build -t test/NAME:1 DIRECTORY before
sending it.
The site is built with Hugo and the Docsy theme. With nspawn installed, a live preview needs nothing else:
git clone https://github.com/nspawn/nspawn.org.git
cd nspawn.org
sudo nspawn compose up
Then open http://localhost:1313. The machine runs as uid and gid 1000; the
README of the repository explains how to pass yours if they differ. Without
nspawn, install Node.js, Go and git and run npm run install:safe followed by
npm run serve. The pages are Markdown files under content/en/; every
documentation page has an “Edit this page” link that opens the file on GitHub.