This is the multi-page printable view of this section. Click here to print.
nspawn manages systemd-nspawn machines the way docker manages containers: images come from an OCI registry (the hub), are stored as shared layers and are started, inspected and stopped through the D-Bus APIs of systemd-machined and systemd. This documentation describes nspawn 0.2.0.
If you are new, start with the overview and then get started. The command reference lists every command and option.
nspawn is a command line tool that manages systemd-nspawn machines with a docker-like workflow:
hub.nspawn.org,
where the nspawn.org team publishes images of common distributions built with
mkosi. Docker Hub and any other registry
work as well: search looks on the hub and on Docker Hub at once, and
login keeps credentials per registry, the way docker login does.systemd.mstack directory, or a flat copy. create makes more machines from
an image that is already local, without touching the registry.machinectl and importctl are never called, and
there is no daemon of nspawn’s own: every machine is an ordinary
systemd-nspawn@NAME.service unit that machinectl, systemctl and
journalctl see like any other. A drop-in makes that unit call nspawn before
the machine starts, once it runs and after it ends, so machinectl start, a
unit enabled at boot or a crash get the same network setup and cleanup as
nspawn start and nspawn stop.-p publishes
ports, -e sets variables, -v mounts host directories or named volumes,
--entrypoint and the arguments after -- change what an app runs.build runs mkosi on a directory with a mkosi.conf and imports the result
as a local image; push uploads an image to a registry, skipping the layers
that are already there.It is a single binary written in Rust. Its own state lives under
/var/lib/nspawn, the assembled machines under /var/lib/machines, and
everything it generates on the host is a plain systemd unit, drop-in or
.nspawn settings file that you can read.
nspawn tells two kinds of images apart when it installs them:
| Kind | What the image contains | How it runs |
|---|---|---|
| boot | An init system (systemd) and no other entrypoint, like the hub images | Booted with --boot, as machinectl start does. shell opens machined’s login session inside; exec enters the machine’s namespaces and needs nothing from it. |
| app | Anything else, for example an image from Docker Hub | Its entrypoint runs as PID 2 under nspawn’s stub init, with the environment, working directory, user and stop signal from the OCI config. Its network namespace is prepared by nspawn before the program starts, so the network is there from the first instruction. exec and shell enter the namespaces. |
The kind is detected at pull or build time and can be forced with
--mode boot or --mode app. Both kinds join the bridge network, are listed
by ps, stopped by stop and read by logs; the differences are described
in Machines.
Every machine joins nspawn0, a docker0 style bridge that nspawn manages
itself: fixed addresses, NAT, published ports (-p 8080:80) and a generated
/etc/hosts with the names of the other machines. --network host shares the
host’s network instead, and --network veth gives booted machines the classic
systemd-nspawn virtual ethernet pair configured by systemd-networkd. See
Networking.
| Piece | Where | Role |
|---|---|---|
nspawn | github.com/nspawn/nspawn | The command line tool this documentation is about. |
| The hub | hub.nspawn.org | An OCI registry with the images the team publishes. It is the default registry of the tool. |
| mkosi definitions | github.com/nspawn/mkosi-definitions | The public mkosi configuration the hub images are built from. |
| Blog | blog.nspawn.org | Release notes and news. |
build needs
mkosi installed on the host.overlay backend, which is the default on hosts older
than systemd 261 and what app images always use. Without it images are
extracted as flat directories.ip and nft) for the bridge network. Nothing
else: the bridge does not need systemd-networkd or NetworkManager on the host.
Only --network veth needs systemd-networkd.pull, create, build,
images rm, start, stop, login and logout write below
/var/lib/machines, /var/lib/nspawn, /etc/systemd and /etc/nspawn.
Listing, search, logs, exec, shell and hub do not need it.shell, which uses machined’s login
session; the hub images have it. exec enters the machine’s namespaces and
needs nothing inside.nspawn is a single binary. Build it with a Rust toolchain of version 1.85 or newer:
git clone https://github.com/nspawn/nspawn.git
cd nspawn
cargo build --release
sudo install -Dm755 target/release/nspawn /usr/local/bin/nspawn
nspawn --version
Install it in /usr/local/bin or /usr/bin, not below a home directory: the
unit of every machine calls nspawn by the path it was installed from, and on
SELinux hosts a system service is refused a binary under /home. The release
profile uses thin LTO and strips the binary, so the result is small and depends
on nothing but glibc.
Find an image. search asks the hub and Docker Hub and prints, for every hit,
the reference pull takes:
nspawn search fedora
SOURCE NAME DESCRIPTION STARS OFFICIAL
hub.nspawn.org fedora tags: 43, 44 - -
Docker Hub docker.io/library/fedora Official Docker builds of Fedora 1300 yes
...
Pull one. References without a registry part go to the hub, and the local name
is derived from the reference unless you pass --name:
sudo nspawn pull fedora:44
hub.nspawn.org/fedora:44: manifest sha256:3f9c... with 1 layer(s), assembling as overlay
blob sha256:8a1e...: downloading
blob sha256:c0de...: downloading
image fedora-44 (boot image) is ready: nspawn start fedora-44
The blobs went to /var/lib/nspawn, the root file system of the machine is
mounted at /var/lib/machines/fedora-44, /etc/systemd/nspawn/fedora-44.nspawn
holds the settings nspawn boots it with, and a drop-in of
systemd-nspawn@fedora-44.service makes the unit call nspawn around its life.
sudo nspawn start fedora-44
nspawn ps
MACHINE IMAGE MODE COMMAND STATE UP PID NETWORK OS
fedora-44 hub.nspawn.org/fedora:44 boot init running 12s 48213 10.99.0.2 fedora
start returns once the machine’s own systemd is up, so a command can follow
right away. exec runs it inside and brings back its exit code; shell opens
a login session as root:
nspawn exec fedora-44 -- systemctl is-system-running
nspawn shell fedora-44
The hub images log in as root without a password on the console. What the
machine printed is in logs, and --inside reads the journal of the machine
itself:
nspawn logs fedora-44
nspawn logs fedora-44 --inside -n 50
Stop it and, when you no longer need it, remove it. Removing an image also frees the layers and blobs that no other image references:
sudo nspawn stop fedora-44
sudo nspawn images rm fedora-44
Images without an init system run as apps: the entrypoint from the image runs
under a stub init, on the bridge like any other machine, so -p publishes its
ports on the host:
nspawn search nginx --source dockerhub
sudo nspawn pull docker.io/library/nginx:latest --name web
sudo nspawn start web -p 8080:80
curl -sI http://localhost:8080/ | head -1
nspawn logs web -f
sudo nspawn stop web
The port and any other flag given to start are remembered, so the next
sudo nspawn start web publishes it again. stop sends the image’s stop
signal (SIGQUIT for nginx) to the program and kills the machine after ten
seconds if it is still there; -t changes the grace period.
Docker Hub limits anonymous pulls per address. sudo nspawn login docker.io -u USER
keeps your credentials for that registry only; see
Registries and credentials.
create, where things are stored.start, stop, exec, shell, logs, ps,
entrypoints, environment and volumes.build and push./etc/nspawn/nspawn.toml, environment
variables and flags.An image reference has the form [registry/]repository[:tag|@digest]:
hub.nspawn.org; the configuration page
shows how to change it. The first path component counts as a registry when it
looks like a host, for example docker.io/library/nginx or
registry.example:5000/team/app.latest.., _, -
and /. Official Docker Hub images live under library/, so nginx is
docker.io/library/nginx; search prints references in that form.The local name of an image, the one start, stop, exec and friends
use, is derived from the reference (fedora:44 becomes fedora-44) unless
--name says otherwise. It has to be a valid machine name, so keep it to
letters, digits, - and _.
The hub is a plain OCI registry. nspawn hub ls lists its repositories with
their tags, and nspawn hub tags REPOSITORY the tags of one of them:
nspawn hub ls # everything
nspawn hub ls deb # repositories whose name contains "deb"
nspawn hub ls --no-tags # faster on big registries
nspawn hub tags fedora
The same commands work against any registry that implements the catalog
endpoint; point them somewhere else with --registry, the NSPAWN_REGISTRY
variable or the configuration file. A private CA is trusted with --ca-cert
or NSPAWN_CA_CERT.
The images on the hub are built with mkosi from the public definitions in nspawn/mkosi-definitions, with the package manager of each distribution.
nspawn search TERM [--source hub|dockerhub] [-n LIMIT]
search is docker search across the sources nspawn knows: the configured
hub (its catalog, case-insensitive substring match, with the tags of every
match) and Docker Hub (its search API). Every hit names its source and the
reference pull takes:
SOURCE NAME DESCRIPTION STARS OFFICIAL
hub.nspawn.org debian tags: 12, 13, bookworm, trixie - -
Docker Hub docker.io/library/debian Debian is a Linux distribution that's... 5100 yes
Docker Hub docker.io/someone/debian-x Debian with extras 3 -
--source asks one side only and -n limits the results per source (25 by
default). A source that cannot be reached is reported as a warning, not as an
error, so the other one still answers.
Registries are used anonymously until you log in:
sudo nspawn login # the hub
sudo nspawn login docker.io -u USER # Docker Hub
echo "$TOKEN" | sudo nspawn login registry.example -u ci --password-stdin
sudo nspawn logout docker.io
login asks for the user name and the password on the terminal (without
echo), or reads the password from standard input with --password-stdin, and
checks them the way docker login does: GET /v2/ on the registry, then basic
authentication or a token request at the realm the registry announces. Bad
credentials are refused with a clear message. A registry that never asks for
credentials gets them stored anyway, with a note saying so.
Credentials are kept in /etc/nspawn/auth.json, mode 0600, in the auth.json
format that podman and skopeo use. Credentials left on the host by
docker login or podman login are picked up as well, from
$XDG_RUNTIME_DIR/containers/auth.json, /run/containers/0/auth.json,
~/.docker/config.json and ~/.config/containers/auth.json, for root and for
the user behind sudo. Every operation chooses the credentials of the registry
it talks to, so the hub’s never travel to Docker Hub; the many names of Docker
Hub (docker.io, index.docker.io, registry-1.docker.io) are one entry.
Docker Hub limits anonymous pulls per address; logging in lifts that. push
authenticates before it uploads anything and, when the registry wants
credentials it does not have, says which login to run.
sudo nspawn pull REFERENCE [--name NAME] [--backend BACKEND] [--mode MODE] [--force]
pull needs root. It resolves the reference to the manifest for the host’s
platform (image indexes are followed), downloads every layer and the config
blob that is not already in the store, checking each one against its sha256
digest while it streams (a blob is written next to its final name and renamed
only once verified, so an interrupted download never passes for a complete
one), and then:
/etc/systemd/nspawn/NAME.nspawn, the settings the machine boots
with, and the drop-in that makes systemd-nspawn@NAME.service call nspawn
around its life;/var/lib/nspawn, and keeps the manifest and blobs so that
the image can be pushed or cloned later.An image with the same name is not replaced unless you pass --force, and
never while its machine is running. The old image only goes once the new one
is downloaded, and the store is locked only for that last step, so a long
download holds up neither other commands nor the unit hooks.
The backend decides how the layers become a directory systemd-nspawn can boot:
| Backend | What it does | Needs |
|---|---|---|
overlay | Layers are extracted once under /var/lib/nspawn and shared. Each machine gets an overlayfs mount at /var/lib/machines/NAME, defined by a generated mount unit, with a private upper directory for its writes. | Any systemd with overlayfs. |
mstack | A native systemd.mstack directory at /var/lib/machines/NAME.mstack that points at the shared layers; the machine boots with managed user namespaces (PrivateUsers=managed) and the layers are shifted into the foreign UID range. | systemd 261 or newer with the systemd-nsresourced and systemd-mountfsd sockets available (nspawn starts them). |
flat | The layers are extracted into a plain directory at /var/lib/machines/NAME. Nothing is shared, maximum compatibility. | Nothing. |
auto (default) | mstack when the host supports it, otherwise overlay when overlayfs is available, otherwise flat. |
Pick one per image with --backend on pull, build or create, or set a
default in the configuration file. nspawn images ls shows which backend each
image uses.
App images are always assembled as overlay (or flat), even where mstack
is available: a machine under managed user namespaces cannot join the network
namespace nspawn prepares for apps on the bridge. pull says so with a note
when it makes that choice.
After assembling, nspawn looks at the root file system and at the OCI config:
usr/lib/systemd/systemd,
lib/systemd/systemd, sbin/init or usr/sbin/init, as the topmost layer
leaves it) and whose entrypoint is that init, or has no entrypoint at all, is
a boot image. It is started with Boot=yes, as machinectl start would.Boot=no, ProcessTwo=yes),
with the config’s environment, working directory, user and stop signal.--mode boot or --mode app overrides the detection. The mode is recorded with
the image; Machines explains how it changes start, exec,
shell and stop.
sudo nspawn create SOURCE NAME [--backend BACKEND] [--network NETWORK] [-p HOST:CONTAINER]...
[--entrypoint PROGRAM] [-e VAR=VALUE]... [-v SOURCE:TARGET[:ro]]... [-f] [-- ARGUMENTS...]
create is docker create: another machine from an image that is already
local, given by its name or by the reference it was pulled from, without
touching the registry. It shares the source’s layers and gets a writable layer,
an address, settings and a record of its own, with create as its origin;
images rm of one never affects the others.
The network kind is inherited from the source unless --network says
otherwise; published ports are not, since two machines cannot publish the same
one. --entrypoint, -e and the arguments after -- apply to app images only
and mean the same as on start; -v works for both kinds. Everything on the
command line is checked before anything is made, so a refused create leaves
nothing behind. pull --name ends up the same way, but resolves the manifest
through the registry first.
nspawn images ls
NAME TYPE BACKEND ORIGIN SOURCE SIZE RO
fedora-44 directory overlay pull hub.nspawn.org/fedora:44 612 MiB no
db directory overlay create hub.nspawn.org/fedora:44 612 MiB no
web directory overlay pull docker.io/library/nginx:latest 190 MiB no
old-arch directory - - - 1.2 GiB no
The list comes from systemd-machined, so it includes images nspawn did not
create (old-arch above, with - in the nspawn columns). ORIGIN is pull,
build or create.
sudo nspawn images rm NAME...
images rm needs root and refuses to remove the image of a running machine.
For an image nspawn manages it unmounts and deletes the assembled root, the
settings file, the generated units and drop-ins, the network namespace of an
app and the record; for any other image, or for the leftovers of a failed
install, it removes what it finds and asks machined to forget the rest.
Afterwards the layers and blobs that no remaining image references are
deleted, the /etc/hosts files of the machines on the bridge are regenerated
and the published ports are brought in line with the machines that still run.
Named volumes are not deleted: they may belong to another machine.
| Path | Contents |
|---|---|
/var/lib/machines/NAME | The root of a machine with the overlay or flat backend (a mount point in the first case). |
/var/lib/machines/NAME.mstack | The systemd.mstack directory of a machine with the mstack backend. |
/var/lib/nspawn/ | nspawn’s own state: blobs as downloaded, extracted layers, image records, stored manifests, the private upper directories of overlay machines, the generated network files and units of each machine, mkosi build output and cache, and the store lock. |
/var/lib/nspawn/volumes/NAME | A named volume (-v NAME:/inside). |
/etc/systemd/nspawn/NAME.nspawn | The settings nspawn generates for a machine; regenerated at every start. |
/etc/systemd/system/systemd-nspawn@NAME.service.d/ | The drop-in with the unit hooks and, for an overlay machine, the one that requires its mount unit. The mount unit itself is next to them in /etc/systemd/system/. |
/etc/nspawn/nspawn.toml | The configuration file, optional. |
/etc/nspawn/auth.json | The credentials login stored, mode 0600. |
/run/netns/nspawn-NAME | The network namespace of a running app machine on the bridge. |
The state is kept outside /var/lib/machines on purpose: machined would list
a directory there as an image, and machinectl clean would delete it.
Every machine nspawn starts is the systemd unit systemd-nspawn@NAME.service,
registered with systemd-machined under its name. machinectl list,
machinectl status NAME, systemctl status systemd-nspawn@NAME and
journalctl -u systemd-nspawn@NAME all work on it; nspawn adds the
docker-like commands on top.
The unit also carries a drop-in, nspawn-hooks.conf, that calls nspawn around
the machine’s life: ExecStartPre prepares its network and settings,
ExecStartPost publishes its ports once it is registered, and ExecStopPost
releases everything however the machine ended. So machinectl start NAME,
systemctl enable systemd-nspawn@NAME for a machine that comes up at boot, a
program that exits on its own or a crash all behave like nspawn start and
nspawn stop. The drop-in names the nspawn binary that wrote it, which is why
nspawn belongs in /usr/local/bin or /usr/bin.
sudo nspawn start NAME [--network bridge|veth|host] [-p HOST:CONTAINER[/udp]]...
[--entrypoint PROGRAM] [-e VAR[=VALUE]]... [-v SOURCE:TARGET[:ro]]...
[--image-command] [--no-wait] [-- ARGUMENTS...]
start regenerates /etc/systemd/nspawn/NAME.nspawn from the machine’s
record, sets up the network it is configured for, starts the
unit through systemd and waits up to 30 seconds for the machine to register
with machined. For a booted machine it then waits, up to 20 more seconds, until
the machine’s own systemd is listening, so that a command can follow right
away; --no-wait skips that last wait. A program that runs and returns at once
is reported as such, not as a failure.
Everything given to start is remembered for the machine, so a plain
nspawn start NAME next time reuses the network, the ports, the command, the
variables and the volumes of the last run:
--network switches the machine between the bridge, a veth pair and the
host’s network; see Networking.-p HOST:CONTAINER[/udp] publishes a port on the host, like docker. It needs
the bridge network; -p none forgets all published ports.--entrypoint, -e and the arguments after -- change what an app
image runs; see Command, entrypoint and environment.
Boot images refuse them.-v mounts a host directory or a named volume into any kind of machine; see
Volumes.Machines are started by name. A reference (fedora:44, docker.io/x) is
refused with a hint to pull it or to create a machine from a local image.
Images that were not installed by nspawn, for example something created with
machinectl import-tar, can be started too: they get the stock template’s veth
networking, and nspawn makes sure systemd-networkd runs on the host so that the
machine actually gets an address, but none of the flags above apply to them.
A boot image is started with Boot=yes: systemd-nspawn runs the image’s init
as PID 1, like machinectl start does, and the machine joins the bridge with
an address that its own systemd-networkd configures.
exec enters the machine’s namespaces directly, like docker exec, so
nothing is needed inside: no D-Bus, no PAM. The exit code comes back and the
program is found on the machine’s PATH.shell opens machined’s login session (OpenMachineShell), which needs
D-Bus inside the machine; the hub images have it. Right after start the
machine’s D-Bus may not be up yet, so shell retries for up to 20 seconds
instead of failing.stop asks machined to power the machine off and repeats the request every
two seconds until the machine is gone, for up to a minute; the repetition
covers the window right after boot in which the init has not installed its
signal handlers yet. --force kills every process at once.nspawn-volumes.service,
holds local-fs.target until all of them are mounted, so services find their
data in place whatever the backend.An app image is started with Boot=no and ProcessTwo=yes: nspawn’s stub init
is PID 1 and the image’s entrypoint runs as PID 2 with the environment, working
directory, user and stop signal from the OCI config. Its network namespace is
built on the host before the program starts, so the network is there from the
first instruction, as in docker. For the same reason an app on the bridge runs
without a user namespace (PrivateUsers=no), which is also docker’s default;
capabilities, seccomp and the other namespaces still apply.
exec and shell enter the namespaces of the machine’s leader process, on a
pseudo terminal, with the image’s environment. shell runs /bin/sh, and
-u USER switches user for both.stop sends the image’s stop signal (StopSignal in the config, SIGTERM
by default) to the program itself, waits --timeout seconds (10 by default)
and kills the machine if it is still there, like docker stop. A stop
through systemd (systemctl stop, shutdown) reaches the stub init as a
poweroff request, which it answers with SIGTERM to the program.nspawn stop on it afterwards only clears
what it left behind and says NAME was not running.What an app runs is decided exactly as with docker:
-- replace the image’s cmd and follow its
entrypoint. nspawn start web -- nginx -T still runs
/docker-entrypoint.sh first, as docker run nginx nginx -T would.--entrypoint PROGRAM replaces the entrypoint with one program; the
arguments after -- follow it. --entrypoint "" drops the entrypoint, so
the arguments run alone.--image-command forgets them and runs the image’s own entrypoint and cmd
again.-e VAR=value adds a variable on top of the image’s, and -e VAR copies it
from the shell that runs nspawn. The later value of a variable wins, -e none
forgets them all, and exec sees the same environment as the program. Names
follow the usual rules (letters, digits and _, not starting with a digit).uid:gid form of the user, the uid part is used; the gid comes from
the image’s passwd.ps shows the effective command; nspawn start NAME -- true is a quick way to
check that an image runs at all.
sudo nspawn start web -v /srv/www:/usr/share/nginx/html:ro -v pgdata:/var/lib/postgresql
-v SOURCE:TARGET[:ro] is docker’s syntax:
SOURCE is a host directory (or file), created when it does not
exist. A SOURCE without a leading / is a named volume that nspawn
keeps under /var/lib/nspawn/volumes/NAME, created on first use and never
deleted by images rm. Names may contain letters, digits, _, . and -.TARGET is an absolute path inside the machine, other than /. The same
target cannot be mounted twice.:ro mounts it read-only; :rw is the default. Paths with whitespace are
not supported.-v is repeatable and remembered; -v none forgets every volume.In machines that run with private users (booted machines on overlay and flat,
which is the default), the mount is idmapped, so root inside owns what it
writes on the host, as docker users expect. On mstack machines nspawn attaches
the volumes from the host right after the machine’s init starts, since
systemd-nspawn cannot idmap binds under managed user namespaces; that is what
nspawn-volumes.service waits for, for up to two minutes, failing visibly
otherwise. On overlay and flat the volumes come from the settings file and are
there before the init even runs.
nspawn ps [-a] # same as: nspawn machines ls [-a]
MACHINE IMAGE MODE COMMAND STATE UP PID NETWORK OS
fedora-44 hub.nspawn.org/fedora:44 boot init running 2h 48213 10.99.0.2 fedora
web docker.io/library/nginx app /docker-entrypoint.sh nginx -g ... running 5m 51002 10.99.0.3 8080->80/tcp debian
db hub.nspawn.org/fedora:44 boot init stopped - - 10.99.0.4 -
ps lists every machine machined knows about; machines that nspawn did not
install show - in the image columns. -a adds the nspawn machines that are
not running. COMMAND is the effective entrypoint and arguments of an app,
and NETWORK the bridge address with the published ports, or host or
veth.
nspawn exec MACHINE [-u USER] COMMAND...
nspawn shell MACHINE [-u USER]
exec runs one command inside a running machine of either kind, attached to
your terminal, and exits with the command’s status, so it works in scripts and
pipelines (what goes through stdin and stdout is byte exact). The program is
looked up on the machine’s PATH, the image’s environment and the -e
variables apply, and the working directory is the image’s. Neither D-Bus nor
anything else is needed inside.
shell opens an interactive shell as root (or -u USER): machined’s login
session for booted machines, /bin/sh in the machine’s namespaces for apps.
nspawn logs MACHINE [-f] [-n N] [--since WHEN] [-t] [--all] [--inside]
systemd-nspawn sends what the machine writes to its console to the journal of
systemd-nspawn@MACHINE.service, and logs reads it with journalctl. By
default only the machine’s own output is shown, from every run of the unit,
earlier ones included:
-f keeps printing new output, starting from the last 10 lines unless -n
says otherwise. -n N shows the last N lines; --since "10 min ago"
accepts anything journalctl --since does; -t prefixes each line with its
timestamp.--all also shows what systemd logged about the unit: start, stop,
failures.--inside reads the journal of a booted machine itself
(journalctl --machine), which is where the services running inside log.logs works for stopped machines too, since the journal keeps what they wrote.
sudo nspawn stop NAME [-f] [-t SECONDS] [--no-wait]
Stops the machine as described above for boot and app machines, waits until it
is gone (a minute at most for a booted machine), stops the unit so that the
image can be removed right away, clears the failure a signal-killed program
leaves on the unit, releases the firewall exceptions of a veth machine and
removes the machine’s published ports and network namespace. --no-wait
returns right after the request, without the kill after --timeout; the unit
hooks release the network when the machine ends. -f kills every process at
once, like docker kill. Stopping a machine that already ended is not an
error.
Every machine records which network it uses. start --network changes it, and
the choice sticks for the next start.
| Network | Default for | What the machine gets |
|---|---|---|
bridge | Every image nspawn installs | A fixed address on the nspawn0 bridge, NAT to the outside, published ports, the names of the other machines and of the host. Needs nothing from the host’s own network manager. |
host | The host’s network namespace, like docker run --network host: the machine sees the host’s interfaces and binds to the host’s ports. Works for both kinds of image. | |
veth | Images not installed by nspawn | The classic systemd-nspawn setup: a virtual ethernet pair whose host end is configured by systemd-networkd through the stock 80-container-ve.network. Booted images only. |
On the first start of a bridged machine (or with nspawn network up, which
is handy at boot) nspawn creates the bridge with the first address of the
subnet, enables IPv4 forwarding and installs the nftables table ip nspawn
with masquerading for the subnet. The defaults are the bridge nspawn0 and the
subnet 10.99.0.0/24; both, and the DNS servers, can be changed in
the configuration file. An interface that already has
that name is only taken over when it is a bridge nspawn made, or an empty one;
bridge = "docker0" is refused rather than acted on. Nothing else on the host
is touched, so it works the same with systemd-networkd, NetworkManager or no
network manager at all.
Each machine gets a fixed address from the subnet, remembered with its record.
A booted machine receives it through a .network file that nspawn
generates and bind-mounts at /run/systemd/network/10-host0.network, for the
systemd-networkd inside to apply, together with the DNS servers: the host’s
upstream resolvers by default, dns from the configuration if set, and public
resolvers as a last resort, with a warning, when none can be determined.
An app machine has nothing inside to configure an interface, so nspawn
builds its network namespace before the program starts: ip netns, a veth pair
on the bridge, the address and the default route, handed to systemd-nspawn with
NamespacePath=, plus a generated /etc/resolv.conf. The namespace lives at
/run/netns/nspawn-NAME while the machine runs and goes away with it.
A generated /etc/hosts, mounted into every bridged machine, resolves the names
of the other machines on the bridge and host.nspawn.internal for the host.
On hosts with systemd 258 or newer, machined also lets the host resolve machine
names by itself.
nspawn network ls
nspawn0 10.99.0.0/24 (gateway 10.99.0.1, host name host.nspawn.internal)
MACHINE ADDRESS PORTS STATE
fedora-44 10.99.0.2 - running
web 10.99.0.3 8080->80/tcp running
db 10.99.0.4 - stopped
sudo nspawn start web -p 8080:80 -p 5353:53/udp
Each -p HOST:CONTAINER[/udp] becomes a DNAT entry in the ip nspawn table.
The port is reachable from other hosts, from the host’s own addresses and from
127.0.0.1 (through route_localnet, as docker does without its userland
proxy); binding to a single host address is not supported. The entries are
installed once the machine is registered and removed when it ends, by the unit
hooks, so they also go away after a crash or when the program exits on its
own. A port another running machine publishes, or one a service of the host
already listens on, is refused before the machine starts, and a refused port
is not remembered.
The list is remembered for the machine: nspawn start web next time publishes
the same ports, and -p none forgets them all. Ports need the bridge network;
a machine on the host’s network listens on the host’s ports directly.
trusted zone at runtime, which
also lets published ports through. The binding does not survive
firewall-cmd --reload; the next start or nspawn network up puts it
back.FORWARD
policy to DROP, which would silence every machine on the bridge. start
then adds two rules to the DOCKER-USER chain, which docker reserves for
that, or to the top of FORWARD itself: anything out of the bridge, and into
the bridge only what was published or belongs to a connection a machine
opened. Nothing happens on hosts without iptables.--network veth keeps the classic systemd-nspawn behaviour for booted
machines: systemd-nspawn creates a virtual ethernet pair and systemd-networkd on
the host brings up the host end (ve-NAME), gives it an address, serves DHCP
to the machine and masquerades its traffic, all through the stock
80-container-ve.network. App images refuse it, since nothing inside would
configure the pair.
Because that needs systemd-networkd, start activates it when the host has no
.network files of its own in /etc/systemd/network or /run/systemd/network,
and refuses with an explanation when it has, so that nspawn never takes over
interfaces another network manager is handling. A masked systemd-networkd is an
error. With firewalld, ve-NAME is bound to the trusted zone while the
machine runs; otherwise the default zone would drop the machine’s DHCP
requests.
--network host sets VirtualEthernet=no: the machine shares the host’s
network namespace, sees the host’s interfaces and binds to the host’s ports.
Published ports do not apply, and an app that runs this way keeps its user
namespace.
sudo nspawn build [DIRECTORY] -t REFERENCE [--name NAME]
[-d DISTRIBUTION] [-r RELEASE] [--profile PROFILE]...
[--backend BACKEND] [--mode MODE] [--force] [--keep-output]
[-- MKOSI_ARGUMENTS...]
build runs mkosi on a directory that has
a mkosi.conf or a mkosi.conf.d (the current directory by default), with
--format=oci, and imports the result exactly as pull would import an image
from a registry. It needs root and mkosi in PATH. The same configuration
tree that works with mkosi on its own works here; --distribution,
--release, --profile and anything after -- are passed through.
The command line that mkosi receives is:
mkosi --directory=DIRECTORY --format=oci --compress-output=zstd
--output-directory=/var/lib/nspawn/builds/NAME-TIMESTAMP
--cache-directory=/var/lib/nspawn/cache/mkosi
--image-id=REPOSITORY [--distribution=...] [--release=...] [--profile=...]...
[MKOSI_ARGUMENTS...] --force build
The / characters of the repository are replaced by - in the image id. The
package cache is kept between builds; the output directory is deleted after the
import unless --keep-output is given.
When mkosi is done, nspawn finds the OCI layout it produced, adds a few
annotations to the manifest (org.opencontainers.image.version with the tag,
org.opencontainers.image.ref.name with the full reference, and
org.nspawn.builder), verifies every blob against its digest, copies the blobs
into the store and installs the image with the chosen backend, with build as
its origin. The image can be started right away, cloned with create or
pushed.
A minimal configuration for a bootable Fedora machine, taken from the test suite of nspawn:
[Distribution]
Distribution=fedora
Release=44
[Output]
ImageId=e2e-built
[Content]
Bootable=no
SELinuxRelabel=no
RootPassword=root
Packages=
systemd
systemd-networkd
systemd-resolved
dbus-broker
passwd
util-linux
systemd-networkd is what configures host0 on the bridge, and dbus-broker
(or dbus) is what lets shell reach the machine’s systemd. Then:
sudo nspawn build -t team/app:1 ./app
sudo nspawn start team-app-1
nspawn push IMAGE [--to REFERENCE]
IMAGE is a local image name or the reference it was pulled from or built as.
By default the image is pushed under its own reference; --to pushes it under
another one, for example to retag it or to send it to a different registry:
nspawn push team/app:1
nspawn push team-app-1 --to registry.example/team/app:2
push authenticates first, with the credentials stored for that registry, and
tells you which nspawn login to run when the registry wants some it does not
have. The blobs that are already on the registry are skipped and the manifest
is uploaded last. A push needs a tag, not a digest, and the image’s manifest
and blobs must still be in the store (images rm removes them). See
Registries and credentials.
The images the team publishes on hub.nspawn.org are built the same way, from
the definitions in
nspawn/mkosi-definitions. Use
them as a starting point for your own images, or open a pull request there to
change what the hub ships.
nspawn needs no configuration to work: the defaults point at the hub, use
/var/lib/machines and /var/lib/nspawn, pick the best backend for the host
and run the bridge on 10.99.0.0/24. Everything below is optional.
--registry or
NSPAWN_REGISTRY, --ca-cert or NSPAWN_CA_CERT).The file is /etc/nspawn/nspawn.toml, read when it exists; --config FILE or
NSPAWN_CONFIG names another one, which then must exist. Every key is
optional, and unknown keys are an error so that a typo does not silently fall
back to a default.
| Key | Type | Default | Meaning |
|---|---|---|---|
registry | string | hub.nspawn.org | Registry for references without a host part. |
ca_cert | path | none | Extra CA certificate (PEM) to trust when talking to the hub. |
backend | auto, overlay, flat or mstack | auto | Backend for pull and build when they are not given --backend. |
machines_dir | absolute path | /var/lib/machines | Where machines are assembled. |
state_dir | absolute path | /var/lib/nspawn | Blobs, layers, records, volumes and everything else nspawn keeps. |
bridge | string, 1 to 15 letters, digits, - or _ | nspawn0 | Name of the bridge the machines join. It must be free, or a bridge nspawn made. |
subnet | IPv4 CIDR, prefix 8 to 30 | 10.99.0.0/24 | Subnet of the bridge; its first address is the bridge’s own. |
dns | list of IPv4 addresses | the host’s upstream servers | DNS servers handed to bridged machines. They must be reachable from the bridge: no loopback, no IPv6. |
An example:
# /etc/nspawn/nspawn.toml
registry = "registry.example:5000"
ca_cert = "/etc/pki/tls/certs/example-ca.pem"
backend = "overlay"
bridge = "br-lab"
subnet = "172.30.5.0/24"
dns = ["172.30.5.1", "9.9.9.9"]
Changing bridge or subnet affects machines started afterwards; the address
recorded for a machine is reassigned from the new subnet on its next start.
When a machine is installed or started with --config, the unit hooks it gets
carry the same --config, so the machine keeps using that file whoever starts
it.
| Variable | Same as |
|---|---|
NSPAWN_REGISTRY | --registry |
NSPAWN_CA_CERT | --ca-cert |
NSPAWN_CONFIG | --config |
They are convenient for scripts and for the end-to-end tests, which run against a private registry with a private CA:
NSPAWN_REGISTRY=hub.nspawn.test:8443 NSPAWN_CA_CERT=/etc/zot/ca.crt nspawn hub ls
nspawn login keeps registry credentials in /etc/nspawn/auth.json, and
credentials left by docker login or podman login are picked up from their
usual files. See Registries and credentials.
Some settings belong to a machine rather than to the host, and are given when it is created or started; every one of them is remembered until it is changed:
--backend and --mode on pull and build; --backend on create.--name on pull and build, to choose the local name.--network, -p, -e, -v, --entrypoint and the arguments after --
on start and create. -p none, -e none, -v none and
--image-command forget what was remembered.nspawn --help and nspawn COMMAND --help print the same information. Errors
are printed as error: ... on standard error and the exit status is 1;
exec exits with the status of the command it ran.
Commands that change the host need root: pull, create, build,
images rm, start, stop, login and logout. The others do not.
These are accepted by every command and can also come from the environment:
| Option | Environment | Meaning |
|---|---|---|
--registry REGISTRY | NSPAWN_REGISTRY | Registry (hub) for image references without a host part. |
--ca-cert FILE | NSPAWN_CA_CERT | Extra CA certificate (PEM) to trust when talking to the registry. |
--config FILE | NSPAWN_CONFIG | Configuration file; see Configuration. |
-h, --help | Help. | |
-V, --version | Version. |
Query the hub (an OCI registry).
nspawn hub ls [FILTER] [--no-tags]
Lists the repositories of the registry with their tags. FILTER keeps the
repositories whose name contains that text; --no-tags skips the tag query of
every repository, which is faster on big registries. hub list is an alias.
nspawn hub tags REPOSITORY
Prints the tags of one repository, one per line.
nspawn search TERM [--source hub|dockerhub] [-n LIMIT]
Finds images on the hub and on Docker Hub, like docker search, and prints
every hit with its source and the reference pull takes.
| Option | Meaning |
|---|---|
TERM | Text to look for in image names. |
--source hub|dockerhub | Only one source instead of both. |
-n, --limit LIMIT | Results per source. Default: 25. |
nspawn login [REGISTRY] [-u USERNAME] [--password-stdin]
Checks credentials against a registry, like docker login, and keeps them in
/etc/nspawn/auth.json for pull, push, search and hub.
| Option | Meaning |
|---|---|
REGISTRY | Registry host, for example docker.io or hub.nspawn.org. Default: the hub. |
-u, --username USERNAME | User name; asked for on the terminal when missing. |
--password-stdin | Read the password from standard input instead of the terminal. |
nspawn logout [REGISTRY]
Forgets the credentials stored for a registry (the hub by default).
nspawn pull REFERENCE [-n NAME] [--backend BACKEND] [--mode MODE] [-f]
Downloads an image from the hub or another registry and makes it available to systemd-machined.
| Option | Meaning |
|---|---|
REFERENCE | `[registry/]repository[:tag |
-n, --name NAME | Local image name. Default: derived from the reference, for example fedora-44. |
--backend auto|overlay|flat|mstack | How to assemble the image on this host. Default: auto, or the backend of the configuration file. App images are assembled as overlay even when mstack is chosen. |
--mode auto|boot|app | Whether the image boots an init system or runs a single program. Default: auto. |
-f, --force | Replace an existing image with the same name. |
nspawn create SOURCE NAME [--backend BACKEND] [--network bridge|veth|host] [-p HOST:CONTAINER[/udp]]...
[--entrypoint PROGRAM] [-e VAR[=VALUE]]... [-v SOURCE:TARGET[:ro]]... [-f] [-- ARGUMENTS...]
Makes another machine from a local image, like docker create, without
touching the registry. The layers are shared with the source.
| Option | Meaning |
|---|---|
SOURCE | Local image to start from: its name, or the reference it was pulled from. |
NAME | Name of the new machine. |
--backend BACKEND | How to assemble it. Default: like the source. |
--network bridge|veth|host | Network of the new machine. Default: like the source. |
-p, --publish HOST:CONTAINER[/udp] | Ports to publish on the host, like start -p. Not inherited from the source. |
--entrypoint PROGRAM | Replace the image’s entrypoint; an empty string runs the arguments alone. App images only. |
-e, --env VAR[=VALUE] | Environment for the program, VAR=value or VAR copied from the calling shell, like docker -e. App images only. |
-v, --volume SOURCE:TARGET[:ro] | Mount a host directory or a named volume, like docker -v. |
-f, --force | Replace an existing machine with the same name. |
-- ARGUMENTS... | App images: replace the image’s cmd; they follow its entrypoint, as with docker. |
nspawn build [DIRECTORY] -t TAG [-n NAME] [-d DISTRIBUTION] [-r RELEASE]
[--profile PROFILE]... [--backend BACKEND] [--mode MODE] [-f]
[--keep-output] [-- MKOSI_ARGS...]
Builds an image with mkosi and makes it available locally, ready to push. Needs mkosi.
| Option | Meaning |
|---|---|
DIRECTORY | Directory with the mkosi configuration (mkosi.conf, mkosi.conf.d, …). Default: .. |
-t, --tag TAG | Reference for the result, for example myapp:1 or hub.example/team/app:2. Required. |
-n, --name NAME | Local image name. Default: derived from the tag. |
-d, --distribution DISTRIBUTION | Distribution to build (mkosi --distribution). |
-r, --release RELEASE | Release to build (mkosi --release). |
--profile PROFILE | mkosi profile to enable; repeatable. |
--backend, --mode, -f | As for pull. |
--keep-output | Keep the mkosi output directory instead of deleting it after the import. |
-- MKOSI_ARGS... | Extra arguments passed to mkosi verbatim. |
nspawn push IMAGE [--to REFERENCE]
Uploads a local image to the hub or another registry, with the credentials
stored for it. IMAGE is a local image name, or the reference it was pulled
from or built as. --to pushes it under a different reference than the one
recorded for the image. The destination needs a tag, not a digest.
Manage local images.
nspawn images ls
Lists the local images known to systemd-machined, with nspawn’s backend, origin
(pull, build or create) and source reference for the ones it installed.
images list is an alias.
nspawn images rm NAME...
Removes local images and the layers and blobs nobody uses any more. Refuses the image of a running machine. Named volumes are kept.
nspawn ps [-a]
nspawn machines ls [-a]
Lists the running machines, like docker ps: image, mode, command, state,
uptime, leader PID, network and OS. -a, --all also lists the nspawn
machines that are not running. machines list is an alias of machines ls.
nspawn start NAME [--network bridge|veth|host] [-p HOST:CONTAINER[/udp]]...
[--entrypoint PROGRAM] [-e VAR[=VALUE]]... [-v SOURCE:TARGET[:ro]]...
[--image-command] [--no-wait] [-- ARGUMENTS...]
Boots an image as a machine. Every option is remembered for the next start.
| Option | Meaning |
|---|---|
--network bridge|veth|host | Network of the machine: the bridge (default), a veth pair configured by systemd-networkd on the host (booted images only), or the host’s own network. |
-p, --publish HOST:CONTAINER[/udp] | Publish a port on the host, like docker -p. Repeatable; none forgets them all. |
--entrypoint PROGRAM | Replace the image’s entrypoint; an empty string runs the arguments alone. App images only. |
-e, --env VAR[=VALUE] | Environment for the program, VAR=value or VAR copied from the calling shell. Repeatable; none forgets them. App images only. |
-v, --volume SOURCE:TARGET[:ro] | Mount a host directory or a named volume. Repeatable; none forgets them. |
--image-command | Forget the remembered entrypoint and arguments and run the image’s own again. |
--no-wait | Do not wait for a booted machine’s init to be up before returning. Its registration is still awaited, so that ports and firewall rules can be applied. |
-- ARGUMENTS... | App images: replace the image’s cmd; they follow its entrypoint, as with docker. |
nspawn stop NAME [-f] [-t SECONDS] [--no-wait]
Powers off a running machine: the image’s stop signal to the program of an app, a poweroff request to a booted machine. Stopping a machine that already ended only cleans up after it.
| Option | Meaning |
|---|---|
-f, --force | Kill every process at once instead of asking the machine to stop. |
-t, --timeout SECONDS | App images: seconds to wait after the stop signal before killing the machine. Default: 10. |
--no-wait | Return right after the stop request, without waiting for the machine to be gone and without the kill after the timeout. |
nspawn exec MACHINE [-u USER] COMMAND...
Runs a command inside a running machine of either kind, attached to the
terminal, in the machine’s namespaces, with the image’s environment and the
-e variables. The program is found on the machine’s PATH. Exits with the
command’s status.
| Option | Meaning |
|---|---|
-u, --user USER | User inside the machine. Default: root. |
COMMAND... | Command and arguments. |
nspawn shell MACHINE [-u USER]
Opens an interactive shell inside a running machine as USER (default
root): machined’s login session for booted machines, /bin/sh in the
machine’s namespaces for app machines.
nspawn logs MACHINE [-f] [-n N] [--since WHEN] [-t] [--all] [--inside]
Shows what a machine printed, like docker logs.
| Option | Meaning |
|---|---|
-f, --follow | Keep printing new output; starts from the last 10 lines unless --lines says otherwise. |
-n, --lines N | Only the last N lines. |
--since WHEN | Only output newer than this, in journalctl --since syntax, for example "10 min ago". |
-t, --timestamps | Prefix every line with its timestamp. |
--all | Also show what systemd says about the machine’s service: start, stop, failures. |
--inside | Booted machines only: read the machine’s own journal instead of its console output. |
The bridge network shared by the machines.
nspawn network up
Creates the bridge with its NAT rules and firewall exceptions, and brings the
published ports in line with the machines that run. start does it too; this
is useful at boot and for troubleshooting.
nspawn network ls
Lists the machines on the bridge with their addresses and published ports.
network list is an alias.
nspawn network prepare NAME, nspawn network publish NAME and
nspawn network release NAME are what the drop-in of
systemd-nspawn@NAME.service runs as ExecStartPre, ExecStartPost and
ExecStopPost. They are not meant to be typed and are hidden from --help.
A hub for systemd-nspawn images and the tool that uses them. The team builds
images of common Linux distributions with mkosi,
publishes them on an OCI registry at hub.nspawn.org, and maintains nspawn,
the command line tool documented on this site. The
mkosi definitions of the images
are public.
Yes. nspawn drives systemd-nspawn through systemd-machined and the systemd service manager, so the host needs both, with cgroup v2. The images on the hub contain systemd as well; images without an init system run as apps.
No. The commands and flags look alike on purpose (-p, -e, -v,
--entrypoint, create, exec, logs), but the machines are systemd-nspawn
containers managed by systemd: they are systemd-nspawn@NAME.service units,
appear in machinectl list, log to the journal and boot a full init when the
image has one. There is no daemon, no compose file and no orchestration.
Yes. nspawn search nginx finds them, and
sudo nspawn pull docker.io/library/nginx:latest --name web fetches one like
any other registry image. Since it has no init system, it is installed as an
app image: its entrypoint runs under nspawn’s stub init, on the bridge, with
-p for its ports. Docker Hub limits anonymous pulls per address;
sudo nspawn login docker.io -u USER lifts that. See
Getting started.
pull, create, build, images rm, start, stop, login and logout
write below /var/lib/machines, /var/lib/nspawn, /etc/systemd and
/etc/nspawn, and start and stop also change the bridge and its nftables
rules. Listing, search, logs, exec, shell and hub work without it.
Every blob is verified against the sha256 digest in the image manifest while it downloads, and registries are reached over HTTPS only. Signatures of manifests are not verified in this version.
The first generation of nspawn was a shell wrapper around machinectl that
downloaded tar.xz and raw.xz images from hub.nspawn.org/storage, signed
with the nspawn.org master key and verified by systemd-importd. Version 0.2
replaced it with the OCI based client described here: images are OCI images on
a registry, layers are shared, and machinectl is no longer involved. The
history of the old script is still in the
nspawn repository.
Yes. nspawn creates the bridge, its addresses and its nftables rules itself,
and hands each machine its address through a generated file or a prepared
network namespace. Neither systemd-networkd nor NetworkManager on the host is
involved; only --network veth depends on systemd-networkd. With firewalld the
bridge is put in the trusted zone, and on hosts where docker or ufw set the
forward policy to drop, nspawn adds the exception the bridge needs. See
Firewalls.
exec enters the machine’s namespaces, like docker exec: nothing is needed
inside, the exit code comes back and the program is found on the machine’s
PATH. shell opens an interactive shell: machined’s login session for a
booted machine, which needs D-Bus inside, or /bin/sh in the namespaces for an
app.
With volumes, as in docker: -v /srv/data:/data mounts a host directory,
-v pgdata:/var/lib/postgresql a named volume that nspawn keeps under
/var/lib/nspawn/volumes/pgdata. Named volumes survive images rm. See
Volumes.
Yes: sudo nspawn create fedora-44 db makes another machine that shares the
layers of fedora-44 and has a writable layer, an address, ports and settings
of its own. See More machines from one image.
Yes. systemctl enable systemd-nspawn@NAME.service (or machinectl enable NAME) is enough: the drop-in nspawn installs on the unit prepares the network
and publishes the ports whoever starts the machine.
nspawn ps and nspawn network ls show it. Inside the machines, the other
machines are reachable by name and the host as host.nspawn.internal; on hosts
with systemd 258 or newer, machined resolves the machine names on the host as
well.
sudo nspawn images rm NAME removes an image, and afterwards every layer and
blob that no remaining image references. Overlay machines keep their writes in
a private directory under /var/lib/nspawn, which goes away with the image;
named volumes stay until you delete them from /var/lib/nspawn/volumes.
For questions, #nspawn-org on Matrix
or on Libera.Chat. For bugs, the issues
of github.com/nspawn/nspawn for the
tool, mkosi-definitions
for the images, and website for
this site.
nspawn is developed on GitHub under the nspawn organization:
| Repository | What it holds |
|---|---|
| nspawn | The tool (Rust). |
| mkosi-definitions | The mkosi configuration of the images on the hub. |
| website | This site. |
| nspawn.github.io | The blog. |
Issues and pull requests are welcome in all of them. The code of conduct applies everywhere.
A Rust toolchain of version 1.85 or newer is enough for the unit tests and clippy:
cargo test
cargo clippy --all-targets -- -D warnings
cargo build --release
The end-to-end test runs the built binary against a real registry and systemd-machined, as root, on a host with systemd-nspawn:
NSPAWN=./target/release/nspawn sudo -E tests/e2e.sh
It exercises the whole tool: hub ls, search on the hub and on Docker Hub,
login and logout, a pull with both the overlay and the flat backend,
booting, exec through the PTY, volumes, the bridge network with names and
published ports between two machines, create from a local image, layer
sharing and garbage collection, an app from Docker Hub with entrypoint,
environment and volumes, machinectl start through the unit hooks, programs
that exit on their own or ignore their stop signal, and (when mkosi is
installed) a build, push and pull round trip. NSPAWN_REGISTRY (and
NSPAWN_CA_CERT for a private CA) must point at a registry that serves the
image named in IMAGE, fedora:44 by default; the Docker Hub steps need
internet access.
Keep pull requests focused, add a unit test when the change has logic that can be tested without a host, and run clippy before sending.
The images are plain mkosi configuration trees. Changes to what the hub ships,
new distributions or releases go to
mkosi-definitions. A definition
can be tried locally with nspawn build -t test/NAME:1 DIRECTORY before
sending it.
The site is built with Hugo and the Docsy theme. With Docker installed, a live preview needs nothing else:
git clone https://github.com/nspawn/website.git
cd website
docker compose up
Then open http://localhost:1313. The container runs as uid and gid 1000; the
README of the repository explains how to pass yours if they differ. Without
Docker, install Node.js, Go and git and run npm run install:safe followed by
npm run serve. The pages are Markdown files under content/en/; every
documentation page has an “Edit this page” link that opens the file on GitHub.