The Docker API
nspawn serves the Docker Engine API, 1.51 and any version down to 1.24 a client
asks for, on a socket of the host and in each machine that mounts
/var/run/docker.sock. Clients make, start, attach to, exec in, copy files to and from and
remove containers, pull and remove images, and make, join and remove volumes and
networks. The server is a client of the service,
as the command line is: every change it makes is a method of org.nspawn, under
the same polkit actions, and what it makes is a machine like any other, which
nspawn ps shows.
The host’s socket
The packages ship it turned off:
sudo systemctl enable --now nspawn-api.socket
export DOCKER_HOST=unix:///run/nspawn/api.sock
Every user may connect to /run/nspawn/api.sock, and polkit decides what each
one may do, with the service’s actions: reads and waits ask for
org.nspawn.inspect, logs and changes for org.nspawn.manage, _ping and
version for nothing. polkit is never asked to prompt, since nothing on a socket
answers an authentication agent: a user other than root needs a rule, such as
nspawn-wheel.rules, which the packages ship as an example. Root may do
everything.
Then the docker command works with it:
$ docker pull busybox:1.37
1.37: Pulling from library/busybox
docker.io/library/busybox:1.37: manifest 66a6306db78b with 1 layer(s), assembling as overlay
68fe9bff2ad4: Already exists
30ecbe150909: Already exists
Digest: sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec
Status: Downloaded newer image for busybox:1.37
docker.io/library/busybox:1.37
$ docker run --rm busybox:1.37 echo hello
hello
$ docker run -d --name web -p 8080:80 busybox:1.37 httpd -f -p 80 -h /tmp
c7dac8ca487657695c14e21240f15fd99e7eac48d6c13e770febe7b0ac74d31e
$ echo hello > index.html && docker cp index.html web:/tmp/index.html
$ curl -s http://127.0.0.1:8080/
hello
$ docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
NAMES IMAGE STATUS PORTS
web busybox:1.37 Up Less than a second 0.0.0.0:8080->80/tcp, [::]:8080->80/tcp
$ docker exec web sh -c 'id; pwd'
uid=0(root) gid=0(root)
/
$ sudo nspawn ps
MACHINE IMAGE MODE COMMAND STATE UP PID NETWORK OS
web docker.io/library/busybox:1.37 app httpd -f -p 80 -h /tmp running 0s 2814430 10.99.0.3 8080->80/tcp -
Volumes and networks are nspawn’s:
$ docker volume create data
data
$ docker network create back
68735561ab7cd7e4da9b248b9c68ff65d4518c05d1730d79056e8dd3c2dc8eda
$ docker network connect back web
$ docker run --rm --network back -v data:/data busybox:1.37 sh -c 'wget -qO- http://web/ | tee /data/page'
hello
$ docker run --rm -v data:/data busybox:1.37 cat /data/page
hello
Where nothing else on the host has /run/docker.sock, clients that only look
there or at /var/run/docker.sock find the socket with a drop-in:
sudo mkdir -p /etc/systemd/system/nspawn-api.socket.d
printf '[Socket]\nSymlinks=/run/docker.sock\n' | sudo tee /etc/systemd/system/nspawn-api.socket.d/symlink.conf
sudo systemctl daemon-reload && sudo systemctl restart nspawn-api.socket
GitLab Runner
GitLab Runner’s docker executor runs its jobs through the socket: the clone, the
job’s image, services with their aliases, artifacts, cache volumes and a network
per build (FF_NETWORK_PER_BUILD). In its config.toml:
[[runners]]
executor = "docker"
[runners.docker]
host = "unix:///run/nspawn/api.sock"
image = "alpine:3.22"
The runner runs as root, or as a user that a polkit rule lets through.
What it serves
_ping,version,infoandsystem df.- Containers: list, inspect, logs, stats, top, events; create, start, stop,
restart, kill, pause, unpause, wait, update, remove and prune. A create’s
settings become the machine’s options. What nspawn has no use for is named in
the answer’s
Warnings(CpuShares,SecurityOpt, a logging driver other thanjson-file,local,journaldornone, among others); what would change what the container does is refused with 400 (PidMode,IpcModeorUTSModeof the host, a user namespace mode, another runtime,VolumesFrom, an IPv6 address). - Attach, with the input:
docker run -i,docker run -itanddocker start -aigive the program their input, and an attach that keeps the input gets the output byte for byte. - Exec: a terminal or the two streams apart, the input, variables, working directory and user. On a terminal, the end of the input or the client’s hangup sends the command SIGHUP.
- Copies:
docker cpboth ways, on running containers and on stopped ones, owners and modes kept. A volume of a stopped container is mounted only while it runs, so a copy to or from it is refused until it starts. - Images: list, inspect, history, pull (
POST /images/create), remove and prune. A record pulled or built that never ran is an image, not a container; a machine made from it is a container. An image’s ID is its config’s digest. An image built here with a name without a registry (nspawn build -t somos/app:1, which nspawn files ashub.nspawn.org/somos/app:1) goes by that short name too while Docker Hub’s image of it is not here, sodocker run somos/app:1finds it. - Volumes and networks: list, inspect, create, remove and prune; connect and
disconnect with aliases and fixed addresses.
bridge,hostandnoneare the pre-defined networks.
Builds, loads, imports, tags, pushes and commits of images, container renames and exports, swarm and plugins answer 501 with “not supported”.
A name the API takes that is no machine name, such as reaper_1, stays the
container’s name in the API, and its machine gets one made of it (reaper-1- and
eight hex digits), which nspawn ps shows. A network keeps the name the API gave
it the same way.
The socket in a machine
A machine that mounts /var/run/docker.sock, /run/docker.sock or
/run/nspawn/api.sock from the host gets nspawn’s Docker API there, whatever the
host has at that path, with -v /var/run/docker.sock:/var/run/docker.sock or
compose’s use_api_socket. Mounted read-write, it takes every request the API serves:
$ sudo nspawn run --rm -v /var/run/docker.sock:/var/run/docker.sock \
docker.io/library/docker:28.5.1-cli -- docker run --rm busybox:1.37 echo from-a-machine
docker-28.5.1-cli-6fd52594: 10 layer(s) shared with docker-28.5.1-cli, assembling as overlay
from-a-machine
Mounted with :ro, it reads, follows logs and kills, and refuses any other change
with 403:
$ sudo nspawn run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro \
docker.io/library/docker:28.5.1-cli -- docker run --rm busybox:1.37 true
docker-28.5.1-cli-c46d6cef: 10 layer(s) shared with docker-28.5.1-cli, assembling as overlay
docker: Error response from daemon: this machine binds nspawn's Docker API read-only
Run 'docker run --help' for more information
Tools that follow containers through the socket find nspawn’s machines the same
way: reverse proxies that route by labels (caddy-docker-proxy, Traefik,
nginx-proxy with docker-gen and acme-companion), dashboards (homepage) and log
viewers (Dozzle). They only read, and kill to have a server read its
configuration again, so :ro is all they need. caddy-docker-proxy in front of a
service, from a compose.yaml in a directory proxy:
services:
caddy:
image: lucaslorentz/caddy-docker-proxy:2.13-alpine
ports:
- "80:80"
environment:
- CADDY_INGRESS_NETWORKS=proxy_default
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
whoami:
image: traefik/whoami
labels:
caddy: http://whoami.example.test
caddy.reverse_proxy: "{{upstreams 80}}"
$ sudo nspawn compose up -d
$ curl -s -H 'Host: whoami.example.test' http://127.0.0.1/ | head -2
Hostname: proxy-whoami-1
IP: 127.0.0.1
A machine that has the socket sees every machine and their -e variables
(--secret values never show). Only root inside the machine can connect.
How it is served
Each machine that takes the socket has its own, /run/nspawn/api/NAME.sock,
opened by systemd (nspawn-api@NAME.socket) before the machine starts and closed
after it stops; the host’s is nspawn-api.socket. The server behind each,
nspawn-api@NAME.service or nspawn-api.service, starts when something connects
and leaves a minute after the last connection, the attached sessions and exec
commands of its clients included. A container ID comes from the machine’s name and
creation time, so it stays across restarts and updates. nspawn binds the machine’s
/etc/hostname from a directory named after that ID, so tools that look for their
own container in /proc/self/mountinfo find it.
What one socket’s clients hold at once is bounded: 64 connections (one more is
closed at once), 16 of them logs, events or attach streams, each running a
journalctl, 16 waits and attaches, and 64 exec commands that run; one more of
any of these gets 503. On the host’s socket a user other than root holds a quarter
of the connections and of the exec commands and half of the streams and of the
waits at most, so that no user keeps the others or root out, and root has 16
streams of its own. mstack machines cannot take the socket.
auto-update
sudo nspawn auto-update moves the machines labelled
org.nspawn.auto-update=registry to the image their tag points to now.
io.containers.autoupdate=registry is taken too, on the machine, never from
its image, with image meaning registry and disabled or an empty value no
updates.
$ sudo nspawn run -d --name tagged -l org.nspawn.auto-update=registry \
docker.io/library/busybox:latest -- sleep 600
$ sudo nspawn auto-update --dry-run
UNIT CONTAINER IMAGE POLICY UPDATED
systemd-nspawn@tagged.service tagged docker.io/library/busybox:latest registry false
For each machine the registry is asked for the tag’s digest, with a HEAD that
registries do not count against their limits. A newer image is pulled, its
signature policy included, and the machine is made anew from it with every
option it had: networks and address, ports, volumes, variables, labels,
limits, restart policy and the rest (generate prints the same command before
and after), and its creation time, so its Docker API ID stays. A running
machine is stopped and started again and has to come up: healthy when it has a
healthcheck, still running ten seconds after its start otherwise. When it does
not, the old image and what the machine had written are put back and it starts
again, which the table reports as rolled back.
A machine on the new image starts with a fresh writable layer: what it keeps
belongs in volumes. Machines that share its
network (--network container:) start again with it, and so do compose
services whose depends_on asks for restart: true. Built images, references
pinned by digest, run --rm machines, booted machines (their state is their
writable layer), flat and mstack ones, and machines whose image declares a
volume (VOLUME) none of their own volumes covers, whose data would stay in the
old writable layer, are left alone and show skipped, as the local policy
does; the reason goes to standard error for a machine named on the command
line, and is in --json always. A policy that is none of these (a misspelt
one) shows failed, so that the run and its timer show it.
--dry-run only says which machines have a newer image waiting (pending).
The UPDATED column takes false, pending, true, failed, rolled back
(the old image back and running) or skipped; --json prints a JSON object per machine, and the command fails
when a machine failed or went back.
The packages ship a daily timer with up to fifteen minutes of random delay, disabled:
sudo systemctl enable --now nspawn-auto-update.timer