systemd-nspawn container architecture

This short document will show how to turn systemd-nspawn into a usable containeration system.

Those instructions here should work under Arch Linux and Debian 10 (Buster)

Host requirements

systemd-nspawn and machinectl (systemd-container package under Debian)
dnsmasq
debootstrap

Setup

We need to create a network bridge device where systemd-nspawn can connect all the containers. Using systemd-networkd you can create those two files under /etc/systemd/network/.

/etc/systemd/network/br0.netdev
```
 # Tell systemd-networkd to create a bridge device
 [NetDev]
 Name=br0
 Kind=bridge
```
/etc/systemd/network/br0.network
```
 # Configure ip address for the bridge
 [Match]
 Name=br0
 
 [Network]
 Address=172.23.0.1/24
 LinkLocalAddressing=yes
 IPMasquerade=yes
 LLDP=yes
 EmitLLDP=customer-bridge
```
After creating those configuration files and running command below you should be good.
```
 systemctl enable --now systemd-networkd
```
If you need to make any changes to the network configuration then you can just restart systemd-networkd to reload the config.
```
 systemctl restart systemd-networkd
```
So that all of our containers would always get a static ip address we are gona use dnsmasq.

Next we are going to tell dnsmasq to do static leases based on container hostname. In this config file im using 172.23.0.1/24 as the container network.

`/etc/dnsmasq.conf`

    domain=.local
    #no-poll # don't constanly poll /etc/resolv.conf
    #resolv-file=/etc/resolv.conf
    no-resolv
    server=8.8.8.8
    server=8.8.4.4
    domain-needed
    bogus-priv
    listen-address=127.0.0.1,172.23.0.1
    # Force bind only on interfaces that are listed in listen-address, allows systemd-resolve to work at the same time
    bind-dynamic
    dhcp-range=172.23.0.100,172.23.0.200,255.255.255.0,12h
    
    # When systemd-networkd or some other dhcp client requests for a ip address
    # they will be given a address based on their hostname.
    # This section can be moved to seperate file using dhcp-hostsfile option
    dhcp-host=http,172.23.0.2
    dhcp-host=mail,172.23.0.3
    dhcp-host=worpress,172.23.0.4
    dhcp-host=sql,172.23.0.5

[email protected] must be told to use the br0 bridge For that we have to edit ExecStart line in the service file and add --network-bridge=br0 to the end of it.

First get the currently used ExecStart value by running
```
 systemctl cat [email protected]
```
After that we need to create a override.conf file that changes that file. You can do that either by using command systemctl edit [email protected] or by manualy creating a file called /etc/systemd/system/[email protected]/override.conf

Contents of the override file should be something like this:
```
 [Service]
 ExecStart=
 ExecStart=<current execstart line> --network-bridge=br0
```
For Arch Linux the contents would be
```
 [Service]
 ExecStart=
 ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i --network-bridge=br0
```
NB: nspawn service by default adds -U argument that turns on private users support and shifts all UID/GID-s up some random amount. If you plan on sharing files between containers then this will mess up yout file owners.

You can enable mymachines nsswitch module that will do user and group id translation between host and container private users. https://www.freedesktop.org/software/systemd/man/nss-mymachines.html

Creating containers

That part is quite simple, just create a folder /var/lib/machines/<name> where name is your container hostname and deboostrap into it. machinectl start <name> should boot your container up and after staring systemd-networkd you should have a working network.

I have created some scripts automate container creation part

create_arch_container.sh - Creates a Arch Linux container
create_deb_container.sh - Creates a Ubuntu 16.04 container, but you can change it quite easily by changing SUITE argument from xenial to something else
create_container.sh - Creates a Debian 10 container on a ZFS pool, also configures locale and timezone inside the container.

NB: debootstrap copies your host /etc/hostname file into the container, remove it or replace it with your container hostname

Administrating nspawn containers

machinectl start <name> - Boot container named name
machinectl poweroff <name> - Poweroff the container
machinectl shell <name> [/bin/sh] - Run a command in the container, by default the command is /bin/sh. For Debian/Ubuntu containers you probably want to tell it to use /bin/bash
machinectl restart <name> - this will probably break nspawn because on startup it will try to register with machined and fail because it takes some time to release a container name in machined. (there is bug report about it somewhere)
Port forwarding If there is a file /etc/systemd/nspawn/<name>.nspawn then systemd-nspawn will load additional configuration options from it.

To do port forwarding you can do something like this:
```
  # /etc/systemd/nspawn/http.nspawn 
  [Network]
  Port=tcp:80:80
  Port=tcp:443:443
```
You can add many more options from systemd.nspawn(5) man page

Things that are broken under Debian

Debian 10 (Buster) defaults to modern nftables but systemd still uses legacy iptables framework for firewall configuration. Don't mix those two.

artizirk/systemd-nspawn container architecture.md

systemd-nspawn container architecture

Host requirements

Setup

Creating containers

Administrating nspawn containers

Things that are broken under Debian

plaes commented Jan 24, 2018 •

edited

Loading

artizirk commented Oct 15, 2019

artizirk commented Nov 10, 2019

artizirk/systemd-nspawn container architecture.md

systemd-nspawn container architecture

Host requirements

Setup

Creating containers

Administrating nspawn containers

Things that are broken under Debian

plaes commented Jan 24, 2018 • edited Loading

artizirk commented Oct 15, 2019

artizirk commented Nov 10, 2019

plaes commented Jan 24, 2018 •

edited

Loading