Skip to content

Instantly share code, notes, and snippets.

@berglh

berglh/minimal-zfs-root.md

Last active May 9, 2025 15:16
Show Gist options
  • Save berglh/2d137c93f8e2370a5b3474bd63b0a386 to your computer and use it in GitHub Desktop.
Save berglh/2d137c93f8e2370a5b3474bd63b0a386 to your computer and use it in GitHub Desktop.
Download ZIP
Minimal ZFS Root on Ubuntu 25.04
Raw
minimal-zfs-root.md

Minimal ZFS Root on Ubuntu 25.04

Topics

Introduction

This follows the guide Ubuntu 22.04 Root on ZFS with some changes.

  • Custom structure of boot and root ZFS pools
  • Lack of intermediary containers i.e. ubuntu/root instead of rpool/ROOT/ubuntu.
  • Changes to work with Ubuntu 25.04 specifically

⚠️ Warning: If more than one Linux distribution will be used on the system, using the structure in the guide with intermediary containers is useful. It allows having different boot and root pools on the same physical device and booting to different operating systems using grub.

Prepare environment

Boot from the Ubuntu Live USB (ISO), preferably the same version you wish to install.

Update and install basic tools, installation via SSH is a bit easier so configure if required.

sudo apt update
apt install --yes openssh-server vim

Disable automounting

gsettings set org.gnome.desktop.media-handling automount false

Install ZFS in the Live CD environment:

apt install --yes debootstrap gdisk zfsutils-linux
systemctl stop zed

Disk Formatting

Set a variable with the disk name:

DISK=/dev/disk/by-id/nvme-Force_MP600_1932822900012855046A

Ensure swap parts are not in use:

swapoff --all

Destroy disk

If contained ZFS:

wipefs -a $DISK

If flash-storage:

blkdiscard -f $DISK

Clear partition table:

sgdisk --zap-all $DISK

Partitions

Create the partition layout for EFI, no BIOS partition required for EFI only.

  • This creates an EFI partition
  • This creates a boot partition as ZFS (which EFI is mounted in)
  • Adjust the swap partition size (-16G) to the desired size at the end of the disk
  • Puts swap partition at the end of the disk formatted as swap
  • Check sgdisk -p /dev/disk/by-id/nvme-Force_MP600_1932822900012855046A to ensure there is no sector overlaps
sgdisk -n1:0:+512M -t1:EF00 -c1:"EFI" $DISK
sgdisk -n2:0:+2G   -t2:BF00 -c2:"ZFS boot" $DISK
sgdisk -n3:0:-16G  -t3:BF00 -c3:"ZFS root" $DISK
sgdisk -n4:0:0     -t4:8200 -c4:"Linux Swap" $DISK
sgdisk -p $DISK

Format Swap

mkswap /dev/disk/by-id/${DISK}$-part4

Format EFI

apt install --yes dosfstools
mkdosfs -F 32 -s 1 -n EFI ${DISK}-part1

Create Zpools

Boot Pool

# boot pool
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -o cachefile=/etc/zfs/zpool.cache \
    -o compatibility=grub2 \
    -o feature@livelist=enabled \
    -o feature@zpool_checkpoint=enabled \
    -O devices=off \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/boot -R /mnt \
    boot ${DISK}-part2

Root/OS Pool

# root (ubuntu) pool
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/ -R /mnt \
    ubuntu ${DISK}-part3

Create ZFS File Systems

zfs create -o mountpoint=/ \
    -o com.ubuntu.zsys:bootfs=yes \
    -o com.ubuntu.zsys:last-used=$(date +%s) ubuntu/root
zfs create -o mountpoint=/boot boot/strap

Create ZFS Datasets

Create home container & datasets.

zfs create -o canmount=off -o mountpoint=/ ubuntu/home
zfs create -o canmount=on -o mountpoint=/root ubuntu/home/root
chmod 700 /mnt/root

# zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off ubuntu/usr
zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off ubuntu/var
zfs create ubuntu/var/lib
zfs create ubuntu/var/log
zfs create ubuntu/var/spool
zfs create ubuntu/var/cache
zfs create ubuntu/var/mail

Create tmp dataset allows excluding from snapshots

zfs create -o com.ubuntu.zsys:bootfs=no ubuntu/tmp
chmod 1777 /mnt/tmp

Mount a tmpfs at /run:

mkdir /mnt/run
mount -t tmpfs tmpfs /mnt/run
mkdir /mnt/run/lock

Bootstrap Ubuntu

#apt install --yes debootstrap
debootstrap plucky /mnt

Copy zpool.cache

mkdir /mnt/etc/zfs
cp /etc/zfs/zpool.cache /mnt/etc/zfs/

System Configuration

hostname
echo quark > /mnt/etc/hostname
vi /mnt/etc/hosts # add 127.0.0.1 ${hostname}

Configure networking

ip a # get inteface name
vim /mnt/etc/netplan/01-network.yaml # add basic config

Change eno1 to your interface name. Configure static networking if required.

network:
  version: 2
  ethernets:
    eno1:
      dhcp4: true

Add app sources

ls /mnt/etc/apt/sources.list.d/
vim /mnt/etc/apt/sources.list.d/ubuntu.sources

Types: deb deb-src
URIs: http://au.archive.ubuntu.com/ubuntu/
Suites: plucky plucky-updates plucky-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb deb-src
URIs: http://security.ubuntu.com/ubuntu
Suites: plucky-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Check keyring exists

ls /usr/share/keyrings/ubuntu-archive-keyring.gpg

Copy terminfo optional (for kitty)

cp -r ~/.terminfo/ /mnt/root/

Chroot into installation

mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK bash --login

Configure basic environment

apt update

Configure language and tzdata

locale-gen --purge "en_AU.UTF-8"
update-locale LANG=en_AU.UTF-8 LANGUAGE=en_AU
ln -fs /usr/share/zoneinfo/Australia/Brisbane /etc/localtime
dpkg-reconfigure -f noninteractive tzdata

Optional if you require a different keyboard layout.

dpkg-reconfigure console-setup

Install neovim or the preferred text editor

apt install --yes neovim

EFI File-system

apt install --yes dosfstools
mkdir /boot/efi
echo '# EFI' > /etc/fstab
echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part1) /boot/efi vfat umask=0077 0 1 >> /etc/fstab
mount /boot/efi

Put /boot/grub on the EFI System Partition

Note: For a single-disk install only.

mkdir /boot/efi/grub /boot/grub
echo /boot/efi/grub /boot/grub none defaults,bind 0 0 >> /etc/fstab
mount /boot/grub

Install GRUB/Linux/ZFS in the chroot environment for the new system (zsys is the guide, but doesn't work in 25.04)

apt install --yes grub-efi-amd64 grub-efi-amd64-signed linux-image-generic shim-signed zfs-initramfs

Optional: Remove os-prober to avoids error messages from update-grub. os-prober is only necessary in dual-boot configurations

apt purge --yes os-prober

Set root password

passwd

Configure swap for an unencrypted single-disk install

echo '# Swap' >> /etc/fstab
echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) none swap discard 0 0 >> /etc/fstab
swapon -a

Add default system groups. Include (lxd) if you plan on using system level containers.

addgroup --system lpadmin
addgroup --system sambashare

Install and configure openssh-server

apt install --yes openssh-server

GRUB Installation

Verify that the ZFS boot filesystem is recognised

grub-probe /boot

Refresh the initrd files

update-initramfs -c -k all

Edit grub config, some settings can be reverted later if preferred

vim /etc/default/grub

# Disable memory zoning: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822
# Add init_on_alloc=0 to: GRUB_CMDLINE_LINUX_DEFAULT

# Comment out: GRUB_TIMEOUT_STYLE=hidden
# Set: GRUB_TIMEOUT=5
# Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5
# Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT
# Uncomment: GRUB_TERMINAL=console
# Save and quit.
update-grub

Update the boot configuration

update-grub

For UEFI booting, install GRUB to the ESP

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck --no-floppy

Fix filesystem mount ordering by activating zfs-mount-generator. This makes systemd aware of the separate mountpoints, which is important for things like /var/log and /var/tmp. In turn, rsyslog.service depends on var-log.mount by way of local-fs.target and services using the PrivateTmp feature of systemd automatically use After=var-tmp.mount.

mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/boot
touch /etc/zfs/zfs-list.cache/ubuntu
zed -F &

Verify that zed updated the cache by making sure these are not empty.

cat /etc/zfs/zfs-list.cache/boot
cat /etc/zfs/zfs-list.cache/ubuntu

If either is empty, force a cache update and check again:

zfs set canmount=on boot/strap
zfs set canmount=on ubuntu/root

If they are still empty, stop zed (as below), start zed (as above) and try again.

Once the files have data, stop zed:

fg
Press Ctrl-C.

Fix the paths to eliminate /mnt prefix.

sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*

Exit from the chroot environment back to the LiveCD environment:

exit

Run these commands in the LiveCD environment to unmount all filesystems

mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
    xargs -i{} umount -lf {}
zpool export -a

If export failed due to ubuntu busy error, try to kill everything that might be using it:

zfs umount ubuntu/root
# Check if processes have the pool open
grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq
# Kill the processes
for p in $(grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq); do kill $p; done
zpool export -a

If even after that the pool is busy, mounting it on boot will fail. In the initramfs prompt type zpool import -f ubuntu, then exit in the initramfs prompt.

Reboot 🤞

reboot

Wait for the newly installed system to boot normally.

First Boot

Login as root

ssh root@server-address

📸 Optional: Take some snapshots

zfs snapshot boot/strap@install
zfs snapshot ubuntu/root@install

🤓 Create a user

username=berg
zfs create -o canmount=on -o mountpoint=/home/$username ubuntu/home/${username}
adduser ${username}
# This isn't necessary if adduser does it
cp -a /etc/skel/. /home/${username}
# optionally copy terminfo if using kitty etc
cp -r /root/.terminfo /home/${username}
# ensure all files are owned by the user
chown -R ${username}:${username} /home/${username}
usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo $username

Full Software Installation

Upgrade the minimal system

apt dist-upgrade --yes

Install a command-line environment only

apt install --yes ubuntu-standard

Install a full GUI environment

Hint: If you are installing a full GUI environment, you will likely want to manage your network with NetworkManager.

apt install --yes ubuntu-desktop

Disable hibernation if not required, will cause warnings in initramfs update

echo "RESUME=none" | sudo tee /etc/initramfs-tools/conf.d/resume
update-initramfs -u

Disable mkinitramfs not found messages

touch /etc/zfs/vdev_id.conf
touch /etc/zfs/initramfs-tools-load-key
mkdir -p /etc/zfs/initramfs-tools-load-key.d
touch /etc/zfs/initramfs-tools-load-key.d/_placeholder
update-initramfs -u

Disable log compression

As /var/log is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if using snapshots of /var/log, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in /etc/logrotate.d by hand to comment out compress, or use this loop (copy-and-paste highly recommended):

for file in /etc/logrotate.d/* ; do
    if grep -Eq "(^|[^#y])compress" "$file" ; then
        sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file"
    fi
done

Final Cleanup

Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally.

Optional: Disable the root password

sudo usermod -p '*' root

Add an SSH authorised key to the created user, that should be currently logged in.

mkdir ~/.ssh
# paste a public SSH key into authorized_keys
vim .ssh/authorized_keys
# set correct permissions for .ssh
# if these are not set, SSH key login will fail
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown -R $(whoami):$(whoami) ~/.ssh

⚠️ Important: Disconnect and try login with authorized_key before disabling the insecure login methods. If it is configured incorrectly and password login is disabled, login to the system via SSH will fail.

Disable root SSH logins. If you installed SSH earlier, revert the temporary change:

sudo vi /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no
sudo systemctl restart ssh

Extras

Install landscape, shows system information on SSH login

sudo apt install landscape-common

Add terminal colours over SSH (scp ~/.bashrc from a Ubuntu desktop machine to home folder)

scp ~/.bashrc user@server:~
# then on the server
vim .bash_profile

Enter the following contents in the ~\.bash_profile file.

if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi

Recovery

To recover the system from a Live USB ISO. After loading into the terminal from the Live USB installer.

  • Once in the termina, go through Prepare enviornment steps.
  • Mount everything correctly: 
    zpool export -a
zpool import -N -R /mnt ubuntu
zpool import -N -R /mnt boot
zfs mount ubuntu/root
zfs mount boot/strap
zfs mount -a
  • If needed, chroot into the installed environment. 
    mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys
mount -t tmpfs tmpfs /mnt/run
mkdir /mnt/run/lock
chroot /mnt /bin/bash --login
mount -a
  • Perform recovery reparations
  • Cleanup by exiting from the chroot environment back to the LiveCD environment 
    mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
    xargs -i{} umount -lf {}
zpool export -a
  • If export failed due to ubuntu busy error, try to kill everything that might be using it: 
    zfs umount ubuntu/root
# Check if processes have the pool open
grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq
# Kill the processes
for p in $(grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq); do kill $p; done
zpool export -a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment