Skip to content

Instantly share code, notes, and snippets.

@Jip-Hop

Jip-Hop/README.md

Last active May 14, 2025 03:04
Download ZIP

Revisions

  1. Jip-Hop revised this gist Aug 13, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,3 @@
    # [Development of Jailmaker conintues in this git repo](https://github.com/Jip-Hop/jailmaker).
    # [Development of Jailmaker continues in this git repo](https://github.com/Jip-Hop/jailmaker).

    Feel free to continue discussing jailmaker in this gist.
    I've opened up [Github Discussions](https://github.com/Jip-Hop/jailmaker/discussions) to further discuss Jailmaker.
  2. Jip-Hop revised this gist Jan 29, 2023. 2 changed files with 2 additions and 402 deletions.
    14 changes: 2 additions & 12 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,13 +1,3 @@
    # Jailmaker
    # [Development of Jailmaker conintues in this git repo](https://github.com/Jip-Hop/jailmaker).

    ### Work in progress, experimental, for testing only! No warranty!

    Create a persistent 'jail' on TrueNAS SCALE to install software, such as docker, without modifying the host OS.

    - Connect to your TrueNAS via SSH as root user
    - Go to a directory where you want to store jailmaker, should be on one of your ZFS datasets (under the /mnt/ directory)
    - Clone this gist (this saves the code inside a directory named `jailmaker`): `git clone -b main --single-branch --depth 1 https://gist.github.com/Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d jailmaker`
    - Change into the `jailmaker` directory
    - Run `./jailmaker.sh` for an interactive installation or run `./jailmaker.sh help` for all available options.

    Please [vote for official systemd-nspawn support](https://ixsystems.atlassian.net/browse/NAS-119787) (login first, then click the thumbs up button in the top right).
    Feel free to continue discussing jailmaker in this gist.
    390 changes: 0 additions & 390 deletions jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,390 +0,0 @@
    #!/bin/bash

    set -euo pipefail
    shopt -s nullglob

    ABSOLUTE_SCRIPT_PATH="$(realpath "${BASH_SOURCE[0]}")"
    SCRIPT_NAME=$(basename "${ABSOLUTE_SCRIPT_PATH}")
    SCRIPT_PARENT_DIR="$(dirname "$ABSOLUTE_SCRIPT_PATH")"
    DEBIAN_RELEASE="$(. /etc/os-release && echo "${VERSION_CODENAME}")"

    USAGE="WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!
    Create persistent Debian 'jails' with systemd-nspawn on TrueNAS SCALE
    Allows installing software, such as docker, without modifying the host OS
    With full access to all files on the host via bind mounts
    Put this script on one of your ZFS datasets (under the /mnt/ directory)
    It will create a 'jails' dir next to it, where it stores all the jails
    Run this script as root user
    Usage: [ENV_VAR=value] ./${SCRIPT_NAME} COMMAND [ARG...]
    Commands:
    new Create a new jail in the jails dir and start it
    wizard Interactive guide to create and start a new jail
    up Bring up all jails by running all *.sh files in the jails dir
    help Show this help message
    All arguments passed to the new command will be added to the systemd-nspawn command that starts the jail
    For available arguments see: https://manpages.debian.org/${DEBIAN_RELEASE}/systemd-container/systemd-nspawn.1.en.html
    Environment variables:
    JAILMAKER_DEBUG=0
    JAILMAKER_JAIL_NAME=${DEBIAN_RELEASE}
    JAILMAKER_INSTALL_DOCKER=0
    JAILMAKER_INSTALL_NVIDIA=0
    JAILMAKER_GPU_PASSTHROUGH=0
    JAILMAKER_MACVLAN_PARENT_INTERFACE=
    Default settings can be overridden via environment variables, documentation about advanced settings is inside ${SCRIPT_NAME}
    Examples:
    # Start a new jail with read/write access to /mnt/ssd/appdata and readonly access to /mnt/Tank/Media
    ./${SCRIPT_NAME} new --bind=/mnt/ssd/appdata --bind-ro=/mnt/Tank/Media
    # Start a new jail named: my-jail-name
    JAILMAKER_JAIL_NAME=my-jail-name ./${SCRIPT_NAME} new
    # Start a new jail with GPU passthrough set-up (if a GPU device can be found during creation)
    JAILMAKER_GPU_PASSTHROUGH=1 ./${SCRIPT_NAME} new
    # Start a new jail, install docker and enable debug mode to output all the steps this script takes
    JAILMAKER_INSTALL_DOCKER=1 JAILMAKER_DEBUG=1 ./${SCRIPT_NAME} new
    # Start a new jail with macvlan networking (container will get its own IP address via DHCP)
    JAILMAKER_MACVLAN_PARENT_INTERFACE=eno1 ./${SCRIPT_NAME} new
    # Bring up all jails by running all *.sh files in the jails dir
    ./${SCRIPT_NAME} up
    "

    fail() {
    echo -e "$1" >&2 && exit 1
    }

    [[ $UID -ne 0 ]] && echo "${USAGE}" && fail "Run this script as root..."

    ####################################################################
    # Read from user-defined environment variables or use default values
    ####################################################################

    # Basic settings, should be safe to override

    # Print each line this script executes in debug mode - valid values: 0 (default), 1
    JAILMAKER_DEBUG="${JAILMAKER_DEBUG:-0}" && [[ "${JAILMAKER_DEBUG}" -eq 1 ]] && set -x
    # Default name for the jail is the debian release name, e.g. bullseye - valid values: a string without the '/' character in it
    JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-${DEBIAN_RELEASE}}"
    # By default we will use host networking - valid values: '' (default), name of a network interface
    JAILMAKER_MACVLAN_PARENT_INTERFACE="${JAILMAKER_MACVLAN_PARENT_INTERFACE:-}"
    # By default we will not install docker inside the jail - valid values: 0 (default), 1
    JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER:-0}"
    # By default we will not install nvidia drivers - valid values: 0 (default), 1
    JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA:-0}"
    # By default we will not setup GPU passthrough (unless we're installing nvidia drivers) - valid values: 0, 1
    JAILMAKER_GPU_PASSTHROUGH="${JAILMAKER_GPU_PASSTHROUGH:-${JAILMAKER_INSTALL_NVIDIA}}"

    # Advanced settings, be careful if you change these!

    # By default make a jail which matches the debian version of TrueNAS - valid values: a debian release name (only bullseye has been tested)
    JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE:-${DEBIAN_RELEASE}}"
    # By default we will use (or create) the 'jails' dir inside this script's parent directory - valid values: a path to a directory (parent directory should exist)
    JAILMAKER_JAILS_DIR="${JAILMAKER_JAILS_DIR:-${SCRIPT_PARENT_DIR}/jails}"
    # Only allow creating jails inside an /mnt sub-folder, unless safety check if off - valid values: 0, 1 (default)
    JAILMAKER_JAILS_DIR_SAFETY_CHECK="${JAILMAKER_JAILS_DIR_SAFETY_CHECK:-1}"

    # Since this file will be executed as root, we need to set appropriate permissions (if not already set)
    [[ "$(stat -c%a "${ABSOLUTE_SCRIPT_PATH}")" -ne 700 ]] && chmod 700 "${ABSOLUTE_SCRIPT_PATH}"

    validate_input() {
    local jails_dir_parent_dir
    jails_dir_parent_dir="$(dirname "${JAILMAKER_JAILS_DIR}")"

    # Check if working directory exists
    if [[ ! -d ${jails_dir_parent_dir} ]]; then
    fail "Destination directory '${jails_dir_parent_dir}' DOES NOT exist."
    # Check if working directory is inside the /mnt directory (so it won't be lost on TrueNAS OS updates)
    elif [[ "$JAILMAKER_JAILS_DIR_SAFETY_CHECK" -ne 0 && "${JAILMAKER_JAILS_DIR}" != /mnt/* ]]; then
    fail "The destination path must begin with '/mnt/'\nProvided destination was '${JAILMAKER_JAILS_DIR}.'"
    # Check for illegal characters in the directory name
    elif [[ "${JAILMAKER_JAIL_NAME}" == */* ]]; then
    fail "Name may not contain the '/' character.\nJAILMAKER_JAIL_NAME: ${JAILMAKER_JAIL_NAME}."
    fi
    }

    new() {
    validate_input

    local arch original_jail_name x jail_path relative_jail_path additional_flags
    arch=$(dpkg --print-architecture)

    # Create the jails dir if it does not exist
    [[ -d "${JAILMAKER_JAILS_DIR}" ]] || mkdir "${JAILMAKER_JAILS_DIR}"

    # Only root should be allowed access to this directory
    chmod 700 "${JAILMAKER_JAILS_DIR}"

    original_jail_name="${JAILMAKER_JAIL_NAME}"
    jail_name=${original_jail_name}
    x=1

    # Append counter to jail path if there already is a jail with this name
    while [[ -d "${JAILMAKER_JAILS_DIR}/${jail_name}" ]]; do jail_name="${original_jail_name}-$((x++))"; done

    jail_path="${JAILMAKER_JAILS_DIR}/${jail_name}"

    # Make the new jail directory
    mkdir "${jail_path}"

    # Download and extract the root filesystem for the jail
    curl -L "https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-${arch}/${JAILMAKER_DEBIAN_RELEASE}/slim/rootfs.tar.xz" | tar -xJ -C "${jail_path}" --numeric-owner

    # Remove resolv.conf, systemd configures this
    rm "${jail_path}/etc/resolv.conf"

    # Bind mount this script inside the jail and run it with the '__do_not_run_this_manually__' command
    # This will install software inside the jail
    systemd-nspawn -q -D "${jail_path}" --bind-ro="${ABSOLUTE_SCRIPT_PATH}" --bind-ro=/lib/modules -E JAILMAKER_MACVLAN_PARENT_INTERFACE="${JAILMAKER_MACVLAN_PARENT_INTERFACE}" -E JAILMAKER_DEBUG="${JAILMAKER_DEBUG}" -E JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE}" -E JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER}" -E JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA}" "${ABSOLUTE_SCRIPT_PATH}" __do_not_run_this_manually__

    # Start with the flags passed as argument
    additional_flags="$1"

    # Add flag for macvlan network
    if [[ -n "${JAILMAKER_MACVLAN_PARENT_INTERFACE}" ]]; then
    additional_flags+=" --network-macvlan=${JAILMAKER_MACVLAN_PARENT_INTERFACE}"
    fi

    # Add flags for GPU access
    if [[ "${JAILMAKER_GPU_PASSTHROUGH}" -eq 1 ]]; then

    additional_flags+=" --property=DeviceAllow='char-drm rw'"

    # Detect intel GPU device and if present add bind flag
    [[ -d /dev/dri ]] && additional_flags+=" --bind=/dev/dri"

    # Detect nvidia GPU devices and if present add bind flag
    for d in /dev/nvidia*; do additional_flags+=" --bind='${d}'"; done
    fi

    if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then
    # TODO: are any of these flags required for GPU access when NOT using docker?
    additional_flags+=" --capability=all --system-call-filter='add_key keyctl bpf'"
    fi

    # If there are flags to add, put a space in front if it isn't already there
    [[ -n "${additional_flags}" && "${additional_flags}" != " "* ]] && additional_flags=" ${additional_flags}"

    # Use SYSTEMD_SECCOMP=0: https://github.com/systemd/systemd/issues/18370
    # Use SYSTEMD_NSPAWN_LOCK=0: otherwise jail won't start jail after a shutdown (but why?)
    # Would give "directory tree currently busy" error and I'd have to run
    # `rm /run/systemd/nspawn/locks/*` and remove the .lck file from JAILMAKER_JAILS_DIR
    # Disabling locking isn't a big deal as systemd-nspawn will prevent starting a container
    # with the same name anyway: as long as we're starting jails using the accompanying .sh script,
    # it won't be possible to start the same jail twice

    relative_jail_path=$(realpath --relative-to="${JAILMAKER_JAILS_DIR}" "${jail_path}")

    # Write the static part of the startup shell script (no variable substitution)
    cat <<-'EOF' >"${jail_path}.sh"
    #!/bin/bash
    umask 077
    cd "$(dirname "${BASH_SOURCE[0]}")" || exit
    EOF

    # Append the dynamic part of the startup shell script (with variable substitution)
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --quiet --boot --directory='${relative_jail_path}'${additional_flags} &> '${relative_jail_path}.log' &" >>"${jail_path}.sh"

    # Make executable and allow access only for root user
    chmod 700 "${jail_path}.sh"

    # Start the jail
    "${jail_path}.sh"

    cat <<-EOF
    The jail will now boot in the background...
    List all the running jails with:
    machinectl list
    When it's running you can start a shell with:
    machinectl shell ${jail_name}
    To poweroff a jail (from inside the jail):
    poweroff
    In order to start the jail automatically after TrueNAS boot, configure '${jail_path}.sh' as Post Init Script from the TrueNAS web interface.
    In order to start all jails automatically after TrueNAS boot, configure '${ABSOLUTE_SCRIPT_PATH} up' as Post Init Script from the TrueNAS web interface.
    EOF
    }

    # This function may be called on each boot of TrueNAS to start all jails
    up() {
    validate_input
    [[ -d "${JAILMAKER_JAILS_DIR}" ]] || fail "Nothing to start: the '${JAILMAKER_JAILS_DIR}' directory does not exist."
    # Find all the *.sh files in the jails directory, and start them
    for d in "${JAILMAKER_JAILS_DIR}/"*.sh; do "${d}" || true; done
    }

    complete_install_inside_jail() {
    # Add a safeguard so we don't accidentally install anything on the TrueNAS host
    [[ $(cli -c 'system version' 2>/dev/null) == TrueNAS* ]] && echo "This function should only be called inside a jail!"

    # Since apt is no longer executable on TrueNAS, the script won't continue,
    # even if the safeguard above stops working
    apt update

    # We need dbus for `machinectl shell`, systemd to properly boot the jail
    # and systemd-sysv to for the `poweroff` command
    apt -y --no-install-recommends install dbus systemd systemd-sysv

    if [[ -n "${JAILMAKER_MACVLAN_PARENT_INTERFACE}" ]]; then
    # Not using host networking, so enable networkd and resolved services
    systemctl enable systemd-networkd
    systemctl enable systemd-resolved

    # Setup DHCP for macvlan network interfaces
    # https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_without_gui
    cat <<-'EOF' >/etc/systemd/network/dhcp.network
    [Match]
    Name=mv-en*
    [Network]
    DHCP=yes
    EOF
    fi

    if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then

    # Install docker according to official guide:
    # https://docs.docker.com/engine/install/debian/
    # Even shorter (but not recommended) would be:
    # curl -fsSL https://get.docker.com | sh

    apt -y --no-install-recommends install ca-certificates curl gnupg
    mkdir -p /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian ${JAILMAKER_DEBIAN_RELEASE} stable" | tee /etc/apt/sources.list.d/docker.list >/dev/null

    apt update
    apt -y --no-install-recommends install docker-ce docker-ce-cli containerd.io docker-compose-plugin
    fi

    if [[ "${JAILMAKER_INSTALL_NVIDIA}" -eq 1 ]]; then
    # TODO: document alternative method of installing nvidia drivers (via apt from the host)
    # with --bind-ro=/etc/apt (also cleanup apt after installing: rm -rf /var/lib/apt/lists/*)
    # Should I `hold` these packages once installed?
    # TODO: Investigate auto update command before starting the jail,
    # to ensure jail is always started with matching nvidia driver version

    apt -y --no-install-recommends install kmod

    local nvidia_version nvidia_url

    nvidia_version=$(modinfo nvidia-current --field version)
    nvidia_url="https://us.download.nvidia.com/XFree86/Linux-x86_64/${nvidia_version}/NVIDIA-Linux-x86_64-${nvidia_version}.run"

    curl -fL --output "/tmp/nvidia_driver.run" "${nvidia_url}" || fail "Failed to download and install nvidia driver.\nDetected version of the driver on TrueNAS host: ${nvidia_version}.\nAttempted download url: ${nvidia_url}."

    sh /tmp/nvidia_driver.run --ui=none --no-questions --no-kernel-modules
    rm /tmp/nvidia_driver.run

    if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then

    # Install container-toolkit according to official guide:
    # https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

    local distribution
    distribution="$(. /etc/os-release && echo "${ID}${VERSION_ID}")"

    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    apt update
    apt -y --no-install-recommends install nvidia-docker2
    fi
    fi
    }

    wizard() {
    # Reset variables to defaults (ignore env variables)
    JAILMAKER_INSTALL_DOCKER=0
    JAILMAKER_GPU_PASSTHROUGH=0
    JAILMAKER_INSTALL_NVIDIA=0
    JAILMAKER_DEBUG=0
    additional_flags=

    read -r -p "Do you want to create a new jail? [y/N]: " CHOICE
    if ! [[ "${CHOICE}" == 'y' || "${CHOICE}" == 'Y' ]]; then
    echo "${USAGE}" && exit
    fi

    echo "This wizard will walk you through the steps of creating a new jail."
    echo ""
    read -r -p "Enter a name for the jail: " JAILMAKER_JAIL_NAME
    # Fallback to default value if user pressed enter
    JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-${DEBIAN_RELEASE}}"
    echo ""
    echo "You may pass additional arguments to systemd-nspawn, e.g. to mount directories."
    echo "For example: --bind=/mnt/dir1 --bind-ro=/mnt/dir"
    echo "This would mount /mnt/dir with read/write access and /mnt/dir with readonly access:"
    echo ""
    read -r -p "Enter additional arguments for systemd-nspawn: " additional_flags
    echo ""
    echo "By default the jail will use host networking."
    echo "You may want to use a macvlan interface instead."
    echo "A macvlan interface is a virtual interface that adds a second MAC address to an existing physical Ethernet link."
    echo "With macvlan the jail can obtain a distinct IP address via DHCP from the same LAN the host is connected to."
    echo ""
    read -r -p "Name of the Ethernet network interface to create a macvlan interface from (leave blank to use host networking): " JAILMAKER_MACVLAN_PARENT_INTERFACE
    echo ""
    read -r -p "Do you want to install docker inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_DOCKER=1 ;;
    esac
    read -r -p "Do you want to access the GPU inside the jail? [y/N]: " CHOICE
    echo ""
    if [[ "${CHOICE}" == 'y' || "${CHOICE}" == 'Y' ]]; then
    JAILMAKER_GPU_PASSTHROUGH=1
    echo "It's possible to use an nvidia GPU if we install the correct driver version."
    echo "When a TrueNAS SCALE update changes the nvidia driver version, a driver mismatch will occur."
    echo "You'll need to MANUALLY update the nvidia driver inside the jail..."
    echo ""
    read -r -p "Do you want to install nvidia drivers inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_NVIDIA=1 ;;
    esac
    fi
    read -r -p "Do you want to see all the steps the jailmaker.sh script takes (debug mode)? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_DEBUG=1 ;;
    esac
    new "${additional_flags}"
    }

    case "${1-""}" in

    new)
    new "${*:2}"
    ;;

    '' | wizard)
    wizard
    ;;

    up)
    up
    ;;

    __do_not_run_this_manually__)
    complete_install_inside_jail
    ;;

    *)
    echo "${USAGE}"
    ;;
    esac
  3. Jip-Hop revised this gist Jan 14, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -190,7 +190,7 @@ new() {
    cat <<-'EOF' >"${jail_path}.sh"
    #!/bin/bash
    umask 077
    cd "$(dirname "${BASH_SOURCE[0]}")"
    cd "$(dirname "${BASH_SOURCE[0]}")" || exit
    EOF

    # Append the dynamic part of the startup shell script (with variable substitution)
  4. Jip-Hop revised this gist Jan 14, 2023. 1 changed file with 35 additions and 1 deletion.
    36 changes: 35 additions & 1 deletion jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -36,6 +36,7 @@ Environment variables:
    JAILMAKER_INSTALL_DOCKER=0
    JAILMAKER_INSTALL_NVIDIA=0
    JAILMAKER_GPU_PASSTHROUGH=0
    JAILMAKER_MACVLAN_PARENT_INTERFACE=
    Default settings can be overridden via environment variables, documentation about advanced settings is inside ${SCRIPT_NAME}
    @@ -53,6 +54,9 @@ JAILMAKER_GPU_PASSTHROUGH=1 ./${SCRIPT_NAME} new
    # Start a new jail, install docker and enable debug mode to output all the steps this script takes
    JAILMAKER_INSTALL_DOCKER=1 JAILMAKER_DEBUG=1 ./${SCRIPT_NAME} new
    # Start a new jail with macvlan networking (container will get its own IP address via DHCP)
    JAILMAKER_MACVLAN_PARENT_INTERFACE=eno1 ./${SCRIPT_NAME} new
    # Bring up all jails by running all *.sh files in the jails dir
    ./${SCRIPT_NAME} up
    "
    @@ -73,6 +77,8 @@ fail() {
    JAILMAKER_DEBUG="${JAILMAKER_DEBUG:-0}" && [[ "${JAILMAKER_DEBUG}" -eq 1 ]] && set -x
    # Default name for the jail is the debian release name, e.g. bullseye - valid values: a string without the '/' character in it
    JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-${DEBIAN_RELEASE}}"
    # By default we will use host networking - valid values: '' (default), name of a network interface
    JAILMAKER_MACVLAN_PARENT_INTERFACE="${JAILMAKER_MACVLAN_PARENT_INTERFACE:-}"
    # By default we will not install docker inside the jail - valid values: 0 (default), 1
    JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER:-0}"
    # By default we will not install nvidia drivers - valid values: 0 (default), 1
    @@ -140,11 +146,16 @@ new() {

    # Bind mount this script inside the jail and run it with the '__do_not_run_this_manually__' command
    # This will install software inside the jail
    systemd-nspawn -q -D "${jail_path}" --bind-ro="${ABSOLUTE_SCRIPT_PATH}" --bind-ro=/lib/modules -E JAILMAKER_DEBUG="${JAILMAKER_DEBUG}" -E JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE}" -E JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER}" -E JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA}" "${ABSOLUTE_SCRIPT_PATH}" __do_not_run_this_manually__
    systemd-nspawn -q -D "${jail_path}" --bind-ro="${ABSOLUTE_SCRIPT_PATH}" --bind-ro=/lib/modules -E JAILMAKER_MACVLAN_PARENT_INTERFACE="${JAILMAKER_MACVLAN_PARENT_INTERFACE}" -E JAILMAKER_DEBUG="${JAILMAKER_DEBUG}" -E JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE}" -E JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER}" -E JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA}" "${ABSOLUTE_SCRIPT_PATH}" __do_not_run_this_manually__

    # Start with the flags passed as argument
    additional_flags="$1"

    # Add flag for macvlan network
    if [[ -n "${JAILMAKER_MACVLAN_PARENT_INTERFACE}" ]]; then
    additional_flags+=" --network-macvlan=${JAILMAKER_MACVLAN_PARENT_INTERFACE}"
    fi

    # Add flags for GPU access
    if [[ "${JAILMAKER_GPU_PASSTHROUGH}" -eq 1 ]]; then

    @@ -229,6 +240,22 @@ complete_install_inside_jail() {
    # and systemd-sysv to for the `poweroff` command
    apt -y --no-install-recommends install dbus systemd systemd-sysv

    if [[ -n "${JAILMAKER_MACVLAN_PARENT_INTERFACE}" ]]; then
    # Not using host networking, so enable networkd and resolved services
    systemctl enable systemd-networkd
    systemctl enable systemd-resolved

    # Setup DHCP for macvlan network interfaces
    # https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_without_gui
    cat <<-'EOF' >/etc/systemd/network/dhcp.network
    [Match]
    Name=mv-en*
    [Network]
    DHCP=yes
    EOF
    fi

    if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then

    # Install docker according to official guide:
    @@ -305,6 +332,13 @@ wizard() {
    echo ""
    read -r -p "Enter additional arguments for systemd-nspawn: " additional_flags
    echo ""
    echo "By default the jail will use host networking."
    echo "You may want to use a macvlan interface instead."
    echo "A macvlan interface is a virtual interface that adds a second MAC address to an existing physical Ethernet link."
    echo "With macvlan the jail can obtain a distinct IP address via DHCP from the same LAN the host is connected to."
    echo ""
    read -r -p "Name of the Ethernet network interface to create a macvlan interface from (leave blank to use host networking): " JAILMAKER_MACVLAN_PARENT_INTERFACE
    echo ""
    read -r -p "Do you want to install docker inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
  5. Jip-Hop revised this gist Jan 13, 2023. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -8,4 +8,6 @@ Create a persistent 'jail' on TrueNAS SCALE to install software, such as docker,
    - Go to a directory where you want to store jailmaker, should be on one of your ZFS datasets (under the /mnt/ directory)
    - Clone this gist (this saves the code inside a directory named `jailmaker`): `git clone -b main --single-branch --depth 1 https://gist.github.com/Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d jailmaker`
    - Change into the `jailmaker` directory
    - Run `./jailmaker.sh` for an interactive installation or run `./jailmaker.sh help` for all available options.
    - Run `./jailmaker.sh` for an interactive installation or run `./jailmaker.sh help` for all available options.

    Please [vote for official systemd-nspawn support](https://ixsystems.atlassian.net/browse/NAS-119787) (login first, then click the thumbs up button in the top right).
  6. Jip-Hop revised this gist Jan 8, 2023. 1 changed file with 1 addition and 3 deletions.
    4 changes: 1 addition & 3 deletions jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -88,8 +88,6 @@ JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE:-${DEBIAN_RELEASE}}"
    JAILMAKER_JAILS_DIR="${JAILMAKER_JAILS_DIR:-${SCRIPT_PARENT_DIR}/jails}"
    # Only allow creating jails inside an /mnt sub-folder, unless safety check if off - valid values: 0, 1 (default)
    JAILMAKER_JAILS_DIR_SAFETY_CHECK="${JAILMAKER_JAILS_DIR_SAFETY_CHECK:-1}"
    # The default command which will be used to start the jail from the .sh file - you should probably not be modifying this and instead pass additional flags to the new command
    JAILMAKER_NSPAWN_BASE_CMD="${JAILMAKER_NSPAWN_BASE_CMD:-systemd-nspawn --quiet --boot --directory=}"

    # Since this file will be executed as root, we need to set appropriate permissions (if not already set)
    [[ "$(stat -c%a "${ABSOLUTE_SCRIPT_PATH}")" -ne 700 ]] && chmod 700 "${ABSOLUTE_SCRIPT_PATH}"
    @@ -185,7 +183,7 @@ new() {
    EOF

    # Append the dynamic part of the startup shell script (with variable substitution)
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 ${JAILMAKER_NSPAWN_BASE_CMD}'${relative_jail_path}'${additional_flags} &> '${relative_jail_path}.log' &" >>"${jail_path}.sh"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --quiet --boot --directory='${relative_jail_path}'${additional_flags} &> '${relative_jail_path}.log' &" >>"${jail_path}.sh"

    # Make executable and allow access only for root user
    chmod 700 "${jail_path}.sh"
  7. Jip-Hop revised this gist Jan 8, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -8,4 +8,4 @@ Create a persistent 'jail' on TrueNAS SCALE to install software, such as docker,
    - Go to a directory where you want to store jailmaker, should be on one of your ZFS datasets (under the /mnt/ directory)
    - Clone this gist (this saves the code inside a directory named `jailmaker`): `git clone -b main --single-branch --depth 1 https://gist.github.com/Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d jailmaker`
    - Change into the `jailmaker` directory
    - Run the `jailmaker.sh` script for an interactive installation or run `jailmaker.sh help` for all available options.
    - Run `./jailmaker.sh` for an interactive installation or run `./jailmaker.sh help` for all available options.
  8. Jip-Hop revised this gist Jan 8, 2023. 3 changed files with 63 additions and 51 deletions.
    8 changes: 5 additions & 3 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,8 @@

    Create a persistent 'jail' on TrueNAS SCALE to install software, such as docker, without modifying the host OS.

    Download this gist as a zip and put it on your TrueNAS machine on one of your ZFS datasets (under the /mnt/ directory).

    Run the `wizard.sh` script for an interactive installation or run `jailmaker.sh` for help on how to use.
    - Connect to your TrueNAS via SSH as root user
    - Go to a directory where you want to store jailmaker, should be on one of your ZFS datasets (under the /mnt/ directory)
    - Clone this gist (this saves the code inside a directory named `jailmaker`): `git clone -b main --single-branch --depth 1 https://gist.github.com/Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d jailmaker`
    - Change into the `jailmaker` directory
    - Run the `jailmaker.sh` script for an interactive installation or run `jailmaker.sh help` for all available options.
    59 changes: 58 additions & 1 deletion jailmaker.sh
    100644 → 100755
    View file
    Original file line number Diff line number Diff line change
    @@ -23,6 +23,7 @@ Usage: [ENV_VAR=value] ./${SCRIPT_NAME} COMMAND [ARG...]
    Commands:
    new Create a new jail in the jails dir and start it
    wizard Interactive guide to create and start a new jail
    up Bring up all jails by running all *.sh files in the jails dir
    help Show this help message
    @@ -85,7 +86,7 @@ JAILMAKER_GPU_PASSTHROUGH="${JAILMAKER_GPU_PASSTHROUGH:-${JAILMAKER_INSTALL_NVID
    JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE:-${DEBIAN_RELEASE}}"
    # By default we will use (or create) the 'jails' dir inside this script's parent directory - valid values: a path to a directory (parent directory should exist)
    JAILMAKER_JAILS_DIR="${JAILMAKER_JAILS_DIR:-${SCRIPT_PARENT_DIR}/jails}"
    # Only allow creating jails inside a /mnt sub-folder, unless safety check if off - valid values: 0, 1 (default)
    # Only allow creating jails inside an /mnt sub-folder, unless safety check if off - valid values: 0, 1 (default)
    JAILMAKER_JAILS_DIR_SAFETY_CHECK="${JAILMAKER_JAILS_DIR_SAFETY_CHECK:-1}"
    # The default command which will be used to start the jail from the .sh file - you should probably not be modifying this and instead pass additional flags to the new command
    JAILMAKER_NSPAWN_BASE_CMD="${JAILMAKER_NSPAWN_BASE_CMD:-systemd-nspawn --quiet --boot --directory=}"
    @@ -281,12 +282,68 @@ complete_install_inside_jail() {
    fi
    }

    wizard() {
    # Reset variables to defaults (ignore env variables)
    JAILMAKER_INSTALL_DOCKER=0
    JAILMAKER_GPU_PASSTHROUGH=0
    JAILMAKER_INSTALL_NVIDIA=0
    JAILMAKER_DEBUG=0
    additional_flags=

    read -r -p "Do you want to create a new jail? [y/N]: " CHOICE
    if ! [[ "${CHOICE}" == 'y' || "${CHOICE}" == 'Y' ]]; then
    echo "${USAGE}" && exit
    fi

    echo "This wizard will walk you through the steps of creating a new jail."
    echo ""
    read -r -p "Enter a name for the jail: " JAILMAKER_JAIL_NAME
    # Fallback to default value if user pressed enter
    JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-${DEBIAN_RELEASE}}"
    echo ""
    echo "You may pass additional arguments to systemd-nspawn, e.g. to mount directories."
    echo "For example: --bind=/mnt/dir1 --bind-ro=/mnt/dir"
    echo "This would mount /mnt/dir with read/write access and /mnt/dir with readonly access:"
    echo ""
    read -r -p "Enter additional arguments for systemd-nspawn: " additional_flags
    echo ""
    read -r -p "Do you want to install docker inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_DOCKER=1 ;;
    esac
    read -r -p "Do you want to access the GPU inside the jail? [y/N]: " CHOICE
    echo ""
    if [[ "${CHOICE}" == 'y' || "${CHOICE}" == 'Y' ]]; then
    JAILMAKER_GPU_PASSTHROUGH=1
    echo "It's possible to use an nvidia GPU if we install the correct driver version."
    echo "When a TrueNAS SCALE update changes the nvidia driver version, a driver mismatch will occur."
    echo "You'll need to MANUALLY update the nvidia driver inside the jail..."
    echo ""
    read -r -p "Do you want to install nvidia drivers inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_NVIDIA=1 ;;
    esac
    fi
    read -r -p "Do you want to see all the steps the jailmaker.sh script takes (debug mode)? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_DEBUG=1 ;;
    esac
    new "${additional_flags}"
    }

    case "${1-""}" in

    new)
    new "${*:2}"
    ;;

    '' | wizard)
    wizard
    ;;

    up)
    up
    ;;
    47 changes: 0 additions & 47 deletions wizard.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,47 +0,0 @@
    #!/bin/bash

    set -euo pipefail

    JAILMAKER_INSTALL_DOCKER=0
    JAILMAKER_GPU_PASSTHROUGH=0
    JAILMAKER_INSTALL_NVIDIA=0
    JAILMAKER_DEBUG=0
    ADDITIONAL_FLAGS=

    echo "This wizard will walk you through the steps of creating a new jail."
    echo ""
    read -r -p "Enter a name for the jail: " JAILMAKER_JAIL_NAME
    echo ""
    echo "You may pass additional arguments to systemd-nspawn, e.g. to mount directories."
    echo "For example: --bind=/mnt/dir1 --bind-ro=/mnt/dir"
    echo "This would mount /mnt/dir with read/write access and /mnt/dir with readonly access:"
    echo ""
    read -r -p "Enter additional arguments for systemd-nspawn: " ADDITIONAL_FLAGS
    echo ""
    read -r -p "Do you want to install docker inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_DOCKER=1 ;;
    esac
    read -r -p "Do you want to access the GPU inside the jail? [y/N]: " CHOICE
    echo ""
    if [[ "${CHOICE}" == 'y' || "${CHOICE}" == 'Y' ]]; then
    JAILMAKER_GPU_PASSTHROUGH=1
    echo "It's possible to use an nvidia GPU if we install the correct driver version."
    echo "When a TrueNAS SCALE update changes the nvidia driver version, a driver mismatch will occur."
    echo "You'll need to MANUALLY update the nvidia driver inside the jail..."
    echo ""
    read -r -p "Do you want to install nvidia drivers inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_NVIDIA=1 ;;
    esac
    fi
    read -r -p "Do you want to see all the steps the jailmaker.sh script takes (debug mode)? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_DEBUG=1 ;;
    esac
    echo "We will now pass your input to the jailmaker.sh script."
    export JAILMAKER_DEBUG JAILMAKER_JAIL_NAME JAILMAKER_INSTALL_DOCKER JAILMAKER_INSTALL_NVIDIA JAILMAKER_GPU_PASSTHROUGH
    ./jailmaker.sh new "${ADDITIONAL_FLAGS}"
  9. Jip-Hop revised this gist Jan 7, 2023. 1 changed file with 9 additions and 0 deletions.
    9 changes: 9 additions & 0 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,9 @@
    # Jailmaker

    ### Work in progress, experimental, for testing only! No warranty!

    Create a persistent 'jail' on TrueNAS SCALE to install software, such as docker, without modifying the host OS.

    Download this gist as a zip and put it on your TrueNAS machine on one of your ZFS datasets (under the /mnt/ directory).

    Run the `wizard.sh` script for an interactive installation or run `jailmaker.sh` for help on how to use.
  10. Jip-Hop revised this gist Jan 7, 2023. 1 changed file with 47 additions and 0 deletions.
    47 changes: 47 additions & 0 deletions wizard.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,47 @@
    #!/bin/bash

    set -euo pipefail

    JAILMAKER_INSTALL_DOCKER=0
    JAILMAKER_GPU_PASSTHROUGH=0
    JAILMAKER_INSTALL_NVIDIA=0
    JAILMAKER_DEBUG=0
    ADDITIONAL_FLAGS=

    echo "This wizard will walk you through the steps of creating a new jail."
    echo ""
    read -r -p "Enter a name for the jail: " JAILMAKER_JAIL_NAME
    echo ""
    echo "You may pass additional arguments to systemd-nspawn, e.g. to mount directories."
    echo "For example: --bind=/mnt/dir1 --bind-ro=/mnt/dir"
    echo "This would mount /mnt/dir with read/write access and /mnt/dir with readonly access:"
    echo ""
    read -r -p "Enter additional arguments for systemd-nspawn: " ADDITIONAL_FLAGS
    echo ""
    read -r -p "Do you want to install docker inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_DOCKER=1 ;;
    esac
    read -r -p "Do you want to access the GPU inside the jail? [y/N]: " CHOICE
    echo ""
    if [[ "${CHOICE}" == 'y' || "${CHOICE}" == 'Y' ]]; then
    JAILMAKER_GPU_PASSTHROUGH=1
    echo "It's possible to use an nvidia GPU if we install the correct driver version."
    echo "When a TrueNAS SCALE update changes the nvidia driver version, a driver mismatch will occur."
    echo "You'll need to MANUALLY update the nvidia driver inside the jail..."
    echo ""
    read -r -p "Do you want to install nvidia drivers inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_NVIDIA=1 ;;
    esac
    fi
    read -r -p "Do you want to see all the steps the jailmaker.sh script takes (debug mode)? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_DEBUG=1 ;;
    esac
    echo "We will now pass your input to the jailmaker.sh script."
    export JAILMAKER_DEBUG JAILMAKER_JAIL_NAME JAILMAKER_INSTALL_DOCKER JAILMAKER_INSTALL_NVIDIA JAILMAKER_GPU_PASSTHROUGH
    ./jailmaker.sh new "${ADDITIONAL_FLAGS}"
  11. Jip-Hop revised this gist Jan 7, 2023. No changes.
  12. Jip-Hop revised this gist Jan 7, 2023. No changes.
  13. Jip-Hop revised this gist Jan 7, 2023. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -23,7 +23,7 @@ Usage: [ENV_VAR=value] ./${SCRIPT_NAME} COMMAND [ARG...]
    Commands:
    new Create a new jail in the jails dir and start it
    up Bring all jails up by running the .sh files in the jails dir
    up Bring up all jails by running all *.sh files in the jails dir
    help Show this help message
    All arguments passed to the new command will be added to the systemd-nspawn command that starts the jail
    @@ -40,19 +40,19 @@ Default settings can be overridden via environment variables, documentation abou
    Examples:
    # Create + start a new jail with read/write access to /mnt/ssd/appdata and readonly access to /mnt/Tank/Media
    # Start a new jail with read/write access to /mnt/ssd/appdata and readonly access to /mnt/Tank/Media
    ./${SCRIPT_NAME} new --bind=/mnt/ssd/appdata --bind-ro=/mnt/Tank/Media
    # Create + start a new jail named: my-jail-name
    # Start a new jail named: my-jail-name
    JAILMAKER_JAIL_NAME=my-jail-name ./${SCRIPT_NAME} new
    # Create + start a new jail with GPU passthrough set-up (if a GPU device can be found during creation)
    # Start a new jail with GPU passthrough set-up (if a GPU device can be found during creation)
    JAILMAKER_GPU_PASSTHROUGH=1 ./${SCRIPT_NAME} new
    # Create + start a new jail, enable debug mode to output all the steps this script makes and install docker
    JAILMAKER_DEBUG=1 JAILMAKER_INSTALL_DOCKER=1 ./${SCRIPT_NAME} new
    # Start a new jail, install docker and enable debug mode to output all the steps this script takes
    JAILMAKER_INSTALL_DOCKER=1 JAILMAKER_DEBUG=1 ./${SCRIPT_NAME} new
    # Runs all the *.sh files in the jails dir to start the jails
    # Bring up all jails by running all *.sh files in the jails dir
    ./${SCRIPT_NAME} up
    "

  14. Jip-Hop revised this gist Jan 7, 2023. 1 changed file with 115 additions and 99 deletions.
    214 changes: 115 additions & 99 deletions jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,52 +1,99 @@
    #!/bin/bash

    # WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!

    # Create Debian 'jails' with systemd-nspawn
    # I suggest to create a new, dedicated, dataset using the TrueNAS SCALE web ui first
    # Then put this script inside this dataset and run it as root

    set -euo pipefail
    shopt -s nullglob

    ABSOLUTE_SCRIPT_PATH="$(realpath "${BASH_SOURCE[0]}")"
    SCRIPT_NAME=$(basename "${ABSOLUTE_SCRIPT_PATH}")
    SCRIPT_PARENT_DIR="$(dirname "$ABSOLUTE_SCRIPT_PATH")"
    DEBIAN_RELEASE="$(. /etc/os-release && echo "${VERSION_CODENAME}")"

    USAGE="WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!
    Create persistent Debian 'jails' with systemd-nspawn on TrueNAS SCALE
    Allows installing software, such as docker, without modifying the host OS
    With full access to all files on the host via bind mounts
    Put this script on one of your ZFS datasets (under the /mnt/ directory)
    It will create a 'jails' dir next to it, where it stores all the jails
    Run this script as root user
    Usage: [ENV_VAR=value] ./${SCRIPT_NAME} COMMAND [ARG...]
    Commands:
    new Create a new jail in the jails dir and start it
    up Bring all jails up by running the .sh files in the jails dir
    help Show this help message
    All arguments passed to the new command will be added to the systemd-nspawn command that starts the jail
    For available arguments see: https://manpages.debian.org/${DEBIAN_RELEASE}/systemd-container/systemd-nspawn.1.en.html
    Environment variables:
    JAILMAKER_DEBUG=0
    JAILMAKER_JAIL_NAME=${DEBIAN_RELEASE}
    JAILMAKER_INSTALL_DOCKER=0
    JAILMAKER_INSTALL_NVIDIA=0
    JAILMAKER_GPU_PASSTHROUGH=0
    Default settings can be overridden via environment variables, documentation about advanced settings is inside ${SCRIPT_NAME}
    Examples:
    # Create + start a new jail with read/write access to /mnt/ssd/appdata and readonly access to /mnt/Tank/Media
    ./${SCRIPT_NAME} new --bind=/mnt/ssd/appdata --bind-ro=/mnt/Tank/Media
    # Create + start a new jail named: my-jail-name
    JAILMAKER_JAIL_NAME=my-jail-name ./${SCRIPT_NAME} new
    # Create + start a new jail with GPU passthrough set-up (if a GPU device can be found during creation)
    JAILMAKER_GPU_PASSTHROUGH=1 ./${SCRIPT_NAME} new
    # Create + start a new jail, enable debug mode to output all the steps this script makes and install docker
    JAILMAKER_DEBUG=1 JAILMAKER_INSTALL_DOCKER=1 ./${SCRIPT_NAME} new
    # Runs all the *.sh files in the jails dir to start the jails
    ./${SCRIPT_NAME} up
    "

    fail() {
    echo -e "$1" >&2 && exit 1
    }

    [[ $UID -ne 0 ]] && fail "Run this script as root"

    ABSOLUTE_SCRIPT_PATH=$(realpath "${BASH_SOURCE[0]}")
    SCRIPT_PARENT_DIR=$(dirname "$ABSOLUTE_SCRIPT_PATH")
    MODE=${1-""}
    [[ $UID -ne 0 ]] && echo "${USAGE}" && fail "Run this script as root..."

    ####################################################################
    # Read from user-defined environment variables or use default values
    ####################################################################

    # Print each line this script executes if debug has been requested (off by default)
    # Basic settings, should be safe to override

    # Print each line this script executes in debug mode - valid values: 0 (default), 1
    JAILMAKER_DEBUG="${JAILMAKER_DEBUG:-0}" && [[ "${JAILMAKER_DEBUG}" -eq 1 ]] && set -x
    # By default we will install docker inside the jail
    JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER:-1}"
    # By default we will not install nvidia drivers
    # Default name for the jail is the debian release name, e.g. bullseye - valid values: a string without the '/' character in it
    JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-${DEBIAN_RELEASE}}"
    # By default we will not install docker inside the jail - valid values: 0 (default), 1
    JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER:-0}"
    # By default we will not install nvidia drivers - valid values: 0 (default), 1
    JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA:-0}"
    # By default we will not setup GPU passthrough (unless we're installing nvidia drivers)
    JAILMAKER_GPU_PASSTHROUGH="${JAILMAKER_GPU_PASSTHROUGH:-$JAILMAKER_INSTALL_NVIDIA}"
    # The default command which will be used to start the jail from within a systemd .service file
    JAILMAKER_NSPAWN_BASE_CMD="${JAILMAKER_NSPAWN_BASE_CMD:-systemd-nspawn --quiet --keep-unit --boot --directory=}"
    # By default make a jail which matches the debian version of TrueNAS
    JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE:-$(lsb_release -cs)}"
    # By default we will use (or create) the 'jails' dir inside this script's parent directory
    # By default we will not setup GPU passthrough (unless we're installing nvidia drivers) - valid values: 0, 1
    JAILMAKER_GPU_PASSTHROUGH="${JAILMAKER_GPU_PASSTHROUGH:-${JAILMAKER_INSTALL_NVIDIA}}"

    # Advanced settings, be careful if you change these!

    # By default make a jail which matches the debian version of TrueNAS - valid values: a debian release name (only bullseye has been tested)
    JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE:-${DEBIAN_RELEASE}}"
    # By default we will use (or create) the 'jails' dir inside this script's parent directory - valid values: a path to a directory (parent directory should exist)
    JAILMAKER_JAILS_DIR="${JAILMAKER_JAILS_DIR:-${SCRIPT_PARENT_DIR}/jails}"
    # Only allow creating jails inside a /mnt sub-folder, unless safety check if off
    # Only allow creating jails inside a /mnt sub-folder, unless safety check if off - valid values: 0, 1 (default)
    JAILMAKER_JAILS_DIR_SAFETY_CHECK="${JAILMAKER_JAILS_DIR_SAFETY_CHECK:-1}"
    # Default name for the jail is the debian codename, e.g. bullseye with jailmaker- prefix]
    JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-jailmaker-${JAILMAKER_DEBIAN_RELEASE}}"
    # The default command which will be used to start the jail from the .sh file - you should probably not be modifying this and instead pass additional flags to the new command
    JAILMAKER_NSPAWN_BASE_CMD="${JAILMAKER_NSPAWN_BASE_CMD:-systemd-nspawn --quiet --boot --directory=}"

    # Since this file will be executed as root, we need to set appropriate permissions (if not already set)
    [[ "$(stat -c%a "${ABSOLUTE_SCRIPT_PATH}")" -ne 700 ]] && chmod 700 "${ABSOLUTE_SCRIPT_PATH}"

    validate_input() {

    local jails_dir_parent_dir
    jails_dir_parent_dir="$(dirname "${JAILMAKER_JAILS_DIR}")"

    @@ -62,23 +109,10 @@ validate_input() {
    fi
    }

    deploy_jail() {
    local name
    name=$(basename "${1}")

    # Copy the .service file to the root filesystem
    # Making a symlink won't work because the /mnt directory isn't available when systemd starts
    install --mode=644 --owner=0 --group=0 "${1}" /etc/systemd/system/"${name}"
    # Register this jail with systemd (so it will run every time we boot TrueNAS)
    systemctl enable "${name}"
    # Start the jail
    systemctl start "${name}"
    }

    create_jail() {
    new() {
    validate_input

    local arch original_jail_name x jail_path additional_flags
    local arch original_jail_name x jail_path relative_jail_path additional_flags
    arch=$(dpkg --print-architecture)

    # Create the jails dir if it does not exist
    @@ -95,6 +129,7 @@ create_jail() {
    while [[ -d "${JAILMAKER_JAILS_DIR}/${jail_name}" ]]; do jail_name="${original_jail_name}-$((x++))"; done

    jail_path="${JAILMAKER_JAILS_DIR}/${jail_name}"

    # Make the new jail directory
    mkdir "${jail_path}"

    @@ -106,7 +141,7 @@ create_jail() {

    # Bind mount this script inside the jail and run it with the '__do_not_run_this_manually__' command
    # This will install software inside the jail
    systemd-nspawn -q -D "${jail_path}" --bind-ro="${ABSOLUTE_SCRIPT_PATH}" --bind-ro=/lib/modules -E JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE}" -E JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER}" -E JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA}" ".${ABSOLUTE_SCRIPT_PATH}" __do_not_run_this_manually__
    systemd-nspawn -q -D "${jail_path}" --bind-ro="${ABSOLUTE_SCRIPT_PATH}" --bind-ro=/lib/modules -E JAILMAKER_DEBUG="${JAILMAKER_DEBUG}" -E JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE}" -E JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER}" -E JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA}" "${ABSOLUTE_SCRIPT_PATH}" __do_not_run_this_manually__

    # Start with the flags passed as argument
    additional_flags="$1"
    @@ -132,39 +167,30 @@ create_jail() {
    [[ -n "${additional_flags}" && "${additional_flags}" != " "* ]] && additional_flags=" ${additional_flags}"

    # Use SYSTEMD_SECCOMP=0: https://github.com/systemd/systemd/issues/18370
    # Use SYSTEMD_NSPAWN_LOCK=0: otherwise jail won't start jail after a shutdown
    # Would give "directory tree currently busy" error
    # Fix: `rm /run/systemd/nspawn/locks/*` and remove the .lck file from JAILMAKER_JAILS_DIR
    # TODO: figure out why I have to use SYSTEMD_NSPAWN_LOCK=0

    # Write the systemd service file
    cat <<-EOF >"${jail_path}.service"
    [Unit]
    Description=${jail_name} [created with jailmaker]
    Documentation=man:systemd-nspawn(1)
    After=network.target systemd-resolved.service
    [Service]
    Environment="SYSTEMD_SECCOMP=0"
    Environment="SYSTEMD_NSPAWN_LOCK=0"
    ExecStart=${JAILMAKER_NSPAWN_BASE_CMD}'${jail_path}'${additional_flags}
    KillMode=mixed
    Type=notify
    RestartForceExitStatus=133
    SuccessExitStatus=133
    Delegate=yes
    TasksMax=16384
    WatchdogSec=3min
    [Install]
    WantedBy=machines.target
    # Use SYSTEMD_NSPAWN_LOCK=0: otherwise jail won't start jail after a shutdown (but why?)
    # Would give "directory tree currently busy" error and I'd have to run
    # `rm /run/systemd/nspawn/locks/*` and remove the .lck file from JAILMAKER_JAILS_DIR
    # Disabling locking isn't a big deal as systemd-nspawn will prevent starting a container
    # with the same name anyway: as long as we're starting jails using the accompanying .sh script,
    # it won't be possible to start the same jail twice

    relative_jail_path=$(realpath --relative-to="${JAILMAKER_JAILS_DIR}" "${jail_path}")

    # Write the static part of the startup shell script (no variable substitution)
    cat <<-'EOF' >"${jail_path}.sh"
    #!/bin/bash
    umask 077
    cd "$(dirname "${BASH_SOURCE[0]}")"
    EOF

    # Use appropriate permission for the .service file
    # https://unix.stackexchange.com/a/433910/477308
    chmod 644 "${jail_path}.service"
    # Enable the service and start the jail
    deploy_jail "${jail_path}.service"
    # Append the dynamic part of the startup shell script (with variable substitution)
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 ${JAILMAKER_NSPAWN_BASE_CMD}'${relative_jail_path}'${additional_flags} &> '${relative_jail_path}.log' &" >>"${jail_path}.sh"

    # Make executable and allow access only for root user
    chmod 700 "${jail_path}.sh"

    # Start the jail
    "${jail_path}.sh"

    cat <<-EOF
    The jail will now boot in the background...
    @@ -178,18 +204,18 @@ create_jail() {
    To poweroff a jail (from inside the jail):
    poweroff
    To start the jail again, run:
    systemctl start ${jail_name}
    In order to start the jail automatically after TrueNAS boot, configure '${jail_path}.sh' as Post Init Script from the TrueNAS web interface.
    In order to start all jails automatically after TrueNAS boot, configure '${ABSOLUTE_SCRIPT_PATH} up' as Post Init Script from the TrueNAS web interface.
    EOF
    }

    # This function needs to be called on each boot of TrueNAS, since after an OS update
    # our systemd services are no longer registered
    sync_jails() {
    # This function may be called on each boot of TrueNAS to start all jails
    up() {
    validate_input
    [[ -d "${JAILMAKER_JAILS_DIR}" ]] || fail "Nothing to sync: there are no jails in '${JAILMAKER_JAILS_DIR}'."
    # Find all the *.service files in the jails directory, and deploy them
    for d in "${JAILMAKER_JAILS_DIR}/"*.service; do deploy_jail "${d}"; done
    [[ -d "${JAILMAKER_JAILS_DIR}" ]] || fail "Nothing to start: the '${JAILMAKER_JAILS_DIR}' directory does not exist."
    # Find all the *.sh files in the jails directory, and start them
    for d in "${JAILMAKER_JAILS_DIR}/"*.sh; do "${d}" || true; done
    }

    complete_install_inside_jail() {
    @@ -224,7 +250,7 @@ complete_install_inside_jail() {
    # TODO: document alternative method of installing nvidia drivers (via apt from the host)
    # with --bind-ro=/etc/apt (also cleanup apt after installing: rm -rf /var/lib/apt/lists/*)
    # Should I `hold` these packages once installed?
    # TODO: Investigate auto update command (from inside the .service file) before starting the jail,
    # TODO: Investigate auto update command before starting the jail,
    # to ensure jail is always started with matching nvidia driver version

    apt -y --no-install-recommends install kmod
    @@ -245,11 +271,7 @@ complete_install_inside_jail() {
    # https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

    local distribution

    distribution=$(
    . /etc/os-release
    echo "$ID$VERSION_ID"
    )
    distribution="$(. /etc/os-release && echo "${ID}${VERSION_ID}")"

    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    @@ -259,27 +281,21 @@ complete_install_inside_jail() {
    fi
    }

    case "${MODE}" in

    create)
    create_jail "${*:2}"
    ;;
    case "${1-""}" in

    sync)
    sync_jails
    new)
    new "${*:2}"
    ;;

    cleanup)
    echo "TODO: some kind of cleanup to disable/remove from systemctl the services of jails we have removed"
    # systemctl stop '*jailmaker*'
    # Still has to to `systemctl disable` for each individual jailmaker service (glob not supported for disable)
    up)
    up
    ;;

    __do_not_run_this_manually__)
    complete_install_inside_jail
    ;;

    *)
    echo "TODO: print usage instructions"
    echo "${USAGE}"
    ;;
    esac
  15. Jip-Hop revised this gist Jan 7, 2023. 1 changed file with 5 additions and 3 deletions.
    8 changes: 5 additions & 3 deletions jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -65,8 +65,12 @@ validate_input() {
    deploy_jail() {
    local name
    name=$(basename "${1}")

    # Copy the .service file to the root filesystem
    # Making a symlink won't work because the /mnt directory isn't available when systemd starts
    install --mode=644 --owner=0 --group=0 "${1}" /etc/systemd/system/"${name}"
    # Register this jail with systemd (so it will run every time we boot TrueNAS)
    systemctl enable "${1}"
    systemctl enable "${name}"
    # Start the jail
    systemctl start "${name}"
    }
    @@ -133,8 +137,6 @@ create_jail() {
    # Fix: `rm /run/systemd/nspawn/locks/*` and remove the .lck file from JAILMAKER_JAILS_DIR
    # TODO: figure out why I have to use SYSTEMD_NSPAWN_LOCK=0

    # TODO: fix service not starting after reboot of TrueNAS. Needs to wait for zfs mount to complete?

    # Write the systemd service file
    cat <<-EOF >"${jail_path}.service"
    [Unit]
  16. Jip-Hop revised this gist Jan 6, 2023. 2 changed files with 283 additions and 190 deletions.
    190 changes: 0 additions & 190 deletions install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,190 +0,0 @@
    #!/bin/bash

    # WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!

    # Creates a systemd-nspawn container with Debian
    # I suggest to create a new, dedicated, dataset using the TrueNAS SCALE web ui first
    # and use that as the destination argument to this script
    # Based on https://gist.github.com/sfan5/52aa53f5dca06ac3af30455b203d3404

    set -euox pipefail
    shopt -s nullglob

    dest=${1-""}
    install_docker=1
    install_nvidia_driver=1
    release=bullseye
    arch=amd64
    dirname=debian
    additional_flags=""

    if [ $UID -ne 0 ]; then
    echo "Run this script as root" >&2
    exit 1
    fi

    if [ -z "${dest}" ]; then
    echo "Usage: $0 <destination>" >&2
    exit 0
    fi

    dest=$(realpath "${dest}")

    if [ ! -d "${dest}" ]; then
    echo "Destination directory '$dest' DOES NOT exist"
    exit 1
    fi

    if [[ ${dest} != "/mnt/"* ]]; then
    echo "The destination path must begin with '/mnt/'"
    echo "Provided destination was '${dest}"
    exit 1
    fi

    # Check if we can find nvidia devices to bind inside the jail
    # If not, then skip installing nvidia drivers
    if [ $install_nvidia_driver -eq 1 ]; then

    nvidia_device_found=0

    for d in /dev/nvidia*; do
    echo "Found $d";
    nvidia_device_found=1
    additional_flags="$additional_flags --bind='$d'"
    done

    if [ $nvidia_device_found -eq 0 ]; then
    echo "Unable to find nvidia devices."
    echo "Has an nvidia GPU been installed?"
    echo "Skip installation of nvidia drivers in the jail."
    install_nvidia_driver=0
    else
    # Add flag for GPU access
    additional_flags="$additional_flags --property=DeviceAllow='char-drm rw'"
    fi

    fi

    # Detect intel GPU and if present add bind flag
    # TODO: what other flags are needed to enable intel GPU? --property=DeviceAllow='char-drm rw'?
    [ -d /dev/dri ] && additional_flags="$additional_flags --bind=/dev/dri"

    dest="${dest}/${dirname}"

    # TODO: auto-increment the dirname with a suffix until we find a folder name which doesn't exist yet
    # TODO: override dir name using env variable
    if [ -d "${dest}" ]; then
    echo "Install directory '${dest}' already exists"
    exit 1
    fi

    trap cleanup EXIT

    cleanup() {
    exit_code=$?
    set +x

    if [[ ${exit_code} -ne 0 ]]; then
    echo "Installation of the jail failed!"
    else
    echo "Now manually run the command below to start Debian:"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf'${additional_flags} &"
    echo "The jail will boot in the background."
    echo ""
    echo "Now start a shell with:"
    echo "machinectl shell ${dirname}"
    if [ $install_docker -eq 1 ]; then
    echo ""
    echo "You may now run docker containers, e.g. with:"
    echo "docker run -p 8080:80 nginx"
    fi
    echo ""
    echo "To poweroff run (from inside the jail):"
    echo "systemctl poweroff"
    echo ""
    echo "It's also possible to bind-mount other directories from the TrueNAS host into our Debian machine,"
    echo "so that these directories can be used inside docker containers."
    echo "Check the man pages for more info:"
    echo "https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html"
    fi
    }

    # TODO: make a parent directory, to which only root has read access
    # Otherwise users (besides root) on the host OS may be able to read/write files inside the jail rootfs
    mkdir -p "${dest}"

    curl -L "https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-${arch}/${release}/slim/rootfs.tar.xz" | tar -xJ -C "${dest}" --numeric-owner

    # Remove resolv.conf, systemd configures this
    rm "${dest}/etc/resolv.conf"

    systemd-nspawn -q -D "${dest}" /usr/bin/apt-get update
    systemd-nspawn -q -D "${dest}" /usr/bin/apt-get -y --no-install-recommends install systemd dbus

    if [ $install_docker -eq 1 ]; then

    # Install docker according to official guide:
    # https://docs.docker.com/engine/install/debian/

    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    apt -y --no-install-recommends install ca-certificates curl gnupg
    mkdir -p /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    echo "deb [arch=${arch} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian ${release} stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
    apt update
    apt -y --no-install-recommends install docker-ce docker-ce-cli containerd.io docker-compose-plugin
    EOF

    fi

    if [ $install_nvidia_driver -eq 1 ]; then

    nvidia_version=$(dpkg-query -W -f='${Version}\n' nvidia-smi | sed 's/\(.*\)-.*/\1/')
    nvidia_url="https://us.download.nvidia.com/XFree86/Linux-x86_64/${nvidia_version}/NVIDIA-Linux-x86_64-${nvidia_version}.run"

    # Download the nvidia driver
    if ! curl -L --output "${dest}/root/nvidia_driver.run" "${nvidia_url}"; then

    echo "Failed to download and install nvidia driver."
    [ $install_docker -eq 1 ] && echo "Skipped installing nvidia-container-toolkit."
    echo "Detected version of the driver on TrueNAS host: ${nvidia_version}."
    echo "Attempted download url: ${nvidia_url}."
    echo "Please manually install the nvidia software inside the jail."
    exit 1

    else

    # Install the nvidia driver in the jail
    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    apt -y --no-install-recommends install kmod
    sh /root/nvidia_driver.run --ui=none --no-questions --no-kernel-modules
    rm /root/nvidia_driver.run
    EOF

    if [ $install_docker -eq 1 ]; then

    # Install container-toolkit according to official guide:
    # https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

    distribution=$(
    . /etc/os-release
    echo "$ID$VERSION_ID"
    )

    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    apt update
    apt -y --no-install-recommends install nvidia-docker2
    EOF
    fi
    fi
    fi

    # TODO: figure out why I have to use SYSTEMD_NSPAWN_LOCK=0
    # Otherwise I keep locking the jail after a shutdown and have to run:
    # rm /run/systemd/nspawn/locks/*
    # And also remove the .\#debian.lck file next to the dir containing the jail rootfs
    283 changes: 283 additions & 0 deletions jailmaker.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,283 @@
    #!/bin/bash

    # WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!

    # Create Debian 'jails' with systemd-nspawn
    # I suggest to create a new, dedicated, dataset using the TrueNAS SCALE web ui first
    # Then put this script inside this dataset and run it as root

    set -euo pipefail
    shopt -s nullglob

    fail() {
    echo -e "$1" >&2 && exit 1
    }

    [[ $UID -ne 0 ]] && fail "Run this script as root"

    ABSOLUTE_SCRIPT_PATH=$(realpath "${BASH_SOURCE[0]}")
    SCRIPT_PARENT_DIR=$(dirname "$ABSOLUTE_SCRIPT_PATH")
    MODE=${1-""}

    ####################################################################
    # Read from user-defined environment variables or use default values
    ####################################################################

    # Print each line this script executes if debug has been requested (off by default)
    JAILMAKER_DEBUG="${JAILMAKER_DEBUG:-0}" && [[ "${JAILMAKER_DEBUG}" -eq 1 ]] && set -x
    # By default we will install docker inside the jail
    JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER:-1}"
    # By default we will not install nvidia drivers
    JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA:-0}"
    # By default we will not setup GPU passthrough (unless we're installing nvidia drivers)
    JAILMAKER_GPU_PASSTHROUGH="${JAILMAKER_GPU_PASSTHROUGH:-$JAILMAKER_INSTALL_NVIDIA}"
    # The default command which will be used to start the jail from within a systemd .service file
    JAILMAKER_NSPAWN_BASE_CMD="${JAILMAKER_NSPAWN_BASE_CMD:-systemd-nspawn --quiet --keep-unit --boot --directory=}"
    # By default make a jail which matches the debian version of TrueNAS
    JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE:-$(lsb_release -cs)}"
    # By default we will use (or create) the 'jails' dir inside this script's parent directory
    JAILMAKER_JAILS_DIR="${JAILMAKER_JAILS_DIR:-${SCRIPT_PARENT_DIR}/jails}"
    # Only allow creating jails inside a /mnt sub-folder, unless safety check if off
    JAILMAKER_JAILS_DIR_SAFETY_CHECK="${JAILMAKER_JAILS_DIR_SAFETY_CHECK:-1}"
    # Default name for the jail is the debian codename, e.g. bullseye with jailmaker- prefix]
    JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-jailmaker-${JAILMAKER_DEBIAN_RELEASE}}"

    # Since this file will be executed as root, we need to set appropriate permissions (if not already set)
    [[ "$(stat -c%a "${ABSOLUTE_SCRIPT_PATH}")" -ne 700 ]] && chmod 700 "${ABSOLUTE_SCRIPT_PATH}"

    validate_input() {

    local jails_dir_parent_dir
    jails_dir_parent_dir="$(dirname "${JAILMAKER_JAILS_DIR}")"

    # Check if working directory exists
    if [[ ! -d ${jails_dir_parent_dir} ]]; then
    fail "Destination directory '${jails_dir_parent_dir}' DOES NOT exist."
    # Check if working directory is inside the /mnt directory (so it won't be lost on TrueNAS OS updates)
    elif [[ "$JAILMAKER_JAILS_DIR_SAFETY_CHECK" -ne 0 && "${JAILMAKER_JAILS_DIR}" != /mnt/* ]]; then
    fail "The destination path must begin with '/mnt/'\nProvided destination was '${JAILMAKER_JAILS_DIR}.'"
    # Check for illegal characters in the directory name
    elif [[ "${JAILMAKER_JAIL_NAME}" == */* ]]; then
    fail "Name may not contain the '/' character.\nJAILMAKER_JAIL_NAME: ${JAILMAKER_JAIL_NAME}."
    fi
    }

    deploy_jail() {
    local name
    name=$(basename "${1}")
    # Register this jail with systemd (so it will run every time we boot TrueNAS)
    systemctl enable "${1}"
    # Start the jail
    systemctl start "${name}"
    }

    create_jail() {
    validate_input

    local arch original_jail_name x jail_path additional_flags
    arch=$(dpkg --print-architecture)

    # Create the jails dir if it does not exist
    [[ -d "${JAILMAKER_JAILS_DIR}" ]] || mkdir "${JAILMAKER_JAILS_DIR}"

    # Only root should be allowed access to this directory
    chmod 700 "${JAILMAKER_JAILS_DIR}"

    original_jail_name="${JAILMAKER_JAIL_NAME}"
    jail_name=${original_jail_name}
    x=1

    # Append counter to jail path if there already is a jail with this name
    while [[ -d "${JAILMAKER_JAILS_DIR}/${jail_name}" ]]; do jail_name="${original_jail_name}-$((x++))"; done

    jail_path="${JAILMAKER_JAILS_DIR}/${jail_name}"
    # Make the new jail directory
    mkdir "${jail_path}"

    # Download and extract the root filesystem for the jail
    curl -L "https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-${arch}/${JAILMAKER_DEBIAN_RELEASE}/slim/rootfs.tar.xz" | tar -xJ -C "${jail_path}" --numeric-owner

    # Remove resolv.conf, systemd configures this
    rm "${jail_path}/etc/resolv.conf"

    # Bind mount this script inside the jail and run it with the '__do_not_run_this_manually__' command
    # This will install software inside the jail
    systemd-nspawn -q -D "${jail_path}" --bind-ro="${ABSOLUTE_SCRIPT_PATH}" --bind-ro=/lib/modules -E JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE}" -E JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER}" -E JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA}" ".${ABSOLUTE_SCRIPT_PATH}" __do_not_run_this_manually__

    # Start with the flags passed as argument
    additional_flags="$1"

    # Add flags for GPU access
    if [[ "${JAILMAKER_GPU_PASSTHROUGH}" -eq 1 ]]; then

    additional_flags+=" --property=DeviceAllow='char-drm rw'"

    # Detect intel GPU device and if present add bind flag
    [[ -d /dev/dri ]] && additional_flags+=" --bind=/dev/dri"

    # Detect nvidia GPU devices and if present add bind flag
    for d in /dev/nvidia*; do additional_flags+=" --bind='${d}'"; done
    fi

    if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then
    # TODO: are any of these flags required for GPU access when NOT using docker?
    additional_flags+=" --capability=all --system-call-filter='add_key keyctl bpf'"
    fi

    # If there are flags to add, put a space in front if it isn't already there
    [[ -n "${additional_flags}" && "${additional_flags}" != " "* ]] && additional_flags=" ${additional_flags}"

    # Use SYSTEMD_SECCOMP=0: https://github.com/systemd/systemd/issues/18370
    # Use SYSTEMD_NSPAWN_LOCK=0: otherwise jail won't start jail after a shutdown
    # Would give "directory tree currently busy" error
    # Fix: `rm /run/systemd/nspawn/locks/*` and remove the .lck file from JAILMAKER_JAILS_DIR
    # TODO: figure out why I have to use SYSTEMD_NSPAWN_LOCK=0

    # TODO: fix service not starting after reboot of TrueNAS. Needs to wait for zfs mount to complete?

    # Write the systemd service file
    cat <<-EOF >"${jail_path}.service"
    [Unit]
    Description=${jail_name} [created with jailmaker]
    Documentation=man:systemd-nspawn(1)
    After=network.target systemd-resolved.service
    [Service]
    Environment="SYSTEMD_SECCOMP=0"
    Environment="SYSTEMD_NSPAWN_LOCK=0"
    ExecStart=${JAILMAKER_NSPAWN_BASE_CMD}'${jail_path}'${additional_flags}
    KillMode=mixed
    Type=notify
    RestartForceExitStatus=133
    SuccessExitStatus=133
    Delegate=yes
    TasksMax=16384
    WatchdogSec=3min
    [Install]
    WantedBy=machines.target
    EOF

    # Use appropriate permission for the .service file
    # https://unix.stackexchange.com/a/433910/477308
    chmod 644 "${jail_path}.service"
    # Enable the service and start the jail
    deploy_jail "${jail_path}.service"

    cat <<-EOF
    The jail will now boot in the background...
    List all the running jails with:
    machinectl list
    When it's running you can start a shell with:
    machinectl shell ${jail_name}
    To poweroff a jail (from inside the jail):
    poweroff
    To start the jail again, run:
    systemctl start ${jail_name}
    EOF
    }

    # This function needs to be called on each boot of TrueNAS, since after an OS update
    # our systemd services are no longer registered
    sync_jails() {
    validate_input
    [[ -d "${JAILMAKER_JAILS_DIR}" ]] || fail "Nothing to sync: there are no jails in '${JAILMAKER_JAILS_DIR}'."
    # Find all the *.service files in the jails directory, and deploy them
    for d in "${JAILMAKER_JAILS_DIR}/"*.service; do deploy_jail "${d}"; done
    }

    complete_install_inside_jail() {
    # Add a safeguard so we don't accidentally install anything on the TrueNAS host
    [[ $(cli -c 'system version' 2>/dev/null) == TrueNAS* ]] && echo "This function should only be called inside a jail!"

    # Since apt is no longer executable on TrueNAS, the script won't continue,
    # even if the safeguard above stops working
    apt update

    # We need dbus for `machinectl shell`, systemd to properly boot the jail
    # and systemd-sysv to for the `poweroff` command
    apt -y --no-install-recommends install dbus systemd systemd-sysv

    if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then

    # Install docker according to official guide:
    # https://docs.docker.com/engine/install/debian/
    # Even shorter (but not recommended) would be:
    # curl -fsSL https://get.docker.com | sh

    apt -y --no-install-recommends install ca-certificates curl gnupg
    mkdir -p /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian ${JAILMAKER_DEBIAN_RELEASE} stable" | tee /etc/apt/sources.list.d/docker.list >/dev/null

    apt update
    apt -y --no-install-recommends install docker-ce docker-ce-cli containerd.io docker-compose-plugin
    fi

    if [[ "${JAILMAKER_INSTALL_NVIDIA}" -eq 1 ]]; then
    # TODO: document alternative method of installing nvidia drivers (via apt from the host)
    # with --bind-ro=/etc/apt (also cleanup apt after installing: rm -rf /var/lib/apt/lists/*)
    # Should I `hold` these packages once installed?
    # TODO: Investigate auto update command (from inside the .service file) before starting the jail,
    # to ensure jail is always started with matching nvidia driver version

    apt -y --no-install-recommends install kmod

    local nvidia_version nvidia_url

    nvidia_version=$(modinfo nvidia-current --field version)
    nvidia_url="https://us.download.nvidia.com/XFree86/Linux-x86_64/${nvidia_version}/NVIDIA-Linux-x86_64-${nvidia_version}.run"

    curl -fL --output "/tmp/nvidia_driver.run" "${nvidia_url}" || fail "Failed to download and install nvidia driver.\nDetected version of the driver on TrueNAS host: ${nvidia_version}.\nAttempted download url: ${nvidia_url}."

    sh /tmp/nvidia_driver.run --ui=none --no-questions --no-kernel-modules
    rm /tmp/nvidia_driver.run

    if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then

    # Install container-toolkit according to official guide:
    # https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

    local distribution

    distribution=$(
    . /etc/os-release
    echo "$ID$VERSION_ID"
    )

    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    apt update
    apt -y --no-install-recommends install nvidia-docker2
    fi
    fi
    }

    case "${MODE}" in

    create)
    create_jail "${*:2}"
    ;;

    sync)
    sync_jails
    ;;

    cleanup)
    echo "TODO: some kind of cleanup to disable/remove from systemctl the services of jails we have removed"
    # systemctl stop '*jailmaker*'
    # Still has to to `systemctl disable` for each individual jailmaker service (glob not supported for disable)
    ;;

    __do_not_run_this_manually__)
    complete_install_inside_jail
    ;;

    *)
    echo "TODO: print usage instructions"
    ;;
    esac
  17. Jip-Hop revised this gist Jan 3, 2023. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -66,6 +66,7 @@ if [ $install_nvidia_driver -eq 1 ]; then
    fi

    # Detect intel GPU and if present add bind flag
    # TODO: what other flags are needed to enable intel GPU? --property=DeviceAllow='char-drm rw'?
    [ -d /dev/dri ] && additional_flags="$additional_flags --bind=/dev/dri"

    dest="${dest}/${dirname}"
    @@ -186,4 +187,4 @@ fi
    # TODO: figure out why I have to use SYSTEMD_NSPAWN_LOCK=0
    # Otherwise I keep locking the jail after a shutdown and have to run:
    # rm /run/systemd/nspawn/locks/*
    # And also remove the .\#debian.lck file next to the dir containing the jail rootfs
    # And also remove the .\#debian.lck file next to the dir containing the jail rootfs
  18. Jip-Hop revised this gist Jan 3, 2023. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -65,6 +65,9 @@ if [ $install_nvidia_driver -eq 1 ]; then

    fi

    # Detect intel GPU and if present add bind flag
    [ -d /dev/dri ] && additional_flags="$additional_flags --bind=/dev/dri"

    dest="${dest}/${dirname}"

    # TODO: auto-increment the dirname with a suffix until we find a folder name which doesn't exist yet
    @@ -84,7 +87,7 @@ cleanup() {
    echo "Installation of the jail failed!"
    else
    echo "Now manually run the command below to start Debian:"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf' ${additional_flags}&"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf'${additional_flags} &"
    echo "The jail will boot in the background."
    echo ""
    echo "Now start a shell with:"
  19. Jip-Hop revised this gist Jan 3, 2023. 1 changed file with 53 additions and 28 deletions.
    81 changes: 53 additions & 28 deletions install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -8,13 +8,15 @@
    # Based on https://gist.github.com/sfan5/52aa53f5dca06ac3af30455b203d3404

    set -euox pipefail
    shopt -s nullglob

    dest=${1-""}
    install_docker=1
    install_nvidia_driver=1
    release=bullseye
    arch=amd64
    dirname=debian
    additional_flags=""

    if [ $UID -ne 0 ]; then
    echo "Run this script as root" >&2
    @@ -39,6 +41,30 @@ if [[ ${dest} != "/mnt/"* ]]; then
    exit 1
    fi

    # Check if we can find nvidia devices to bind inside the jail
    # If not, then skip installing nvidia drivers
    if [ $install_nvidia_driver -eq 1 ]; then

    nvidia_device_found=0

    for d in /dev/nvidia*; do
    echo "Found $d";
    nvidia_device_found=1
    additional_flags="$additional_flags --bind='$d'"
    done

    if [ $nvidia_device_found -eq 0 ]; then
    echo "Unable to find nvidia devices."
    echo "Has an nvidia GPU been installed?"
    echo "Skip installation of nvidia drivers in the jail."
    install_nvidia_driver=0
    else
    # Add flag for GPU access
    additional_flags="$additional_flags --property=DeviceAllow='char-drm rw'"
    fi

    fi

    dest="${dest}/${dirname}"

    # TODO: auto-increment the dirname with a suffix until we find a folder name which doesn't exist yet
    @@ -49,6 +75,7 @@ if [ -d "${dest}" ]; then
    fi

    trap cleanup EXIT

    cleanup() {
    exit_code=$?
    set +x
    @@ -57,7 +84,7 @@ cleanup() {
    echo "Installation of the jail failed!"
    else
    echo "Now manually run the command below to start Debian:"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf' &"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf' ${additional_flags}&"
    echo "The jail will boot in the background."
    echo ""
    echo "Now start a shell with:"
    @@ -84,13 +111,17 @@ mkdir -p "${dest}"

    curl -L "https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-${arch}/${release}/slim/rootfs.tar.xz" | tar -xJ -C "${dest}" --numeric-owner

    rm "${dest}/etc/resolv.conf" # systemd configures this
    # Remove resolv.conf, systemd configures this
    rm "${dest}/etc/resolv.conf"

    systemd-nspawn -q -D "${dest}" /usr/bin/apt-get update
    systemd-nspawn -q -D "${dest}" /usr/bin/apt-get -y --no-install-recommends install systemd dbus

    if [ $install_docker -eq 1 ]; then

    # Install docker according to official guide:
    # https://docs.docker.com/engine/install/debian/

    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    apt -y --no-install-recommends install ca-certificates curl gnupg
    @@ -104,54 +135,48 @@ EOF
    fi

    if [ $install_nvidia_driver -eq 1 ]; then
    # EXPERIMENTAL

    # TODO: check if nvidia devices exist, else skip driver installation
    # If they exist, also bind mount them
    # --property=DeviceAllow='char-drm rw'
    # --bind=/dev/nvidia0
    # --bind=/dev/nvidia-modeset
    # --bind=/dev/nvidia-uvm
    # --bind=/dev/nvidia-uvm-tools
    # --bind=/dev/nvidiactl

    nvidia_version=$(dpkg-query -W -f='${Version}\n' nvidia-smi | sed 's/\(.*\)-.*/\1/')
    nvidia_url="https://us.download.nvidia.com/XFree86/Linux-x86_64/${nvidia_version}/NVIDIA-Linux-x86_64-${nvidia_version}.run"

    if curl -L --output "${dest}/root/nvidia_driver.run" "${nvidia_url}"; then
    # Download the nvidia driver
    if ! curl -L --output "${dest}/root/nvidia_driver.run" "${nvidia_url}"; then

    echo "Failed to download and install nvidia driver."
    [ $install_docker -eq 1 ] && echo "Skipped installing nvidia-container-toolkit."
    echo "Detected version of the driver on TrueNAS host: ${nvidia_version}."
    echo "Attempted download url: ${nvidia_url}."
    echo "Please manually install the nvidia software inside the jail."
    exit 1

    else

    # Install the nvidia driver in the jail
    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    apt update
    apt -y --no-install-recommends install kmod
    chmod +x /root/nvidia_driver.run
    ./root/nvidia_driver.run --ui=none --no-questions --no-kernel-modules
    sh /root/nvidia_driver.run --ui=none --no-questions --no-kernel-modules
    rm /root/nvidia_driver.run
    EOF

    if [ $install_docker -eq 1 ]; then

    # Install container-toolkit according to official guide:
    # https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

    distribution=$(
    . /etc/os-release
    echo $ID$VERSION_ID
    echo "$ID$VERSION_ID"
    )

    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
    && curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    apt update
    apt -y --no-install-recommends install nvidia-docker2
    EOF
    fi
    else
    echo "Failed to download and install nvidia driver."
    [ $install_docker -eq 1 ] && echo "Skipped installing nvidia-container-toolkit."
    echo "Detected version of the driver on TrueNAS host: ${nvidia_version}."
    echo "Attempted download url: ${nvidia_url}."
    echo "Please manually install the nvidia software inside the jail."
    exit 1
    fi
    fi

  20. Jip-Hop revised this gist Jan 3, 2023. 1 changed file with 92 additions and 23 deletions.
    115 changes: 92 additions & 23 deletions install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -7,10 +7,11 @@
    # and use that as the destination argument to this script
    # Based on https://gist.github.com/sfan5/52aa53f5dca06ac3af30455b203d3404

    set -euo pipefail
    set -euox pipefail

    dest=${1-""}
    install_docker=1
    install_nvidia_driver=1
    release=bullseye
    arch=amd64
    dirname=debian
    @@ -40,11 +41,43 @@ fi

    dest="${dest}/${dirname}"

    # TODO: auto-increment the dirname with a suffix until we find a folder name which doesn't exist yet
    # TODO: override dir name using env variable
    if [ -d "${dest}" ]; then
    echo "Install directory '${dest}' already exists"
    exit 1
    fi

    trap cleanup EXIT
    cleanup() {
    exit_code=$?
    set +x

    if [[ ${exit_code} -ne 0 ]]; then
    echo "Installation of the jail failed!"
    else
    echo "Now manually run the command below to start Debian:"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf' &"
    echo "The jail will boot in the background."
    echo ""
    echo "Now start a shell with:"
    echo "machinectl shell ${dirname}"
    if [ $install_docker -eq 1 ]; then
    echo ""
    echo "You may now run docker containers, e.g. with:"
    echo "docker run -p 8080:80 nginx"
    fi
    echo ""
    echo "To poweroff run (from inside the jail):"
    echo "systemctl poweroff"
    echo ""
    echo "It's also possible to bind-mount other directories from the TrueNAS host into our Debian machine,"
    echo "so that these directories can be used inside docker containers."
    echo "Check the man pages for more info:"
    echo "https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html"
    fi
    }

    # TODO: make a parent directory, to which only root has read access
    # Otherwise users (besides root) on the host OS may be able to read/write files inside the jail rootfs
    mkdir -p "${dest}"
    @@ -59,34 +92,70 @@ systemd-nspawn -q -D "${dest}" /usr/bin/apt-get -y --no-install-recommends insta
    if [ $install_docker -eq 1 ]; then

    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    apt-get -y --no-install-recommends install ca-certificates curl gnupg
    set -euox pipefail
    apt -y --no-install-recommends install ca-certificates curl gnupg
    mkdir -p /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    echo \
    "deb [arch=${arch} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
    ${release} stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
    apt-get update
    apt-get -y --no-install-recommends install docker-ce docker-ce-cli containerd.io docker-compose-plugin
    echo "deb [arch=${arch} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian ${release} stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
    apt update
    apt -y --no-install-recommends install docker-ce docker-ce-cli containerd.io docker-compose-plugin
    EOF

    fi

    if [ $install_nvidia_driver -eq 1 ]; then
    # EXPERIMENTAL

    # TODO: check if nvidia devices exist, else skip driver installation
    # If they exist, also bind mount them
    # --property=DeviceAllow='char-drm rw'
    # --bind=/dev/nvidia0
    # --bind=/dev/nvidia-modeset
    # --bind=/dev/nvidia-uvm
    # --bind=/dev/nvidia-uvm-tools
    # --bind=/dev/nvidiactl

    nvidia_version=$(dpkg-query -W -f='${Version}\n' nvidia-smi | sed 's/\(.*\)-.*/\1/')
    nvidia_url="https://us.download.nvidia.com/XFree86/Linux-x86_64/${nvidia_version}/NVIDIA-Linux-x86_64-${nvidia_version}.run"

    if curl -L --output "${dest}/root/nvidia_driver.run" "${nvidia_url}"; then

    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    apt update
    apt -y --no-install-recommends install kmod
    chmod +x /root/nvidia_driver.run
    ./root/nvidia_driver.run --ui=none --no-questions --no-kernel-modules
    rm /root/nvidia_driver.run
    EOF

    if [ $install_docker -eq 1 ]; then

    distribution=$(
    . /etc/os-release
    echo $ID$VERSION_ID
    )
    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    set -euox pipefail
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
    && curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    apt update
    apt -y --no-install-recommends install nvidia-docker2
    EOF
    fi
    else
    echo "Failed to download and install nvidia driver."
    [ $install_docker -eq 1 ] && echo "Skipped installing nvidia-container-toolkit."
    echo "Detected version of the driver on TrueNAS host: ${nvidia_version}."
    echo "Attempted download url: ${nvidia_url}."
    echo "Please manually install the nvidia software inside the jail."
    exit 1
    fi
    fi

    # TODO: figure out why I have to use SYSTEMD_NSPAWN_LOCK=0
    # Otherwise I keep locking the jail after a shutdown and have to run:
    # rm /run/systemd/nspawn/locks/*
    # And also remove the .\#debian.lck file next to the dir containing the jail rootfs

    echo "Now manually run the command below to start Debian:"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf' &"
    echo "The jail will boot in the background."
    echo ""
    echo "Now start a shell with:"
    echo "machinectl shell ${dirname}"
    echo ""
    echo "You may now run docker containers, e.g. with:"
    echo "docker run -p 8080:80 nginx"
    echo ""
    echo "It's also possible to bind-mount other directories from the TrueNAS host into our Debian machine,"
    echo "so that these directories can be used inside docker containers."
    echo "Check the man pages for more info:"
    echo "https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html"
    # And also remove the .\#debian.lck file next to the dir containing the jail rootfs
  21. Jip-Hop revised this gist Jan 2, 2023. 1 changed file with 42 additions and 27 deletions.
    69 changes: 42 additions & 27 deletions install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -10,65 +10,80 @@
    set -euo pipefail

    dest=${1-""}
    install_docker=1
    release=bullseye
    arch=amd64
    dirname=debian

    if [ $UID -ne 0 ]; then
    echo "Run this script as root" >&2
    exit 1
    fi

    if [ -z "$dest" ]; then
    if [ -z "${dest}" ]; then
    echo "Usage: $0 <destination>" >&2
    exit 0
    fi

    dest=$(realpath "$dest")
    dest=$(realpath "${dest}")

    if [ ! -d "$dest" ]
    then
    echo "Destination directory '$dest' DOES NOT exist"
    if [ ! -d "${dest}" ]; then
    echo "Destination directory '$dest' DOES NOT exist"
    exit 1
    fi

    if [[ $dest != "/mnt/"* ]]; then
    if [[ ${dest} != "/mnt/"* ]]; then
    echo "The destination path must begin with '/mnt/'"
    echo "Provided destination was '$dest"
    echo "Provided destination was '${dest}"
    exit 1
    fi

    dest="$dest/debian"
    dest="${dest}/${dirname}"

    if [ -d "$dest" ]
    then
    echo "Install directory '$dest' already exists"
    if [ -d "${dest}" ]; then
    echo "Install directory '${dest}' already exists"
    exit 1
    fi

    # TODO: make a parent directory, to which only root has read access
    # Otherwise users (besides root) on the host OS may be able to read/write files inside the jail rootfs
    mkdir -p "$dest"
    mkdir -p "${dest}"

    curl -L https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-amd64/bullseye/slim/rootfs.tar.xz | tar -xJ -C "$dest" --numeric-owner
    curl -L "https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-${arch}/${release}/slim/rootfs.tar.xz" | tar -xJ -C "${dest}" --numeric-owner

    sed '/^root:/ s|\*||' -i "$dest/etc/shadow" # passwordless login
    rm "$dest/etc/resolv.conf" # systemd configures this
    # https://github.com/systemd/systemd/issues/852
    [ -f "$dest/etc/securetty" ] && printf 'pts/%d\n' $(seq 0 10) >>"$dest/etc/securetty"
    echo "" > "$dest/etc/fstab" # clear fstab file
    rm "${dest}/etc/resolv.conf" # systemd configures this

    systemd-nspawn -q -D "$dest" /usr/bin/apt-get update
    systemd-nspawn -q -D "$dest" /usr/bin/apt-get -y --no-install-recommends install systemd
    systemd-nspawn -q -D "${dest}" /usr/bin/apt-get update
    systemd-nspawn -q -D "${dest}" /usr/bin/apt-get -y --no-install-recommends install systemd dbus

    curl -fsSL https://get.docker.com -o "$dest/root/get-docker.sh"
    systemd-nspawn -q -D "$dest" /bin/bash /root/get-docker.sh
    rm "$dest/root/get-docker.sh"
    if [ $install_docker -eq 1 ]; then

    systemd-nspawn -q -D "$dest" systemctl enable docker
    systemd-nspawn -q -D "${dest}" --pipe /bin/bash <<EOF
    apt-get -y --no-install-recommends install ca-certificates curl gnupg
    mkdir -p /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    echo \
    "deb [arch=${arch} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
    ${release} stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
    apt-get update
    apt-get -y --no-install-recommends install docker-ce docker-ce-cli containerd.io docker-compose-plugin
    EOF

    fi

    # TODO: figure out why I have to use SYSTEMD_NSPAWN_LOCK=0
    # Otherwise I keep locking the jail after a shutdown and have to run:
    # rm /run/systemd/nspawn/locks/*
    # And also remove the .\#debian.lck file next to the dir containing the jail rootfs

    echo "Now manually run the command below to start Debian:"
    echo "systemd-nspawn --capability=all -b -D "$dest" --system-call-filter='add_key keyctl bpf'"
    echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 systemd-nspawn --capability=all -b -D '${dest}' --system-call-filter='add_key keyctl bpf' &"
    echo "The jail will boot in the background."
    echo ""
    echo "Now start a shell with:"
    echo "machinectl shell ${dirname}"
    echo ""
    echo "Login as user root (no password)"
    echo "You may now run docker containers, e.g. with:"
    echo "You may now run docker containers, e.g. with:"
    echo "docker run -p 8080:80 nginx"
    echo ""
    echo "It's also possible to bind-mount other directories from the TrueNAS host into our Debian machine,"
  22. Jip-Hop revised this gist Jan 2, 2023. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -43,6 +43,8 @@ then
    exit 1
    fi

    # TODO: make a parent directory, to which only root has read access
    # Otherwise users (besides root) on the host OS may be able to read/write files inside the jail rootfs
    mkdir -p "$dest"

    curl -L https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-amd64/bullseye/slim/rootfs.tar.xz | tar -xJ -C "$dest" --numeric-owner
  23. Jip-Hop renamed this gist Jan 2, 2023. 1 changed file with 15 additions and 21 deletions.
    36 changes: 15 additions & 21 deletions install-ubuntu-jail.sh → install-debian-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -2,15 +2,13 @@

    # WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!

    # Creates a systemd-nspawn container with Ubuntu
    # Creates a systemd-nspawn container with Debian
    # I suggest to create a new, dedicated, dataset using the TrueNAS SCALE web ui first
    # and use that as the destination argument to this script
    # Based on https://gist.github.com/sfan5/52aa53f5dca06ac3af30455b203d3404

    set -euo pipefail

    CODENAME=${CODENAME:-jammy}

    dest=${1-""}

    if [ $UID -ne 0 ]; then
    @@ -31,13 +29,13 @@ then
    exit 1
    fi

    if [[ $dest != "/mnt/data/"* ]]; then
    echo "The destination path must begin with '/mnt/data/'"
    if [[ $dest != "/mnt/"* ]]; then
    echo "The destination path must begin with '/mnt/'"
    echo "Provided destination was '$dest"
    exit 1
    fi

    dest="$dest/ubuntu"
    dest="$dest/debian"

    if [ -d "$dest" ]
    then
    @@ -47,35 +45,31 @@ fi

    mkdir -p "$dest"

    # # TODO:
    # # Create a symlink so machinectl will be able to find the ubuntu install
    # # Then we can use 'machinectl enable' to run the machine automatically on startup
    # # This symlink won't survive upgrades of TrueNAS, so need to be recreated
    # ln -s "$UBUNTU_PATH" /var/lib/machines

    wget -c "http://cloud-images.ubuntu.com/${CODENAME}/current/${CODENAME}-server-cloudimg-amd64-root.tar.xz" -O - | tar -xJ -C "$dest" --numeric-owner
    curl -L https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-amd64/bullseye/slim/rootfs.tar.xz | tar -xJ -C "$dest" --numeric-owner

    sed '/^root:/ s|\*||' -i "$dest/etc/shadow" # passwordless login
    rm "$dest/etc/resolv.conf" # systemd configures this
    # https://github.com/systemd/systemd/issues/852
    [ -f "$dest/etc/securetty" ] && printf 'pts/%d\n' $(seq 0 10) >>"$dest/etc/securetty"
    echo "" > "$dest/etc/fstab" # clear fstab file
    systemd-nspawn -q -D "$dest" /bin/systemctl disable ssh systemd-{timesyncd,networkd-wait-online,resolved}
    # uninstall some packages
    systemd-nspawn -q -D "$dest" /usr/bin/apt-get -qq satisfy -y --purge 'Conflicts: lxcfs, lxd, snapd, cloud-init' || \
    systemd-nspawn -q -D "$dest" /usr/bin/apt-get -qq purge --autoremove snapd lxcfs lxd cloud-init

    systemd-nspawn -q -D "$dest" /usr/bin/curl -fsSL https://get.docker.com -o get-docker.sh
    systemd-nspawn -q -D "$dest" /usr/bin/sh get-docker.sh
    systemd-nspawn -q -D "$dest" /usr/bin/apt-get update
    systemd-nspawn -q -D "$dest" /usr/bin/apt-get -y --no-install-recommends install systemd

    curl -fsSL https://get.docker.com -o "$dest/root/get-docker.sh"
    systemd-nspawn -q -D "$dest" /bin/bash /root/get-docker.sh
    rm "$dest/root/get-docker.sh"

    systemd-nspawn -q -D "$dest" systemctl enable docker

    echo "Now manually run the command below to start ubuntu:"
    echo "Now manually run the command below to start Debian:"
    echo "systemd-nspawn --capability=all -b -D "$dest" --system-call-filter='add_key keyctl bpf'"
    echo ""
    echo "Login as user root (no password)"
    echo "You may now run docker containers, e.g. with:"
    echo "docker run -p 8080:80 nginx"
    echo ""
    echo "It's also possible to bind-mount other directories from the TrueNAS host into our Ubuntu machine,"
    echo "It's also possible to bind-mount other directories from the TrueNAS host into our Debian machine,"
    echo "so that these directories can be used inside docker containers."
    echo "Check the man pages for more info:"
    echo "https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html"
  24. Jip-Hop created this gist Dec 29, 2022.
    81 changes: 81 additions & 0 deletions install-ubuntu-jail.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,81 @@
    #!/bin/bash

    # WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!

    # Creates a systemd-nspawn container with Ubuntu
    # I suggest to create a new, dedicated, dataset using the TrueNAS SCALE web ui first
    # and use that as the destination argument to this script
    # Based on https://gist.github.com/sfan5/52aa53f5dca06ac3af30455b203d3404

    set -euo pipefail

    CODENAME=${CODENAME:-jammy}

    dest=${1-""}

    if [ $UID -ne 0 ]; then
    echo "Run this script as root" >&2
    exit 1
    fi

    if [ -z "$dest" ]; then
    echo "Usage: $0 <destination>" >&2
    exit 0
    fi

    dest=$(realpath "$dest")

    if [ ! -d "$dest" ]
    then
    echo "Destination directory '$dest' DOES NOT exist"
    exit 1
    fi

    if [[ $dest != "/mnt/data/"* ]]; then
    echo "The destination path must begin with '/mnt/data/'"
    echo "Provided destination was '$dest"
    exit 1
    fi

    dest="$dest/ubuntu"

    if [ -d "$dest" ]
    then
    echo "Install directory '$dest' already exists"
    exit 1
    fi

    mkdir -p "$dest"

    # # TODO:
    # # Create a symlink so machinectl will be able to find the ubuntu install
    # # Then we can use 'machinectl enable' to run the machine automatically on startup
    # # This symlink won't survive upgrades of TrueNAS, so need to be recreated
    # ln -s "$UBUNTU_PATH" /var/lib/machines

    wget -c "http://cloud-images.ubuntu.com/${CODENAME}/current/${CODENAME}-server-cloudimg-amd64-root.tar.xz" -O - | tar -xJ -C "$dest" --numeric-owner

    sed '/^root:/ s|\*||' -i "$dest/etc/shadow" # passwordless login
    rm "$dest/etc/resolv.conf" # systemd configures this
    # https://github.com/systemd/systemd/issues/852
    [ -f "$dest/etc/securetty" ] && printf 'pts/%d\n' $(seq 0 10) >>"$dest/etc/securetty"
    echo "" > "$dest/etc/fstab" # clear fstab file
    systemd-nspawn -q -D "$dest" /bin/systemctl disable ssh systemd-{timesyncd,networkd-wait-online,resolved}
    # uninstall some packages
    systemd-nspawn -q -D "$dest" /usr/bin/apt-get -qq satisfy -y --purge 'Conflicts: lxcfs, lxd, snapd, cloud-init' || \
    systemd-nspawn -q -D "$dest" /usr/bin/apt-get -qq purge --autoremove snapd lxcfs lxd cloud-init

    systemd-nspawn -q -D "$dest" /usr/bin/curl -fsSL https://get.docker.com -o get-docker.sh
    systemd-nspawn -q -D "$dest" /usr/bin/sh get-docker.sh

    echo "Now manually run the command below to start ubuntu:"
    echo "systemd-nspawn --capability=all -b -D "$dest" --system-call-filter='add_key keyctl bpf'"
    echo ""
    echo "Login as user root (no password)"
    echo "You may now run docker containers, e.g. with:"
    echo "docker run -p 8080:80 nginx"
    echo ""
    echo "It's also possible to bind-mount other directories from the TrueNAS host into our Ubuntu machine,"
    echo "so that these directories can be used inside docker containers."
    echo "Check the man pages for more info:"
    echo "https://manpages.debian.org/bullseye/systemd-container/systemd-nspawn.1.en.html"