Persistent Debian 'jail' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts. Without modifying the host OS at all thanks to systemd-nspawn!

jailmaker.sh

#!/bin/bash

set -euo pipefail
shopt -s nullglob

ABSOLUTE_SCRIPT_PATH="$(realpath "${BASH_SOURCE[0]}")"
SCRIPT_NAME=$(basename "${ABSOLUTE_SCRIPT_PATH}")
SCRIPT_PARENT_DIR="$(dirname "$ABSOLUTE_SCRIPT_PATH")"
DEBIAN_RELEASE="$(. /etc/os-release && echo "${VERSION_CODENAME}")"

USAGE="WARNING: EXPERIMENTAL AND WORK IN PROGRESS, USE ONLY FOR TESTING!

Create persistent Debian 'jails' with systemd-nspawn on TrueNAS SCALE
Allows installing software, such as docker, without modifying the host OS
With full access to all files on the host via bind mounts

Put this script on one of your ZFS datasets (under the /mnt/ directory)
It will create a 'jails' dir next to it, where it stores all the jails

Run this script as root user

Usage: [ENV_VAR=value] ./${SCRIPT_NAME} COMMAND [ARG...]

Commands:
  new		Create a new jail in the jails dir and start it
  up		Bring up all jails by running all *.sh files in the jails dir
  help		Show this help message

All arguments passed to the new command will be added to the systemd-nspawn command that starts the jail
For available arguments see: https://manpages.debian.org/${DEBIAN_RELEASE}/systemd-container/systemd-nspawn.1.en.html

Environment variables:
  JAILMAKER_DEBUG=0
  JAILMAKER_JAIL_NAME=${DEBIAN_RELEASE}
  JAILMAKER_INSTALL_DOCKER=0
  JAILMAKER_INSTALL_NVIDIA=0
  JAILMAKER_GPU_PASSTHROUGH=0

Default settings can be overridden via environment variables, documentation about advanced settings is inside ${SCRIPT_NAME}

Examples:

# Start a new jail with read/write access to /mnt/ssd/appdata and readonly access to /mnt/Tank/Media
./${SCRIPT_NAME} new --bind=/mnt/ssd/appdata --bind-ro=/mnt/Tank/Media

# Start a new jail named: my-jail-name
JAILMAKER_JAIL_NAME=my-jail-name ./${SCRIPT_NAME} new

# Start a new jail with GPU passthrough set-up (if a GPU device can be found during creation)
JAILMAKER_GPU_PASSTHROUGH=1 ./${SCRIPT_NAME} new

# Start a new jail, install docker and enable debug mode to output all the steps this script takes 
JAILMAKER_INSTALL_DOCKER=1 JAILMAKER_DEBUG=1 ./${SCRIPT_NAME} new

# Bring up all jails by running all *.sh files in the jails dir
./${SCRIPT_NAME} up
"

fail() {
	echo -e "$1" >&2 && exit 1
}

[[ $UID -ne 0 ]] && echo "${USAGE}" && fail "Run this script as root..."

####################################################################
# Read from user-defined environment variables or use default values
####################################################################

# Basic settings, should be safe to override

# Print each line this script executes in debug mode - valid values: 0 (default), 1
JAILMAKER_DEBUG="${JAILMAKER_DEBUG:-0}" && [[ "${JAILMAKER_DEBUG}" -eq 1 ]] && set -x
# Default name for the jail is the debian release name, e.g. bullseye - valid values: a string without the '/' character in it
JAILMAKER_JAIL_NAME="${JAILMAKER_JAIL_NAME:-${DEBIAN_RELEASE}}"
# By default we will not install docker inside the jail - valid values: 0 (default), 1
JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER:-0}"
# By default we will not install nvidia drivers - valid values: 0 (default), 1
JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA:-0}"
# By default we will not setup GPU passthrough (unless we're installing nvidia drivers) - valid values: 0, 1
JAILMAKER_GPU_PASSTHROUGH="${JAILMAKER_GPU_PASSTHROUGH:-${JAILMAKER_INSTALL_NVIDIA}}"

# Advanced settings, be careful if you change these!

# By default make a jail which matches the debian version of TrueNAS - valid values: a debian release name (only bullseye has been tested)
JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE:-${DEBIAN_RELEASE}}"
# By default we will use (or create) the 'jails' dir inside this script's parent directory - valid values: a path to a directory (parent directory should exist)
JAILMAKER_JAILS_DIR="${JAILMAKER_JAILS_DIR:-${SCRIPT_PARENT_DIR}/jails}"
# Only allow creating jails inside a /mnt sub-folder, unless safety check if off - valid values: 0, 1 (default)
JAILMAKER_JAILS_DIR_SAFETY_CHECK="${JAILMAKER_JAILS_DIR_SAFETY_CHECK:-1}"
# The default command which will be used to start the jail from the .sh file - you should probably not be modifying this and instead pass additional flags to the new command
JAILMAKER_NSPAWN_BASE_CMD="${JAILMAKER_NSPAWN_BASE_CMD:-systemd-nspawn --quiet --boot --directory=}"

# Since this file will be executed as root, we need to set appropriate permissions (if not already set)
[[ "$(stat -c%a "${ABSOLUTE_SCRIPT_PATH}")" -ne 700 ]] && chmod 700 "${ABSOLUTE_SCRIPT_PATH}"

validate_input() {
	local jails_dir_parent_dir
	jails_dir_parent_dir="$(dirname "${JAILMAKER_JAILS_DIR}")"

	# Check if working directory exists
	if [[ ! -d ${jails_dir_parent_dir} ]]; then
		fail "Destination directory '${jails_dir_parent_dir}' DOES NOT exist."
	# Check if working directory is inside the /mnt directory (so it won't be lost on TrueNAS OS updates)
	elif [[ "$JAILMAKER_JAILS_DIR_SAFETY_CHECK" -ne 0 && "${JAILMAKER_JAILS_DIR}" != /mnt/* ]]; then
		fail "The destination path must begin with '/mnt/'\nProvided destination was '${JAILMAKER_JAILS_DIR}.'"
	# Check for illegal characters in the directory name
	elif [[ "${JAILMAKER_JAIL_NAME}" == */* ]]; then
		fail "Name may not contain the '/' character.\nJAILMAKER_JAIL_NAME: ${JAILMAKER_JAIL_NAME}."
	fi
}

new() {
	validate_input

	local arch original_jail_name x jail_path relative_jail_path additional_flags
	arch=$(dpkg --print-architecture)

	# Create the jails dir if it does not exist
	[[ -d "${JAILMAKER_JAILS_DIR}" ]] || mkdir "${JAILMAKER_JAILS_DIR}"

	# Only root should be allowed access to this directory
	chmod 700 "${JAILMAKER_JAILS_DIR}"

	original_jail_name="${JAILMAKER_JAIL_NAME}"
	jail_name=${original_jail_name}
	x=1

	# Append counter to jail path if there already is a jail with this name
	while [[ -d "${JAILMAKER_JAILS_DIR}/${jail_name}" ]]; do jail_name="${original_jail_name}-$((x++))"; done

	jail_path="${JAILMAKER_JAILS_DIR}/${jail_name}"

	# Make the new jail directory
	mkdir "${jail_path}"

	# Download and extract the root filesystem for the jail
	curl -L "https://github.com/debuerreotype/docker-debian-artifacts/raw/dist-${arch}/${JAILMAKER_DEBIAN_RELEASE}/slim/rootfs.tar.xz" | tar -xJ -C "${jail_path}" --numeric-owner

	# Remove resolv.conf, systemd configures this
	rm "${jail_path}/etc/resolv.conf"

	# Bind mount this script inside the jail and run it with the '__do_not_run_this_manually__' command
	# This will install software inside the jail
	systemd-nspawn -q -D "${jail_path}" --bind-ro="${ABSOLUTE_SCRIPT_PATH}" --bind-ro=/lib/modules -E JAILMAKER_DEBUG="${JAILMAKER_DEBUG}" -E JAILMAKER_DEBIAN_RELEASE="${JAILMAKER_DEBIAN_RELEASE}" -E JAILMAKER_INSTALL_DOCKER="${JAILMAKER_INSTALL_DOCKER}" -E JAILMAKER_INSTALL_NVIDIA="${JAILMAKER_INSTALL_NVIDIA}" "${ABSOLUTE_SCRIPT_PATH}" __do_not_run_this_manually__

	# Start with the flags passed as argument
	additional_flags="$1"

	# Add flags for GPU access
	if [[ "${JAILMAKER_GPU_PASSTHROUGH}" -eq 1 ]]; then

		additional_flags+=" --property=DeviceAllow='char-drm rw'"

		# Detect intel GPU device and if present add bind flag
		[[ -d /dev/dri ]] && additional_flags+=" --bind=/dev/dri"

		# Detect nvidia GPU devices and if present add bind flag
		for d in /dev/nvidia*; do additional_flags+=" --bind='${d}'"; done
	fi

	if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then
		# TODO: are any of these flags required for GPU access when NOT using docker?
		additional_flags+=" --capability=all --system-call-filter='add_key keyctl bpf'"
	fi

	# If there are flags to add, put a space in front if it isn't already there
	[[ -n "${additional_flags}" && "${additional_flags}" != " "* ]] && additional_flags=" ${additional_flags}"

	# Use SYSTEMD_SECCOMP=0: https://github.com/systemd/systemd/issues/18370
	# Use SYSTEMD_NSPAWN_LOCK=0: otherwise jail won't start jail after a shutdown (but why?)
	# Would give "directory tree currently busy" error and I'd have to run
	# `rm /run/systemd/nspawn/locks/*` and remove the .lck file from JAILMAKER_JAILS_DIR
	# Disabling locking isn't a big deal as systemd-nspawn will prevent starting a container
	# with the same name anyway: as long as we're starting jails using the accompanying .sh script,
	# it won't be possible to start the same jail twice

	relative_jail_path=$(realpath --relative-to="${JAILMAKER_JAILS_DIR}" "${jail_path}")

	# Write the static part of the startup shell script (no variable substitution)
	cat <<-'EOF' >"${jail_path}.sh"
		#!/bin/bash
		umask 077
		cd "$(dirname "${BASH_SOURCE[0]}")"
	EOF

	# Append the dynamic part of the startup shell script (with variable substitution)
	echo "SYSTEMD_SECCOMP=0 SYSTEMD_NSPAWN_LOCK=0 ${JAILMAKER_NSPAWN_BASE_CMD}'${relative_jail_path}'${additional_flags} &> '${relative_jail_path}.log' &" >>"${jail_path}.sh"

	# Make executable and allow access only for root user
	chmod 700 "${jail_path}.sh"

	# Start the jail
	"${jail_path}.sh"

	cat <<-EOF
		The jail will now boot in the background...

		List all the running jails with:
		machinectl list

		When it's running you can start a shell with:
		machinectl shell ${jail_name}

		To poweroff a jail (from inside the jail):
		poweroff

		In order to start the jail automatically after TrueNAS boot, configure '${jail_path}.sh' as Post Init Script from the TrueNAS web interface.

		In order to start all jails automatically after TrueNAS boot, configure '${ABSOLUTE_SCRIPT_PATH} up' as Post Init Script from the TrueNAS web interface.
	EOF
}

# This function may be called on each boot of TrueNAS to start all jails
up() {
	validate_input
	[[ -d "${JAILMAKER_JAILS_DIR}" ]] || fail "Nothing to start: the '${JAILMAKER_JAILS_DIR}' directory does not exist."
	# Find all the *.sh files in the jails directory, and start them
	for d in "${JAILMAKER_JAILS_DIR}/"*.sh; do "${d}" || true; done
}

complete_install_inside_jail() {
	# Add a safeguard so we don't accidentally install anything on the TrueNAS host
	[[ $(cli -c 'system version' 2>/dev/null) == TrueNAS* ]] && echo "This function should only be called inside a jail!"

	# Since apt is no longer executable on TrueNAS, the script won't continue,
	# even if the safeguard above stops working
	apt update

	# We need dbus for `machinectl shell`, systemd to properly boot the jail
	# and systemd-sysv to for the `poweroff` command
	apt -y --no-install-recommends install dbus systemd systemd-sysv

	if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then

		# Install docker according to official guide:
		# https://docs.docker.com/engine/install/debian/
		# Even shorter (but not recommended) would be:
		# curl -fsSL https://get.docker.com | sh

		apt -y --no-install-recommends install ca-certificates curl gnupg
		mkdir -p /etc/apt/keyrings
		curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
		echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian ${JAILMAKER_DEBIAN_RELEASE} stable" | tee /etc/apt/sources.list.d/docker.list >/dev/null

		apt update
		apt -y --no-install-recommends install docker-ce docker-ce-cli containerd.io docker-compose-plugin
	fi

	if [[ "${JAILMAKER_INSTALL_NVIDIA}" -eq 1 ]]; then
		# TODO: document alternative method of installing nvidia drivers (via apt from the host)
		# with --bind-ro=/etc/apt (also cleanup apt after installing: rm -rf /var/lib/apt/lists/*)
		# Should I `hold` these packages once installed?
		# TODO: Investigate auto update command before starting the jail,
		# to ensure jail is always started with matching nvidia driver version

		apt -y --no-install-recommends install kmod

		local nvidia_version nvidia_url

		nvidia_version=$(modinfo nvidia-current --field version)
		nvidia_url="https://us.download.nvidia.com/XFree86/Linux-x86_64/${nvidia_version}/NVIDIA-Linux-x86_64-${nvidia_version}.run"

		curl -fL --output "/tmp/nvidia_driver.run" "${nvidia_url}" || fail "Failed to download and install nvidia driver.\nDetected version of the driver on TrueNAS host: ${nvidia_version}.\nAttempted download url: ${nvidia_url}."

		sh /tmp/nvidia_driver.run --ui=none --no-questions --no-kernel-modules
		rm /tmp/nvidia_driver.run

		if [[ "${JAILMAKER_INSTALL_DOCKER}" -eq 1 ]]; then

			# Install container-toolkit according to official guide:
			# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

			local distribution
			distribution="$(. /etc/os-release && echo "${ID}${VERSION_ID}")"

			curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
			curl -s -L https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
			apt update
			apt -y --no-install-recommends install nvidia-docker2
		fi
	fi
}

case "${1-""}" in

new)
	new "${*:2}"
	;;

up)
	up
	;;

__do_not_run_this_manually__)
	complete_install_inside_jail
	;;

*)
	echo "${USAGE}"
	;;
esac

wizard.sh

#!/bin/bash

set -euo pipefail

JAILMAKER_INSTALL_DOCKER=0
JAILMAKER_GPU_PASSTHROUGH=0
JAILMAKER_INSTALL_NVIDIA=0
JAILMAKER_DEBUG=0
ADDITIONAL_FLAGS=

echo "This wizard will walk you through the steps of creating a new jail."
echo ""
read -r -p "Enter a name for the jail: " JAILMAKER_JAIL_NAME
echo ""
echo "You may pass additional arguments to systemd-nspawn, e.g. to mount directories."
echo "For example: --bind=/mnt/dir1 --bind-ro=/mnt/dir"
echo "This would mount /mnt/dir with read/write access and /mnt/dir with readonly access:"
echo ""
read -r -p "Enter additional arguments for systemd-nspawn: " ADDITIONAL_FLAGS
echo ""
read -r -p "Do you want to install docker inside the jail? [y/N]: " CHOICE
echo ""
case ${CHOICE} in
[yY]*) JAILMAKER_INSTALL_DOCKER=1 ;;
esac
read -r -p "Do you want to access the GPU inside the jail? [y/N]: " CHOICE
echo ""
if [[ "${CHOICE}" == 'y' || "${CHOICE}" == 'Y' ]]; then
    JAILMAKER_GPU_PASSTHROUGH=1
    echo "It's possible to use an nvidia GPU if we install the correct driver version."
    echo "When a TrueNAS SCALE update changes the nvidia driver version, a driver mismatch will occur."
    echo "You'll need to MANUALLY update the nvidia driver inside the jail..."
    echo ""
    read -r -p "Do you want to install nvidia drivers inside the jail? [y/N]: " CHOICE
    echo ""
    case ${CHOICE} in
    [yY]*) JAILMAKER_INSTALL_NVIDIA=1 ;;
    esac
fi
read -r -p "Do you want to see all the steps the jailmaker.sh script takes (debug mode)? [y/N]: " CHOICE
echo ""
case ${CHOICE} in
[yY]*) JAILMAKER_DEBUG=1 ;;
esac
echo "We will now pass your input to the jailmaker.sh script."
export JAILMAKER_DEBUG JAILMAKER_JAIL_NAME JAILMAKER_INSTALL_DOCKER JAILMAKER_INSTALL_NVIDIA JAILMAKER_GPU_PASSTHROUGH
./jailmaker.sh new "${ADDITIONAL_FLAGS}"

Thank you Sir and thank you for the wonderful script. I finally got everything talking to one another (Internet, apps). I am obviously doing something wrong with portainer. I never had an issue using the docker-compose truechart and pointing it at a docker-compose.yml file. I can't see how I would use that approach here. I am at a complete loss, which saddens me as I am sure this system seems to work swimmingly for everyone else. And sadly how to install and access portainer is not within the scope of this gist.

Should there be an address in this? (Output from ./jlmkr.py list)
MACHINE CLASS SERVICE OS VERSION ADDRESSES
docker-compose container systemd-nspawn debian 11 -

@worksmarter share your portainer compose file.

There is where it appears to be:
Run this with docker compose up -d and it comes right up as everyone said it would. I feel so stupid but it had to be the ports.

`version: '3'

services:
portainer:
image: portainer/portainer-ce:latest
container_name: portainer
restart: unless-stopped
security_opt:
- no-new-privileges:true
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./portainer-data:/data
ports:
- 9000:9000`

Wish I had come to that conclusion before posting. AND hasltleing around with bridging the networks, what a hassle that was. Thanks for your patience everyone. Great script JipHop!

I just heard read the news 😞

Update: 15-6-2023. Looks like the systemd-container package has been removed since 22.12.3...

I haven't used your jailmaker script, but this still sucks.

I just heard read the news 😞

Update: 15-6-2023. Looks like the systemd-container package has been removed since 22.12.3...

I haven't used your jailmaker script, but this still sucks.

wow... Good thing I am busy converting back to Proxmox for vm/container needs and will be running TrueNAS as a VM in that. For the little bit of VM's i was doing it just became too much hassle fighting Truenas imho. Now it is being relegated to ONLY a file server.

Now that would be annoying as jailmakr works perfectly, doesn't have the crappy IX routing that IX has inflicted on K3S and is simpler.

I might not uprgade for a while

I mean, by now we should all figure that Ix Systems is dead-set on SCALE being used their way or the highway, and non-commercial users are simply a pool of free beta testers for them, so if you hate how they handle apps - and it is dumb, no two ways about it - then best case use it as a simple, dumb fileserver, or leave it. Time to move on from the AIO idea.

You want to know what really doesn’t make sense about this, they are using K3S as the platform to scale your app server requirements . I don’t see many enterprises using that for their app servers. Also note that it is called TrueNAS and not TrueApp, they are shoe horning application enterprise server requirements into an enterprise file server.

It only doesn't make sense because we're missing context. Likely it is as simple as: Ix Systems, being a for-profit business in a tough market (there's no money in storage hardware sales except at scale, so you have to differentiate with software these days), is trying to better define the space they are in so they can expand, and based on (presumably) their own internal market research concluded this is the way to go. Will it work or not? Does it make sense outside of Ix Systems? Who knows? It doesn't matter.

Folks like us are their field beta testers, more or less. Heck they aren't even subtle about that, their Enterprise marketing highlights how many Truenas (and SCALE) users there are, etc. We get a decent fileserver and whatever else they want to throw on top of it but let's not be under any illusion that they really care how any of us want to use the system. If we aren't paying for it then we are the product simple as that.

I'd like to give a more positive notion into this discussion. Yes, iX, at the moment, is presumably trying to contain their user base inside their newly build ecosystem, which happen to be a catalogue of k8s apps. Nobody says that this is not gonna change in the future. It is just what they decided to do and it's their product, so that's the way it is. BUT, they have a over a decade long track record of changing their product according to what the community wants and needs. See i. e. the change from FreeBSD warden jails to iocage. Every time a "breaking" change was introduced, it lead to some kind of "backlash" in the Forums, later also on reddit. But over time everybody agreed that it was for the better. And they still have a huge list of stuff users want and I'm convinced they are looking into it. Though, at the end of the day we have the great pleasure of using their product for free and they still need to pay staff. So I'm pledging for this: we vote the according JIRA-Tickets with the features we want to see and we further voice our constructive criticism here, in the forums, on Discord and on (reddit maybe not anymore).

For me, honestly it just became way too much effort to keep on "Fixing" and finding work-arounds on their updates. I want to use Docker, I want to use VM's, I want to use ZFS and I want to use shares.

TrueNAS can no longer fulfill these needs for me without constant fighting, so I gave up. I have reinstalled Proxmox on my box, installed Docker Natively, and setup my ZFS storage. I just need to add in Shares via SMB and NFS with config files. Then a cronjob to do snapshots and replication. Is it a pain that it is not gui, sure, but end of the day is that it works and I have no worries that a single point update will kill everything.

I am done with TrueNAS... If I want it, I will go to TrueNAS Core because Scale is a wast of time for Home User.

FWIW, there is another option if you're just looking for a clean Docker/Compose workspace on Scale and have a decent grasp on things. Basically I use Scale's "Launch Docker Image" to create a pod running Docker (technically a Docker-in-Docker image ala https://hub.docker.com/_/docker or similar -- I'm running one based on DIND some nVidia additions).

At that point there is a "safe" and isolated install of Docker running which can be used -- as Docker on the Scale host could be removed at any point. Using that Docker pod I just start up a container that gives me access and all the tools I would want on a "Docker host" (sshd, compose, code-server, cli tools, etc) which I then use to get access to my "Docker host".

Anyway, that's been working well since the first release of Scale with roughly 30 containers running. "Launch Docker Image" allows the base Docker install to have host networking and even nVidia passthrough (in addition to using it with official IX apps), so it's pretty flexible overall. Obviously in the App UI in Scale I just see my "Docker" app which is running the show. ;)

Not exactly the same as just an isolated host workspace, but quite flexible in the end. Just tossing it out there for those who might find it as an interesting option.

@Codelica I understand that docker is going away soon - so your solution (if I am correct) will no longer work

@aardvarkl that would be the Docker installation that is on the TrueNAS Scale host itself, which has always been in jeopardy. My solution was to run my own Docker install separately as a pod on Scale and use that instance for services. (It's similar to how TrueCharts does their docker-compose chart) So removal of Docker on the Scale host won't affect it. The only real danger would be if IX decides that people can't launch any of their own custom pods and must choose from a catalog. Could happen, but seems doubtful to me.

@Codelica would you try something for me. In one of your containers that has traceroute, just run a traceroute to a device on your LAN that isn't your router / gateway and post the result please?

@aardvarkl sure, although that will depend what networking driver the container uses (host, bridge, macvlan,etc).

But like this is in a host mode container pinging a LAN host:

and this is in a bridge mode container pinging a LAN host:

And .1 isn't your internet router? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: James ***@***.***> Sent: Monday, June 26, 2023 3:50:18 PM To: Codelica ***@***.***> Cc: Mention ***@***.***>; Manual ***@***.***> Subject: Re: Jip-Hop/README.md @Codelica commented on this gist.

________________________________ @aardvarkl<https://github.com/aardvarkl> sure, although that will depend what networking driver the container uses (host, bridge, macvlan,etc). But like this is in a host mode container pinging a LAN host: [SeaShell Docker 2023-06-26 08-49-30]<https://user-images.githubusercontent.com/386101/248842204-b3500427-481f-44dc-b48c-ec462f374de7.png> and this is in a bridge mode container pinging a LAN host: [email-tr]<https://user-images.githubusercontent.com/386101/248842012-4674d433-233a-4a78-8b6d-c25503af2af3.png> — Reply to this email directly, view it on GitHub<https://gist.github.com/Jip-Hop/4704ba4aa87c99f342b2846ed7885a5d#gistcomment-4610820> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJQGKPU4L3LYZHSTEAH54CTXNGOSXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTEMBQGQ3DMOJYU52HE2LHM5SXFJTDOJSWC5DF>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

@aardvarkl it's not. I use a class B at home (10.10.x.x). 10.10.100.1 is a desktop in my bedroom. My router is 10.10.0.1.

The reason I ask is a few months ago I tested exactly that scenario - due to a wierd issue I was experiencing.
I found that all traffic leaving the container was going to the default gateway and from there being diverted back to the LAN.

I was running Policy Based Routing on my router which was diverting the traffic out of a VPN even when it was destined to the LAN which was initially puzzling, and then annoying even if easy enough to get around.

At the moment I cannot find a suitable container that has traceroute in (the one I used to use was Heimdall - but that no longer has traceroute inbuilt)

@aardvarkl that does sound odd. in the end you do have some control over the network side of things though, especially if you have an extra interface on your Scale box. I attach 2 interfaces to my main Docker/DIND container, one of which is directly LAN connected with static IP. That will route LAN traffic directly using that interface. If I want to route Internet traffic out that interface also (vs Scale's default interface) then it would take something like a ip route change default via 10.10.0.1 dev net1 to move it from using eth0 (Scale's) by default.

But what you're describing sounds like it doesn't have a direct route for the LAN. Would probably take an ip route list in the main Docker/DIND container to see what it's thinking.

It is / was an IX bug. All traffic leaving a K3S container (from Truecharts or IX) that I tested was going to the default gateway, even if on-net and relying on the GW to redirect back to the LAN. IX declined to accept this as a bug saying that this was working correctly. We had something of an argument during which it was agreed that one of us didn't know what the other was talking about when it came to routing behaviour
A Traceroute went as follows:

As far as I am aware thats still going on - but I cannot test for it atm as I cannot find a container that has traceroute any longer (not that I have tried them all and in fact don't use many any longer) and I don't feel like collecting traffic at the firewall.

Jailmaker routes properly, allows me access to the entire hardware of the host and does so simply with less overhead as far as I can tell. Long may it continue (@Jip-Hop )

That does sound crazy. I don't want to litter this gist, and can't say I fully understand your configuration, but that's definitely not what I'm seeing. Just to confirm I pulled up a Web shell session in Scale for my "IX Official" Plex container. Then did an apt-get update && apt-get install traceroute net-tools and checked traceroute and routing:

So either that was resolved (I'm on the latest Scale) or there is some other routing issue at play there.

are there PM's in github? IF you would like to take this offline / elsewhere
cos I am seeing - when I duiplicate what you have done - the same odd behaviour. I would love this to be a configuration issue

@aardvarkl Github doesn't but I'm Codelica on Discord and on the TrueNAS community there also.

and I am Confused on TN on Discord - I think I have sent you a request

At that point there is a "safe" and isolated install of Docker running which can be used

can you define safe? If you add an external interface is any network traffic to/from the Docker going through that interface or is there still shared network resources of the host?

can you define safe? If you add an external interface is any network traffic to/from the Docker going through that interface or is there still shared network resources of the host?

By safe I mean if IX decides to remove Docker from their base Scale host install (like they going to do I believe) and just keep k3s/containerd, my install of Docker (running under their system) shouldn't be affected. As it's really no different than any other custom container running. So unless they do away with all custom containers, leaving only installs from app catalogs, it should be fine. That would be extreme IMO, and even then I guess it could be done by creating a catalog and config, etc -- but would be a pain.

As far as networking goes, adding other interfaces just gives flexibility. Basically I wanted my Docker apps with their own interface (leaving NAS & Plex stuff alone on my main 10G interface). So I gave it one interface on the LAN one one that's a private network for internal service backends for other machines (dbs, message bus, etc) as I do dev work. Both show (net1, net2) in the docker container with local routes to their subnets. So local traffic from Docker apps to the LAN uses net1 for example. But by default the default internet route would be the eth0 interface that the custom container provides via k3s. But that can be changed by just changing the default route within the Docker container to point to my gateway off net1 for example.

Anyway, I have messages from you guys on Discord so we can continue there. : )

I think it makes more sense now to continue the discussion over here:
https://github.com/Jip-Hop/jailmaker/discussions

😄

Does any of you use jailmaker alongside Apps? Please let me know about your experience in this poll in order to support this pull request.