Complete Guide (ish): Thunderbolt 4 OpenFabric Mesh on Proxmox VE 9

Acknowledgments

This guide is heavily adapted from the excellent work by @scyto.

Original gist and foundational research: https://gist.github.com/scyto/76e94832927a89d977ea989da157e9dc

The Thunderbolt hardware detection, kernel module loading, systemd configuration, and udev automation techniques are based on scyto's pioneering work on Thunderbolt networking with Proxmox. This guide adapts those proven methods for Proxmox VE 9's new native SDN OpenFabric capabilities.

Key contributions from scyto's work:

Thunderbolt interface detection and naming methodology
Kernel module and systemd configuration patterns
Udev rules and automation scripts for reliable interface bringup
MTU optimization strategies for TB4 performance
Troubleshooting approaches and verification commands

This guide builds upon that foundation to leverage PVE 9's native SDN fabric features, replacing manual OpenFabric routing configuration with GUI-managed networking.

Overview

This guide shows how to configure a 3-node Thunderbolt 4 mesh network using Proxmox VE 9's native SDN OpenFabric feature, replacing manual OpenFabric routing with GUI-managed networking.

Hardware: 3x MS01 nodes with dual TB4 ports wired in 3-ring mesh topology
Software: Proxmox VE 9.0 beta with native SDN fabric support
Network: IPv6 OpenFabric fabric for high-performance Ceph cluster traffic

Prerequisites

Physical Setup

3 nodes: Each with dual Thunderbolt 4 ports
Ring topology: n2→n3→n4→n2 using TB4 cables
Management network: Nodes accessible via standard network (10.11.11.12/13/14)

Software Requirements

Proxmox VE 9.0 beta or later
SSH access to all nodes as root
Web GUI access to PVE cluster

Phase 1: Thunderbolt Interface Detection

Step 1: Load Kernel Modules

Add required kernel modules to each node:

# On each node (n2, n3, n4):
ssh [email protected] "echo 'thunderbolt' >> /etc/modules"
ssh [email protected] "echo 'thunderbolt-net' >> /etc/modules"
ssh [email protected] "modprobe thunderbolt && modprobe thunderbolt-net"

ssh [email protected] "echo 'thunderbolt' >> /etc/modules"
ssh [email protected] "echo 'thunderbolt-net' >> /etc/modules"
ssh [email protected] "modprobe thunderbolt && modprobe thunderbolt-net"

ssh [email protected] "echo 'thunderbolt' >> /etc/modules"
ssh [email protected] "echo 'thunderbolt-net' >> /etc/modules"
ssh [email protected] "modprobe thunderbolt && modprobe thunderbolt-net"

Verification:

ssh [email protected] "lsmod | grep thunderbolt"
# Should show: thunderbolt and thunderbolt-net modules loaded

Step 2: Identify PCI Paths

Determine Thunderbolt controller PCI paths:

ssh [email protected] "lspci | grep -i thunderbolt"
# Typical output: 00:0d.2 and 00:0d.3 for MS01 nodes

Step 3: Create Systemd Link Files

Create interface renaming rules based on PCI paths:

Node n2:

ssh [email protected] "cat > /etc/systemd/network/00-thunderbolt0.link << 'EOF'
[Match]
Path=pci-0000:00:0d.2
Driver=thunderbolt-net

[Link]
MACAddressPolicy=none
Name=en05
EOF"

ssh [email protected] "cat > /etc/systemd/network/00-thunderbolt1.link << 'EOF'
[Match]
Path=pci-0000:00:0d.3
Driver=thunderbolt-net

[Link]
MACAddressPolicy=none
Name=en06
EOF"

Repeat for n3 and n4 with same PCI paths (adjust if different).

Step 4: Configure Network Interfaces

Add TB4 interfaces to network configuration:

# Add to /etc/network/interfaces on each node:
ssh [email protected] "cat >> /etc/network/interfaces << 'EOF'

auto en05
iface en05 inet manual
    mtu 65520

auto en06
iface en06 inet manual
    mtu 65520
EOF"

Repeat for n3 and n4.

Step 5: Enable systemd-networkd

# On all nodes:
ssh [email protected] "systemctl enable systemd-networkd && systemctl start systemd-networkd"
ssh [email protected] "systemctl enable systemd-networkd && systemctl start systemd-networkd"
ssh [email protected] "systemctl enable systemd-networkd && systemctl start systemd-networkd"

Step 6: Create Udev Rules and Scripts

Create automation for interface bringup on cable insertion:

Udev rules (on all nodes):

ssh [email protected] "cat > /etc/udev/rules.d/10-tb-en.rules << 'EOF'
ACTION==\"move\", SUBSYSTEM==\"net\", KERNEL==\"en05\", RUN+=\"/usr/local/bin/pve-en05.sh\"
ACTION==\"move\", SUBSYSTEM==\"net\", KERNEL==\"en06\", RUN+=\"/usr/local/bin/pve-en06.sh\"
EOF"

Bringup scripts (on all nodes):

ssh [email protected] "mkdir -p /usr/local/bin && cat > /usr/local/bin/pve-en05.sh << 'EOF'
#!/bin/bash
LOGFILE=\"/tmp/udev-debug.log\"
VERBOSE=\"\"
IF=\"en05\"
echo \"\$(date): pve-\$IF.sh triggered by udev\" >> \"\$LOGFILE\"
for i in {1..10}; do
    echo \"\$(date): Attempt \$i to bring up \$IF\" >> \"\$LOGFILE\"
    /usr/sbin/ifup \$VERBOSE \$IF >> \"\$LOGFILE\" 2>&1 && {
        echo \"\$(date): Successfully brought up \$IF on attempt \$i\" >> \"\$LOGFILE\"
        break
    }
    echo \"\$(date): Attempt \$i failed, retrying in 3 seconds...\" >> \"\$LOGFILE\"
    sleep 3
done
EOF"

ssh [email protected] "cat > /usr/local/bin/pve-en06.sh << 'EOF'
#!/bin/bash
LOGFILE=\"/tmp/udev-debug.log\"
VERBOSE=\"\"
IF=\"en06\"
echo \"\$(date): pve-\$IF.sh triggered by udev\" >> \"\$LOGFILE\"
for i in {1..10}; do
    echo \"\$(date): Attempt \$i to bring up \$IF\" >> \"\$LOGFILE\"
    /usr/sbin/ifup \$VERBOSE \$IF >> \"\$LOGFILE\" 2>&1 && {
        echo \"\$(date): Successfully brought up \$IF on attempt \$i\" >> \"\$LOGFILE\"
        break
    }
    echo \"\$(date): Attempt \$i failed, retrying in 3 seconds...\" >> \"\$LOGFILE\"
    sleep 3
done
EOF"

ssh [email protected] "chmod +x /usr/local/bin/pve-en*.sh"

Repeat script creation for n3 and n4.

Step 7: Update Initramfs and Reboot

# On all nodes:
ssh [email protected] "update-initramfs -u -k all"
ssh [email protected] "update-initramfs -u -k all"
ssh [email protected] "update-initramfs -u -k all"

# Reboot all nodes:
ssh [email protected] "reboot"
ssh [email protected] "reboot"
ssh [email protected] "reboot"

Step 8: Verify TB4 Interface Detection

After reboot, verify interfaces are visible:

ssh [email protected] "ip link show | grep en0"
# Expected output: en05 and en06 interfaces with 65520 MTU

ssh [email protected] "ip link show | grep en0"
ssh [email protected] "ip link show | grep en0"

Step 9: Optimize MTU

Set optimal TB4 MTU:

ssh [email protected] "ip link set en05 mtu 65520 && ip link set en06 mtu 65520"
ssh [email protected] "ip link set en05 mtu 65520 && ip link set en06 mtu 65520"
ssh [email protected] "ip link set en05 mtu 65520 && ip link set en06 mtu 65520"

Verification:

ssh [email protected] "ip link show en05 | grep mtu && ip link show en06 | grep mtu"
# Expected: mtu 65520 on both interfaces

Phase 2: IPv6 Forwarding Configuration

Step 10: Enable IPv6 Forwarding

CRITICAL: Use sysctl.conf, NOT /etc/network/interfaces post-up commands.

# On all nodes:
ssh [email protected] "echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.conf && sysctl -p"
ssh [email protected] "echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.conf && sysctl -p"
ssh [email protected] "echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.conf && sysctl -p"

Verification:

ssh [email protected] "sysctl net.ipv6.conf.all.forwarding"
# Expected output: net.ipv6.conf.all.forwarding = 1

Phase 3: PVE 9 SDN OpenFabric Configuration

Step 11: Create OpenFabric Fabric (GUI)

In PVE Web GUI:

Navigate: Datacenter → SDN → Fabrics
Click: Create → OpenFabric
Configure:
- Name: tb4
- IPv6 Prefix: fd00:100::/64
- Hello Interval: 3 (default)
- CSNP Interval: 10 (default)
Click: Create

Step 12: Add Nodes to Fabric (GUI)

For each node, add to the fabric:

Node n2:

Navigate: Datacenter → SDN → Fabrics → tb4
Click: Create → Node
Configure:
- Node: n2
- IPv6: fd00:100::12
- Interfaces: Select en05 and en06
Click: Create

Node n3:

IPv6: fd00:100::13
Interfaces: Select en05 and en06

Node n4:

IPv6: fd00:100::14
Interfaces: Select en05 and en06

Step 13: Configure Interface IPv6 Addresses (GUI)

Edit each node's interfaces to add point-to-point IPv6 addresses:

Node n2:

en05: fd00:100:1::0/127 (connects to n3's en06)
en06: fd00:100:3::1/127 (connects to n4's en05)

Node n3:

en05: fd00:100:2::0/127 (connects to n4's en06)
en06: fd00:100:1::1/127 (connects to n2's en05)

Node n4:

en05: fd00:100:3::0/127 (connects to n2's en06)
en06: fd00:100:2::1/127 (connects to n3's en05)

Step 14: Start FRR Service

Start the routing daemon on all nodes:

ssh [email protected] "systemctl start frr && systemctl enable frr"
ssh [email protected] "systemctl start frr && systemctl enable frr"
ssh [email protected] "systemctl start frr && systemctl enable frr"

Verification:

ssh [email protected] "systemctl status frr"
# Expected: Active (running)

Step 15: Apply SDN Configuration (GUI)

In PVE Web GUI:

Navigate: Datacenter → SDN
Click: Apply
Wait: For configuration to deploy to all nodes
Verify: No errors in the process

Step 16: Troubleshoot SDN Commit Service

Check if SDN commit service is working:

ssh [email protected] "systemctl status pve-sdn-commit"

If failed with IPv6 forwarding error:

Remove problematic lines from /etc/network/interfaces:

ssh [email protected] "sed -i '/post-up sysctl -w net.ipv6.conf.all.forwarding=1/d' /etc/network/interfaces"

Restart SDN commit service:

ssh [email protected] "systemctl restart pve-sdn-commit && systemctl status pve-sdn-commit"

Repeat for all nodes if needed.
Re-apply SDN configuration in GUI.

Phase 4: Verification and Testing

Step 17: Verify Router ID Loopback Interfaces

Check that PVE created router-ID loopback interfaces:

ssh [email protected] "ip -6 addr show | grep fd00:100::12"
# Expected: inet6 fd00:100::12/128 scope global

ssh [email protected] "ip -6 addr show | grep fd00:100::13"
ssh [email protected] "ip -6 addr show | grep fd00:100::14"

Step 18: Verify Interface IPv6 Configuration

Check that TB4 interfaces have correct IPv6 addresses:

ssh [email protected] "ip -6 addr show en05 && ip -6 addr show en06"
# Expected: IPv6 addresses as configured, interfaces UP, mtu 65520

ssh [email protected] "ip -6 addr show en05 && ip -6 addr show en06"
ssh [email protected] "ip -6 addr show en05 && ip -6 addr show en06"

Step 19: Test OpenFabric Connectivity

Test ping between router IDs to verify IS-IS routing:

# Test from n2:
ssh [email protected] "ping6 -c 3 fd00:100::13"
ssh [email protected] "ping6 -c 3 fd00:100::14"

# Test from n3:
ssh [email protected] "ping6 -c 3 fd00:100::12"
ssh [email protected] "ping6 -c 3 fd00:100::14"

# Test from n4:
ssh [email protected] "ping6 -c 3 fd00:100::12"
ssh [email protected] "ping6 -c 3 fd00:100::13"

Expected Results:

0% packet loss
Sub-millisecond latency (~0.3-0.8ms)
All router IDs reachable from all nodes

Step 20: Verify GUI Integration

Check that the fabric appears in PVE GUI:

Navigate: Datacenter → SDN → Fabrics → tb4
Verify: All nodes listed with correct IPv6 router IDs
Check: Interface status shows as UP

Troubleshooting

Common Issues

1. TB4 Interfaces Not Detected:

Verify kernel modules loaded: lsmod | grep thunderbolt
Check PCI paths: lspci | grep -i thunderbolt
Review systemd link files in /etc/systemd/network/
Check udev debug log: tail -f /tmp/udev-debug.log

2. SDN Commit Service Failed:

Check service status: systemctl status pve-sdn-commit
Review logs: journalctl -u pve-sdn-commit
Remove post-up sysctl commands from /etc/network/interfaces
Use /etc/sysctl.conf for IPv6 forwarding instead

3. Router IDs Not Reachable:

Verify FRR service running: systemctl status frr
Check IPv6 forwarding: sysctl net.ipv6.conf.all.forwarding
Re-apply SDN configuration in GUI
Check IS-IS neighbors: vtysh -c "show isis neighbor"

4. Interface MTU Issues:

Set MTU manually: ip link set en05 mtu 65520
Update /etc/network/interfaces with mtu 65520
Restart networking: systemctl restart networking

Performance Optimization

Expected Performance:

Latency: <1ms between router IDs
MTU: 65520 (jumbo frames)
Throughput: Near TB4 maximum (~40Gbps)

Monitoring Commands:

# Check IS-IS routing table:
ssh [email protected] "ip -6 route | grep fd00:100"

# Monitor interface statistics:
ssh [email protected] "ip -s link show en05"

# Test bandwidth (between nodes):
ssh [email protected] "iperf3 -s -B fd00:100::12"
ssh [email protected] "iperf3 -c fd00:100::12 -B fd00:100::13"

Next Steps: Ceph Integration

Once the OpenFabric mesh is operational:

Configure Ceph to use fd00:100::/64 as cluster network
Update ceph.conf with IPv6 addresses
Test Ceph performance over TB4 mesh
Implement failover testing by unplugging TB4 cables
Monitor IS-IS for automatic route adaptation

Summary

This guide successfully replaces some manual OpenFabric configuration with PVE 9's native SDN fabric networking, providing:

GUI-managed networking instead of manual config files
Automatic IS-IS routing with failover capabilities
High-performance IPv6 mesh optimized for Ceph traffic
65520 MTU for maximum TB4 throughput
Sub-millisecond latency between all nodes

taslabs-net/pve9tb4-original.md

Complete Guide (ish): Thunderbolt 4 OpenFabric Mesh on Proxmox VE 9

Acknowledgments

Overview

Prerequisites

Physical Setup

Software Requirements

Phase 1: Thunderbolt Interface Detection

Step 1: Load Kernel Modules

Step 2: Identify PCI Paths

Step 3: Create Systemd Link Files

Step 4: Configure Network Interfaces

Step 5: Enable systemd-networkd

Step 6: Create Udev Rules and Scripts

Step 7: Update Initramfs and Reboot

Step 8: Verify TB4 Interface Detection

Step 9: Optimize MTU

Phase 2: IPv6 Forwarding Configuration

Step 10: Enable IPv6 Forwarding

Phase 3: PVE 9 SDN OpenFabric Configuration

Step 11: Create OpenFabric Fabric (GUI)

Step 12: Add Nodes to Fabric (GUI)

Step 13: Configure Interface IPv6 Addresses (GUI)

Step 14: Start FRR Service

Step 15: Apply SDN Configuration (GUI)

Step 16: Troubleshoot SDN Commit Service

Phase 4: Verification and Testing

Step 17: Verify Router ID Loopback Interfaces

Step 18: Verify Interface IPv6 Configuration

Step 19: Test OpenFabric Connectivity

Step 20: Verify GUI Integration

Troubleshooting

Common Issues

Performance Optimization

Next Steps: Ceph Integration

Summary