📚 Proxmox FRR OpenFabric IPv6 Initial Setup (fc00::/128 Design)

🔢 Overview

This document describes the original setup to establish an FRR (Free Range Routing) OpenFabric IS-IS based IPv6 routed mesh over Thunderbolt networking between Proxmox nodes, using static /128 loopback addresses in the fc00::/8 ULA space.

This provided:

Dynamic routed fabric between nodes
Static, globally unique IPv6 addresses per node
Fully resilient Thunderbolt mesh network

👨‍💻 Environment

Nodes: Proxmox VE hosts (pve1, pve2, pve3)
Interfaces: Thunderbolt networking (en05, en06)
Routing: IS-IS Level-2 OpenFabric
FRR Version: 8.5.2

🔹 Loopback Interface Configuration

Each Proxmox node had a static /128 IPv6 address assigned to lo.

Example for pve1:

# /etc/network/interfaces.d/thunderbolt

# Thunderbolt interfaces for pve1 (Node 81)

auto en05
iface en05 inet6 static
    pre-up ip link set $IFACE up
    mtu 65520

auto en06
iface en06 inet6 static
    pre-up ip link set $IFACE up
    mtu 65520

# Loopback for Ceph MON and routing

auto lo
iface lo inet loopback
    up ip -6 addr add fc00::81/128 dev lo

Similar for pve2 and pve3, changing fc00::82/128 and fc00::83/128 respectively.

mtu 65520 used to maximize Thunderbolt jumbo frames.
pre-up ensures interfaces come up cleanly before IP assignment.

🔹 FRR Configuration

/etc/frr/frr.conf (example for pve1)

frr version 8.5.2
frr defaults datacenter
hostname pve1
log syslog informational
service integrated-vtysh-config

interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric hello-interval 1
 openfabric hello-multiplier 3
 openfabric csnp-interval 5
 openfabric psnp-interval 2
exit

interface en06
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric hello-interval 1
 openfabric hello-multiplier 3
 openfabric csnp-interval 5
 openfabric psnp-interval 2
exit

interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive
exit

router openfabric 1
 net 49.0000.0000.0081.00
 lsp-gen-interval 5
exit

Notes:

openfabric passive on lo ensures loopback IPs are advertised but no IS-IS hellos sent on lo.
net 49.0000.0000.0081.00 sets the IS-IS NET address for pve1.
CSNP/PSNP/Hello timers tuned for fast convergence.

Similar configs on pve2 and pve3 (adjust NET addresses).

🔹 Key IS-IS/OpenFabric Settings

Setting	Value	Purpose
openfabric hello-interval	1	Faster neighbor discovery
openfabric hello-multiplier	3	Aggressive failure detection
openfabric csnp-interval	5	Faster LSP synchronization
openfabric psnp-interval	2	Faster partial database repair
lsp-gen-interval	5	Faster LSP updates

🔹 Topology

Physical Ring:

[pve1] en05 <--> en06 [pve2] en05 <--> en06 [pve3] en05 <--> en06 [pve1]

Logical Layer-2 IS-IS full mesh:

[pve1] <--> [pve2]
[pve1] <--> [pve3]
[pve2] <--> [pve3]

Each node had full dynamic IPv6 routing over Thunderbolt links.

📅 Results

Fast dynamic IPv6 routing between Proxmox nodes
Fully utilized Thunderbolt high MTU network
OpenFabric auto-discovery and auto-tuning in FRR
Stable routed foundation for Ceph MONs on fc00::XX/128

🚀 Outcome

Working IPv6-only routed mesh fabric over Thunderbolt
No bridges required
Prepared the ground for Ceph migration to dual loopbacks and future expansions!

scyto/mesh-network.md