OPNsense - Intel i226 NIC tunables

opnsense-i226-nic-tunables.txt

<item>
  <tunable>net.inet.icmp.drop_redirect</tunable>
  <value>1</value>
  <descr/>
</item>
<item>
  <tunable>net.isr.bindthreads</tunable>
  <value>1</value>
  <descr/>
</item>
<item>
  <tunable>net.isr.maxthreads</tunable>
  <value>-1</value>
  <descr/>
</item>
<item>
  <tunable>net.inet.rss.bits</tunable>
  <value>2</value>
  <descr/>
</item>
<item>
  <tunable>net.inet.rss.enabled</tunable>
  <value>1</value>
  <descr/>
</item>
<item>
  <tunable>kern.ipc.nmbclusters</tunable>
  <value>1000000</value>
  <descr/>
</item>
<item>
  <tunable>kern.ipc.nmbjumbop</tunable>
  <value>524288</value>
  <descr/>
</item>
<item>
  <tunable>hw.intr_storm_threshold</tunable>
  <value>10000</value>
  <descr/>
</item>
<item>
  <tunable>net.inet.ip.intr_queue_maxlen</tunable>
  <value>3000</value>
  <descr/>
</item>
<item>
  <tunable>net.inet6.ip6.intr_queue_maxlen</tunable>
  <value>3000</value>
  <descr/>
</item>
<item>
  <tunable>hw.ix.flow_control</tunable>
  <value>0</value>
  <descr/>
</item>
<item>
  <tunable>dev.igc.0.fc</tunable>
  <value>0</value>
  <descr/>
</item>
<item>
  <tunable>dev.igc.1.fc</tunable>
  <value>0</value>
  <descr/>
</item>
<item>
  <tunable>dev.igc.2.fc</tunable>
  <value>0</value>
  <descr/>
</item>
<item>
  <tunable>dev.igc.3.fc</tunable>
  <value>0</value>
  <descr/>
</item>
<item>
  <tunable>net.isr.dispatch</tunable>
  <value>deferred</value>
  <descr/>
</item>

My current settings (i226v yes of course, n100):

# Too high (3500+) would cause jitter or long connection establish delay
net.inet6.ip6.intr_queue_maxlen=3000
net.inet.ip.intr_queue_maxlen=3000
# Too high (16000+) would cause jitter or long connection establish delay
hw.igc.max_interrupt_rate=12000 # boot-time, needs reboot
net.inet.tcp.soreceive_stream=1 # boot-time
net.isr.maxthreads=-1 # boot-time
net.inet.rss.enabled=1 # boot-time
net.inet.rss.bits=2 # boot-time
net.isr.bindthreads=1 # boot-time
hw.igc.rx_process_limit=-1
net.isr.dispatch=direct 
hw.ix.flow_control=0
dev.igc.0.fc=0
dev.igc.1.fc=0
dev.igc.2.fc=0
dev.igc.3.fc=0
dev.igc.4.fc=0

My line is sold as 300mbps, both ul/dl peaks stable at 311mbps. Its a GPON/ONT setup.

I use fqcodel pipe on WAN-Download with bandwidth=295 (mbps), this is accompanied by a WAN-Download-Queue with weight=100 and WAN-Download-Any-Rule attached to WAN-Download-Queue.

Using fqcodel on the upload pipe only resulted in worse throughput and latency, so I assume the ISP is already doing some sort of shaping on the upload. I also believe this (potential shaping from ISP) might be the reason I had to set the bandwidths to theoretical speedtest max (311mbps) and NOT 80-85%. It was unnaturally rock-stable at 311mbps almost like it targets it deliberately.. reducing the bandwidth in any way would just make worse latency and over-compensate with bandwidth limits (a 280mbps limit would hit 180-200 for instance)

Settings for my upload pipe:

bandwidth=311 # mbps
scheduler=qfq # should be more lightweight than wfq, didn't spot a difference.
enable_codel=y
codel_target=14
codel_interval=140

Read (target+interval)[https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html#target-interval], don't just use my settings. I also found that these settings (target+interval) did very little when used with only 2 [b]fqcodel[/b] pipes without queue+rules. Only after I finally attempted another scheduler on the upload-pipe I finally had progress and now these kicked into effect.

For all my queues below I have
source=$MY_SUBNETS_CIDR_AND_WAN_IP + direction=out for upload pipe rules.

destination=$MY_SUBNETS_CIDR_AND_WAN_IP + direction=in for download pipe rules.

My upload-pipe with the quick-fair-queue (qfq) is combined with

• WAN-Upload-ICMP-queue weight=100, WAN-Upload-ICMP-Rule
• WAN-Download-ICMP,DNS,NTP,DHCP-queue weight=100, WAN-Download-ICMP,DNS,NTP,DHCP-Rule

Next a catch-all down-prioritize

• WAN-Download-Rule weight=1
• WAN-Upload-Any-queue weight=1, WAN-Upload-Any-Rule
• WAN-Download-Any-queue weight=1, WAN-Download-Any-Rule

I have never had so fast internet, browser and such stable loaded latency.

I recommend and use https://speed.cloudflare.com as it is the most advanced in terms of both data and function. It both downloads and uploads meanwhile it tests with pauses in-between which makes a really good realworld speedtest that highlights all issues for finetuning.

My results under load;
loaded latency dl (9ms), up (9ms), jitter dl (0.684ms), ul (2.94ms)

Flood-pinging under load (while qbittorrent is 30mb/s on multiple torrents)

ping -i0.002 -c1000 1.1.1.1

1000 packets transmitted, 1000 received, 0% packet loss, time 9069ms
rtt min/avg/max/mdev = 8.531/9.070/13.014/0.381 ms, pipe 2

9ms avg and 13ms max when your rtt is 14ms and qbittorrent is maxing out is pretty darn good! Anyways these are my findings after speedtesting for 3 days straight and tuning lol. Enjoy!