Contains notes about using the Versal™ Devices Integrated 100G Multirate Ethernet MAC Subsystem IP.
Differences between Designing with UltraScale+ CMAC and Versal MRMAC has some useful links.
With Viavdo 2025.2 adding the MRMAC IP to a block design for a Versal AI Edge gets IP v3.2. However, attempting to select Documentation -> Product Guide on the Re-customize IP dialog get the following error:
This document was not found in the current document catalog: https://www.xilinx.com/cgi-bin/docs/ipdoc?c=mrmac;v=v3_2;d=pg314-versal-mrmac.pdf
Please make sure the catalog is up to date. The program will now try to open the document on the Xilinx web site.
As of 14 Mar 2026 the https://www.xilinx.com/cgi-bin/docs/ipdoc?c=mrmac;v=v3_2;d=pg314-versal-mrmac.pdf now reports:
pg314-versal-mrmac.pdf
You have successfully reached the future site of pg314-versal-mrmac.pdf.
- Core: mrmac
- Version: v3_2
- Document: pg314-versal-mrmac.pdf URL Base: https://www.xilinx.com
I.e. looks like the document will become available.
Also can't find PG314 v3.2 on the Xilinx web site.
Most recent version of https://docs.amd.com/r/en-US/pg314-versal-mrmac/ available is 3.1.
The Vivado 2025.2 Documentation -> Change Log has the following for 3.2:
2025.2:
- Version 3.2
- General: Versal New GT Wiz support added for Mixed mode.
- General: Enable visibility of hierarchy, cells and schematics in Vivado GUI
- General: Device support added
- General: Removed quiet switch from create_waiver command in XDC files.
Both MRMAC 3.1 and 3.2 IP have a gtpowergood_in port, which can't find described in PG314.
On running Block Automation:
- A
gtpowergood_inExternal Input port is created, connected to the mrmac block. - A
gtpowergoodExternal Output port is crested, connected to the gt_quad_base Versal Adaptive SoC Transeivers Wizard.
For now, assume OK to connect the gt_quad_base gtpowergood output to the MRMAC gtpowergood_in input.
In mrmac-registers-v3-0.xlsx:
- The Legend sheet has:
Access Type Description RO ATTR Read Only ; Attribute1 RW ATTR Read Write ; Attribute1 Notes:
1 . Attribute access type indicates a register that can be configued at device programming. A reset does not affect the value stored in such a register.
- The Description sheet Type column does use "RO ATTR" and "RW ATTR".
- Whereas the MRMAC_REGISTERS_ALL sheet instead uses "RO DRP" and "RW DRP".
MRMAC Memory Map gives the base address of the Configuration registers, Status registers, and Statistics counters for each port.
Based upon that at least 15 address bits are needed to decode the registers.
The MRMAC on the block diagram shows the s_axi interface has s_axi_araddr[31:0] and s_axi_awaddr[31:0]. I.e. apparently 32 address bits.
Looking at a schematic shows only 16 address bits out of the axi_apb_bridge. The hard MRMAC block has a APB3_PADDR[15:0] address input.
The auto assignment of address segments allocated 64K for the MRMAC s_axi.
PG314 Table 33: AXI4-Lite Interface Signal Descriptions has:
| Port Name | I/O | Description |
|---|---|---|
| s_axi_areset | I | Active-High reset for the AXI4-Lite port. Asserting this reset alters the Port control logic and stops any in-flight writes/reads. It does not reset the internal configuration registers with exceptions listed below |
Whereas Versal Adapative SoC 100G Multirate Ethernet MAC (MRMAC) block in a diagram has a s_axi_aresetn input with an active low polarity in the properties.
The VD100_10G_ether_dual.gen/sources_1/bd/VD100_10G_ether_dual/bd/mrmac_10G_dual_inst_0/ip/mrmac_10G_dual_inst_0_mrmac_0_0/mrmac_v3_2_0/mrmac_10G_dual_inst_0_mrmac_0_0_wrapper.v generated output product has
) i_mrmac_10G_dual_inst_0_mrmac_0_0_top (
//<snip>
.apb3_preset (~s_axi_aresetn ),I.e. looks like PG314 is incorrect, in that:
- For the soft IP wrapper there is a
s_axi_aresetnactive low, rather thans_axi_aresetactive high reset input. - The hard IP has the active high
apb3_presetreset input - The soft IP wrapper inverts the
s_axi_aresetninput to drive the hard IPapb3_preset.
PG314 and the linked mrmac-registers-v3.0.xlsx don't summarise which statistic counters are common across all ports, and which are different. Initially wrote code which used the offset from mrmac-registers-v3.0.xlsx for port 0 to apply to all ports.
With the VD100_10G_ether_dual design loaded, a DAC cable between ports 0 and 1 and FEC disabled then the following statistics were reported when no packets being sent:
VD100_10G_ether_dual port 0 statistics (over 10.000 secs):
TX_CYCLE_COUNT : 6445567702
RX_CYCLE_COUNT : 6445567701
VD100_10G_ether_dual port 1 statistics (over 10.000 secs):
TX_CYCLE_COUNT : 6445565888
RX_CYCLE_COUNT : 6445565888
RX_BIP_ERR_0 : 6445565888
RX_BIP_ERR_1 : 6445565888
RX_BIP_ERR_2 : 6445565888
RX_BIP_ERR_3 : 6445565888
RX_BIP_ERR_4 : 6445565888
RX_BIP_ERR_5 : 6445565888
RX_BIP_ERR_6 : 6445565888
RX_BIP_ERR_7 : 6445565888
RX_BIP_ERR_8 : 6445565888
RX_BIP_ERR_9 : 6445565888
RX_BIP_ERR_10 : 6445565888
RX_BIP_ERR_11 : 6445565888
RX_BIP_ERR_12 : 6445565888
RX_BIP_ERR_13 : 6445565888
RX_BIP_ERR_14 : 6445565888
RX_BIP_ERR_15 : 6445565888
RX_BIP_ERR_16 : 6445565888
RX_BIP_ERR_17 : 6445565888
RX_BIP_ERR_18 : 6445565888
RX_BIP_ERR_19 : 6445565888
RX_FRAMING_ERR_0 : 6445565888
RX_FRAMING_ERR_1 : 6445565888
RX_FRAMING_ERR_2 : 6445565888
RX_FRAMING_ERR_3 : 6445565888
RX_FRAMING_ERR_4 : 6445565888
RX_FRAMING_ERR_5 : 6445565888
RX_FRAMING_ERR_6 : 6445565888
RX_FRAMING_ERR_7 : 6445565888
RX_FRAMING_ERR_8 : 6445565888
RX_FRAMING_ERR_9 : 6445565888
RX_FRAMING_ERR_10 : 6445565888
RX_FRAMING_ERR_11 : 6445565888
RX_FRAMING_ERR_12 : 6445565888
RX_FRAMING_ERR_13 : 6445565888
RX_FRAMING_ERR_14 : 6445565888
RX_FRAMING_ERR_15 : 6445565888
RX_FRAMING_ERR_16 : 6445565888
RX_FRAMING_ERR_17 : 6445565888
RX_FRAMING_ERR_18 : 6445565888
RX_FRAMING_ERR_19 : 6445565888
RX_FEC_CW_1 : 6445565888
RX_FEC_CW_2 : 6445565888
RX_FEC_CW_3 : 6445565888
RX_FEC_CORRECTED_CW_1 : 6445565888
RX_FEC_CORRECTED_CW_2 : 6445565888
RX_FEC_CORRECTED_CW_3 : 6445565888
RX_FEC_UNCORRECTED_CW_1 : 6445565888
RX_FEC_UNCORRECTED_CW_2 : 6445565888
RX_FEC_UNCORRECTED_CW_3 : 6445565888
RX_FEC_BIT_ERR_0TO1_1 : 6445565888
RX_FEC_BIT_ERR_0TO1_2 : 6445565888
RX_FEC_BIT_ERR_0TO1_3 : 6445565888
RX_FEC_BIT_ERR_1TO0_1 : 6445565888
RX_FEC_BIT_ERR_1TO0_2 : 6445565888
RX_FEC_BIT_ERR_1TO0_3 : 6445565888
RX_FEC_ERR_COUNT_1 : 6445565888
RX_FEC_ERR_COUNT_2 : 6445565888
RX_FEC_ERR_COUNT_3 : 6445565888
And after enabling "IEEE 802.3 CL74 FEC":
VD100_10G_ether_dual port 0 statistics (over 10.000 secs):
TX_CYCLE_COUNT : 6445218764
RX_CYCLE_COUNT : 6445218763
RX_FEC_CW_0 : 48827415
VD100_10G_ether_dual port 1 statistics (over 10.000 secs):
TX_CYCLE_COUNT : 6445218758
RX_CYCLE_COUNT : 6445218759
RX_BIP_ERR_0 : 6445218759
RX_BIP_ERR_1 : 6445218759
RX_BIP_ERR_2 : 6445218759
RX_BIP_ERR_3 : 6445218759
RX_BIP_ERR_4 : 6445218759
RX_BIP_ERR_5 : 6445218759
RX_BIP_ERR_6 : 6445218759
RX_BIP_ERR_7 : 6445218759
RX_BIP_ERR_8 : 6445218759
RX_BIP_ERR_9 : 6445218759
RX_BIP_ERR_10 : 6445218759
RX_BIP_ERR_11 : 6445218759
RX_BIP_ERR_12 : 6445218759
RX_BIP_ERR_13 : 6445218759
RX_BIP_ERR_14 : 6445218759
RX_BIP_ERR_15 : 6445218759
RX_BIP_ERR_16 : 6445218759
RX_BIP_ERR_17 : 6445218759
RX_BIP_ERR_18 : 6445218759
RX_BIP_ERR_19 : 6445218759
RX_FRAMING_ERR_0 : 6445218759
RX_FRAMING_ERR_1 : 6445218759
RX_FRAMING_ERR_2 : 6445218759
RX_FRAMING_ERR_3 : 6445218759
RX_FRAMING_ERR_4 : 6445218759
RX_FRAMING_ERR_5 : 6445218759
RX_FRAMING_ERR_6 : 6445218759
RX_FRAMING_ERR_7 : 6445218759
RX_FRAMING_ERR_8 : 6445218759
RX_FRAMING_ERR_9 : 6445218759
RX_FRAMING_ERR_10 : 6445218759
RX_FRAMING_ERR_11 : 6445218759
RX_FRAMING_ERR_12 : 6445218759
RX_FRAMING_ERR_13 : 6445218759
RX_FRAMING_ERR_14 : 6445218759
RX_FRAMING_ERR_15 : 6445218759
RX_FRAMING_ERR_16 : 6445218759
RX_FRAMING_ERR_17 : 6445218759
RX_FRAMING_ERR_18 : 6445218759
RX_FRAMING_ERR_19 : 6445218759
RX_FEC_CW_0 : 48827414
RX_FEC_CW_1 : 6445218759
RX_FEC_CW_2 : 6445218759
RX_FEC_CW_3 : 6445218759
RX_FEC_CORRECTED_CW_1 : 6445218759
RX_FEC_CORRECTED_CW_2 : 6445218759
RX_FEC_CORRECTED_CW_3 : 6445218759
RX_FEC_UNCORRECTED_CW_1 : 6445218759
RX_FEC_UNCORRECTED_CW_2 : 6445218759
RX_FEC_UNCORRECTED_CW_3 : 6445218759
RX_FEC_BIT_ERR_0TO1_1 : 6445218759
RX_FEC_BIT_ERR_0TO1_2 : 6445218759
RX_FEC_BIT_ERR_0TO1_3 : 6445218759
RX_FEC_BIT_ERR_1TO0_1 : 6445218759
RX_FEC_BIT_ERR_1TO0_2 : 6445218759
RX_FEC_BIT_ERR_1TO0_3 : 6445218759
RX_FEC_ERR_COUNT_1 : 6445218759
RX_FEC_ERR_COUNT_2 : 6445218759
RX_FEC_ERR_COUNT_3 : 6445218759
The values for port 0 make sense. For port 1 the following counters have bogus values, which happen to match the value of RX_CYCLE_COUNT:
- RX_BIP_ERR_0..RX_BIP_ERR_19
- RX_FRAMING_ERR_0..RX_FRAMING_ERR_19
- RX_FEC_CW_1..RX_FEC_CW_3
- RX_FEC_CORRECTED_CW_1..RX_FEC_CORRECTED_CW_3
- RX_FEC_UNCORRECTED_CW_1..RX_FEC_UNCORRECTED_CW_3
- RX_FEC_BIT_ERR_0TO1_1..RX_FEC_BIT_ERR_0TO1_3
- RX_FEC_BIT_ERR_1TO0_1..RX_FEC_BIT_ERR_1TO0_3
- RX_FEC_ERR_COUNT_1..RX_FEC_ERR_COUNT_3
The absolute offset for the LSB registers of the port 1 statistics counters with the bogus values the software was reading are:
- 0x1C08..0x1CA0
- 0x1CA8..0x1D40
- 0x1D78..0x1D88
- 0x1D98..0x1DA8
- 0x1DB8..0x1DC8
- 0x1DD8..0x1DE8
- 0x1DF8..0x1E08
- 0x1E18..0x1E28
The above register offsets are not shown as mapped in mrmac-registers-v3.0.xlsx, which explains why the software would read bogus values.
The Customize the IP step in the PG314 MRMAC Example Design Simulation and Validation Steps has:
Select the required GT Wizard. For the legacy GT Wizard, enable Use Legacy GT Wizard in Example Design and disable it for the New GT wizard selection. This option will be deprecated in a future release.
If try and run block automation with the New GT wizard no block automation occurs and the following is reported in the TCL console:
INFO:: Block Automation is not supported with Versal Adaptive SoC GT Wizard Subsystem
Occurs with Vivado 2025.1 and 2025.2.
If run the block automation with the Legacy GT Wizzard then the following MRMAC clocks get connected to external interfaces, rather than to the Utility Buffer clock buffer outputs:
rx_alt_serdes_clk[3:0]rx_core_clk[3:0]rx_serdes_clk[3:0]tx_alt_serdes_clk[3:0]tx_core_clk[3:0]
Added a MRMAC block in a hierarchy sub-block. Wanted to use a sub-block to hide the GT blocks and connections on the top-level diagram. Attempting to run Block Automation fails with:
apply_bd_automation -rule xilinx.com:bd_rule:mrmac -config { DataPath_Interface_Connection {Auto} Lane0_selection {NULL} Lane1_selection {NULL} Lane2_selection {NULL} Lane3_selection {NULL} Quad0_selection {NULL} Quad1_selection {NULL} Quad2_selection {NULL} Quad3_selection {NULL}} [get_bd_cells mrac_10G_dual/mrmac_0]
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT0_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT0_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT1_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT1_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT2_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT2_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT3_ADD_CONFIG_EN' from 'false' to 'true' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'PROT3_ADD_CONFIG_FILE' from 'no_addn_file_loaded' to 'undef' has been ignored for IP 'mrac_10G_dual/gt_quad_base'
INFO: [BD_TCL-6] Checking if the following IPs exist in the project's IP catalog:
xilinx.com:ip:mrmac:3.2 xilinx.com:ip:bufg_gt:1.0 xilinx.com:ip:gt_quad_base:1.1 xilinx.com:inline_hdl:ilconstant:* .
ERROR: [Common 17-55] 'get_property' expects at least one object.
Resolution: If [get_<value>] was used to populate the object, check to make sure this command returns at least one valid object.
ERROR: [BD 41-2168] Errors found in procedure apply_rule:
INFO: [BD 5-145] Automation rule xilinx.com:bd_rule:mrmac was not applied to object mrmac_0
The error messages don't specify where in the block automation the problem occured.
Seen trying a xcve2302 in Vivado 2025.2 and a xcvm1802 in Vivado 2025.1.
PG314 doesn't seem to explicity define the maximum packet size which can be transmitted or received.
In mrmac-registers-v3.0.xlsx the CONFIGURATION_RX_MTU_x registers have:
| Attribute | Contents |
|---|---|
| Bits | 30:16 |
| Default | 9600 |
| Description | Port 0 Maximum Packet Length. Any packet longer than this value is considered to be oversized. The allowed value for this register can range from 64 to 16,383. |
Given the value for the maximum packet length being 16,383 initially assumed that was the maximum possible. Albeit given there are 15 bits in the maximum length register, that would put the maxium at 32727. TBD what the actual maximum limit is.
Both PG314 and mrmac-registers-v3.0.xlsx define the histogram registers indicate statistics are mainainted for packets sized between:
- 64
- 65-127
- 128-255
- 256-511
- 512-1023
- 1024-1518
- 1519-1522
- 1523-1548
- 1549-2047
- 2048-4095
- 4096-8191
- 8192-9215
mrmac-registers-v3.0.xlsx also shows statistics registers which count:
- less than 64
- more than 9215
If have a working design with two MRMAC ports could try exchanging packets between the ports to test the actual maximum, without being limited by the maximum in a switch / other host.
Set the MTU to 16383 which is the maximum from mrmac-registers-v3.0.xlsx. The TX_TOTAL_BYTES, TX_TOTAL_GOOD_BYTES, RX_TOTAL_BYTES and RX_TOTAL_GOOD_BYTES values all match the expected values and the test passes:
$ mrmac_ethernet/mrmac_loopback_test -n 0:1
Opening device 0000:10:00.0 (10ee:b044) with IOMMU group 10
Enabled bus master for 0000:10:00.0
Warning: Device device 0000:10:00.0 (10ee:b044) has reduced bandwidth
Max width x4 speed 16 GT/s. Negotiated width x4 speed 8 GT/s
Waiting to link to be ready to receive.
Link ready at 10 Gb/s (latched rx_status initial 0x00000180 last 0x00000003)
Testing VD100_10G_ether_dual Tx port 0 Rx Port 1 with 16320 packet lengths (including FCS) from 64 to 16383 bytes
VD100_10G_ether_dual port 0 statistics (over 0.537 secs):
TX_CYCLE_COUNT : 346254116
TX_TOTAL_PACKETS : 16320
TX_TOTAL_GOOD_PACKETS : 16320
TX_TOTAL_BYTES : 134207520
TX_TOTAL_GOOD_BYTES : 134207520
TX_PACKET_64_BYTES : 1
TX_PACKET_65_127_BYTES : 63
TX_PACKET_128_255_BYTES : 128
TX_PACKET_256_511_BYTES : 256
TX_PACKET_512_1023_BYTES : 512
TX_PACKET_1024_1518_BYTES: 495
TX_PACKET_1519_1522_BYTES: 4
TX_PACKET_1523_1548_BYTES: 26
TX_PACKET_1549_2047_BYTES: 499
TX_PACKET_2048_4095_BYTES: 2048
TX_PACKET_4096_8191_BYTES: 4096
TX_PACKET_8192_9215_BYTES: 1024
TX_PACKET_LARGE : 7168
TX_UNICAST : 16320
RX_CYCLE_COUNT : 346251681
VD100_10G_ether_dual port 1 statistics (over 0.537 secs):
TX_CYCLE_COUNT : 346252441
RX_CYCLE_COUNT : 346247025
RX_TOTAL_PACKETS : 16320
RX_TOTAL_GOOD_PACKETS : 16320
RX_TOTAL_BYTES : 134207520
RX_TOTAL_GOOD_BYTES : 134207520
RX_PACKET_64_BYTES : 1
RX_PACKET_65_127_BYTES : 63
RX_PACKET_128_255_BYTES : 128
RX_PACKET_256_511_BYTES : 256
RX_PACKET_512_1023_BYTES : 512
RX_PACKET_1024_1518_BYTES: 495
RX_PACKET_1519_1522_BYTES: 4
RX_PACKET_1523_1548_BYTES: 26
RX_PACKET_1549_2047_BYTES: 499
RX_PACKET_2048_4095_BYTES: 2048
RX_PACKET_4096_8191_BYTES: 4096
RX_PACKET_8192_9215_BYTES: 1024
RX_PACKET_LARGE : 7168
RX_UNICAST : 16320
Total byte including FCS: 134207520
Loopback test: PASS
Increase the MTU by one byte to 16384. The test passes but the TX_TOTAL_GOOD_BYTES and RX_TOTAL_GOOD_BYTES are now lower than the expected values. TX_TOTAL_BYTES and RX_TOTAL_BYTES match the expected values. TX_TOTAL_GOOD_BYTES is one byte less than expected and RX_TOTAL_GOOD_BYTES is 16384 less than expected (size of final frame sent including the FCS):
mrmac_ethernet/mrmac_loopback_test -n 0:1
Opening device 0000:10:00.0 (10ee:b044) with IOMMU group 10
Enabled bus master for 0000:10:00.0
Warning: Device device 0000:10:00.0 (10ee:b044) has reduced bandwidth
Max width x4 speed 16 GT/s. Negotiated width x4 speed 8 GT/s
Waiting to link to be ready to receive.
Link ready at 10 Gb/s (latched rx_status initial 0x00000180 last 0x00000003)
Testing VD100_10G_ether_dual Tx port 0 Rx Port 1 with 16321 packet lengths (including FCS) from 64 to 16384 bytes
VD100_10G_ether_dual port 0 statistics (over 0.520 secs):
TX_CYCLE_COUNT : 335311476
TX_TOTAL_PACKETS : 16321
TX_TOTAL_GOOD_PACKETS : 16321
TX_TOTAL_BYTES : 134223904
TX_TOTAL_GOOD_BYTES : 134223903
TX_PACKET_64_BYTES : 1
TX_PACKET_65_127_BYTES : 63
TX_PACKET_128_255_BYTES : 128
TX_PACKET_256_511_BYTES : 256
TX_PACKET_512_1023_BYTES : 512
TX_PACKET_1024_1518_BYTES: 495
TX_PACKET_1519_1522_BYTES: 4
TX_PACKET_1523_1548_BYTES: 26
TX_PACKET_1549_2047_BYTES: 499
TX_PACKET_2048_4095_BYTES: 2048
TX_PACKET_4096_8191_BYTES: 4096
TX_PACKET_8192_9215_BYTES: 1024
TX_PACKET_LARGE : 7169
TX_UNICAST : 16321
RX_CYCLE_COUNT : 335309124
VD100_10G_ether_dual port 1 statistics (over 0.520 secs):
TX_CYCLE_COUNT : 335309809
RX_CYCLE_COUNT : 335304572
RX_TOTAL_PACKETS : 16321
RX_TOTAL_GOOD_PACKETS : 16320
RX_TOTAL_BYTES : 134223904
RX_TOTAL_GOOD_BYTES : 134207520
RX_PACKET_64_BYTES : 1
RX_PACKET_65_127_BYTES : 63
RX_PACKET_128_255_BYTES : 128
RX_PACKET_256_511_BYTES : 256
RX_PACKET_512_1023_BYTES : 512
RX_PACKET_1024_1518_BYTES: 495
RX_PACKET_1519_1522_BYTES: 4
RX_PACKET_1523_1548_BYTES: 26
RX_PACKET_1549_2047_BYTES: 499
RX_PACKET_2048_4095_BYTES: 2048
RX_PACKET_4096_8191_BYTES: 4096
RX_PACKET_8192_9215_BYTES: 1024
RX_PACKET_LARGE : 7169
RX_OVERSIZE : 1
RX_TOOLONG : 1
RX_UNICAST : 16321
Total byte including FCS: 134223904
Loopback test: PASS
Increase the MTU to 32767 bytes and the test still passes. TX_TOTAL_BYTES and RX_TOTAL_BYTES are the expected values. TX_TOTAL_GOOD_BYTES and RX_TOTAL_GOOD_BYTES are lower than expected:
$ mrmac_ethernet/mrmac_loopback_test -n 0:1
Opening device 0000:10:00.0 (10ee:b044) with IOMMU group 10
Enabled bus master for 0000:10:00.0
Warning: Device device 0000:10:00.0 (10ee:b044) has reduced bandwidth
Max width x4 speed 16 GT/s. Negotiated width x4 speed 8 GT/s
Waiting to link to be ready to receive.
Link ready at 10 Gb/s (latched rx_status initial 0x00000180 last 0x00000003)
Testing VD100_10G_ether_dual Tx port 0 Rx Port 1 with 32704 packet lengths (including FCS) from 64 to 32767 bytes
VD100_10G_ether_dual port 0 statistics (over 1.400 secs):
TX_CYCLE_COUNT : 902202007
TX_TOTAL_PACKETS : 32704
TX_TOTAL_GOOD_PACKETS : 32704
TX_TOTAL_BYTES : 536852512
TX_TOTAL_GOOD_BYTES : 402626592
TX_PACKET_64_BYTES : 1
TX_PACKET_65_127_BYTES : 63
TX_PACKET_128_255_BYTES : 128
TX_PACKET_256_511_BYTES : 256
TX_PACKET_512_1023_BYTES : 512
TX_PACKET_1024_1518_BYTES: 495
TX_PACKET_1519_1522_BYTES: 4
TX_PACKET_1523_1548_BYTES: 26
TX_PACKET_1549_2047_BYTES: 499
TX_PACKET_2048_4095_BYTES: 2048
TX_PACKET_4096_8191_BYTES: 4096
TX_PACKET_8192_9215_BYTES: 1024
TX_PACKET_LARGE : 23552
TX_UNICAST : 32704
RX_CYCLE_COUNT : 902195656
VD100_10G_ether_dual port 1 statistics (over 1.400 secs):
TX_CYCLE_COUNT : 902200259
RX_CYCLE_COUNT : 902186156
RX_TOTAL_PACKETS : 32704
RX_TOTAL_GOOD_PACKETS : 32704
RX_TOTAL_BYTES : 536852512
RX_TOTAL_GOOD_BYTES : 402626592
RX_PACKET_64_BYTES : 1
RX_PACKET_65_127_BYTES : 63
RX_PACKET_128_255_BYTES : 128
RX_PACKET_256_511_BYTES : 256
RX_PACKET_512_1023_BYTES : 512
RX_PACKET_1024_1518_BYTES: 495
RX_PACKET_1519_1522_BYTES: 4
RX_PACKET_1523_1548_BYTES: 26
RX_PACKET_1549_2047_BYTES: 499
RX_PACKET_2048_4095_BYTES: 2048
RX_PACKET_4096_8191_BYTES: 4096
RX_PACKET_8192_9215_BYTES: 1024
RX_PACKET_LARGE : 23552
RX_UNICAST : 32704
Total byte including FCS: 536852512
Loopback test: PASS
Therefore looks like:
- Can send frames of 16384..32767 bytes (including FCS) and can be received corrected;
- However, with frames >= 16384 bytes the TX_TOTAL_GOOD_BYTES and RX_TOTAL_GOOD_BYTES undereport.
Configured MRMAC to set client0 and client1 as 10GE, with client2 and client3 as N/U.
The tdata and tkeep ports are only present for client0 and clint1 as expected.
However, the tlast, tready and tvalid ports are present for client0, client1 and client2. I.e. unexpected client2 signals. The PG314 Table 14: 10G Non-Segmented Signaling for 32 Bits shows the client2 signals should not be needed. Therefore, will try leaving the tlast, tready and tvalid ports for client2 as unconnected.
For these initial tests just checking the port status when connected to a T1700G-28TQ 3.0 switch via 10GBase-SR fibre optic modules.
When only port 0 connected to the switch, and the correponding port on the switch was up:
$ identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:01:00.0 (10ee:b044) with IOMMU group 12
Enabled bus master for 0000:01:00.0
Design VD100_10G_ether_dual:
PCI device 0000:01:00.0 rev 00 IOMMU group 12
DMA bridge bar 2 AXI Stream
Channel ID addr_alignment len_granularity num_address_bits
H2C 0 1 1 64
H2C 1 1 1 64
C2H 0 1 1 64
C2H 1 1 1 64
IIC registers at bar 0 offset 0x11000
MRMAC configuration revision: 0x00000001
Port 0:
Data rate: 10GE
GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
Rx min packet len: 64
Rx max packet len: 9600
FEC Operating Mode: FEC Disabled
TX realtime status: 0x00000000
Port TX local fault : 0
Port TX axis underflow : 0
Port TX axis error : 0
Port TX flexif error : 0
Port TX pcs bad code : 0
Port TX CL82 CL49 convert error: 0
Port TX flex fifo overflow : 0
Port TX flex fifo underflow : 0
RX realtime status: 0x00000003
Port RX Status: 1
Port RX Block Lock: 1
Port RX aligned: 0
Port RX misaligned: 0
Port RX aligned error: 0
Port RX High BER: 0
Port RX remote fault: 0
Port RX local fault: 0
Port RX internal local fault: 0
Port RX received local fault: 0
Port RX bad code: 0
Port RX bad preamble: 0
Port RX bad SFD: 0
Port RX got signal ordered set: 0
Port RX flex if error: 0
Port RX Framing Error: 0
Port RX Synced: 0
Port RX Synced Error: 0
Port RX BIP Error: 0
Port RX CL49_82 convert error: 0
Port RX pcs bad code: 0
Port RX AXIS fifo overflow: 0
Port RX AXIS error: 0
Port RX invalid start: 0
Port RX flex fifo overflow: 0
Port RX flex fifo underflow: 0
Port 1:
Data rate: 10GE
GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
Rx min packet len: 64
Rx max packet len: 9600
FEC Operating Mode: FEC Disabled
TX realtime status: 0x00000000
Port TX local fault : 0
Port TX axis underflow : 0
Port TX axis error : 0
Port TX flexif error : 0
Port TX pcs bad code : 0
Port TX CL82 CL49 convert error: 0
Port TX flex fifo overflow : 0
Port TX flex fifo underflow : 0
RX realtime status: 0x00000180
Port RX Status: 0
Port RX Block Lock: 0
Port RX aligned: 0
Port RX misaligned: 0
Port RX aligned error: 0
Port RX High BER: 0
Port RX remote fault: 0
Port RX local fault: 1
Port RX internal local fault: 1
Port RX received local fault: 0
Port RX bad code: 0
Port RX bad preamble: 0
Port RX bad SFD: 0
Port RX got signal ordered set: 0
Port RX flex if error: 0
Port RX Framing Error: 0
Port RX Synced: 0
Port RX Synced Error: 0
Port RX BIP Error: 0
Port RX CL49_82 convert error: 0
Port RX pcs bad code: 0
Port RX AXIS fifo overflow: 0
Port RX AXIS error: 0
Port RX invalid start: 0
Port RX flex fifo overflow: 0
Port RX flex fifo underflow: 0
When only port 1 connected to the switch, and the correponding port on the switch was up:
$ identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:01:00.0 (10ee:b044) with IOMMU group 12
Enabled bus master for 0000:01:00.0
Design VD100_10G_ether_dual:
PCI device 0000:01:00.0 rev 00 IOMMU group 12
DMA bridge bar 2 AXI Stream
Channel ID addr_alignment len_granularity num_address_bits
H2C 0 1 1 64
H2C 1 1 1 64
C2H 0 1 1 64
C2H 1 1 1 64
IIC registers at bar 0 offset 0x11000
MRMAC configuration revision: 0x00000001
Port 0:
Data rate: 10GE
GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
Rx min packet len: 64
Rx max packet len: 9600
FEC Operating Mode: FEC Disabled
TX realtime status: 0x00000000
Port TX local fault : 0
Port TX axis underflow : 0
Port TX axis error : 0
Port TX flexif error : 0
Port TX pcs bad code : 0
Port TX CL82 CL49 convert error: 0
Port TX flex fifo overflow : 0
Port TX flex fifo underflow : 0
RX realtime status: 0x00000180
Port RX Status: 0
Port RX Block Lock: 0
Port RX aligned: 0
Port RX misaligned: 0
Port RX aligned error: 0
Port RX High BER: 0
Port RX remote fault: 0
Port RX local fault: 1
Port RX internal local fault: 1
Port RX received local fault: 0
Port RX bad code: 0
Port RX bad preamble: 0
Port RX bad SFD: 0
Port RX got signal ordered set: 0
Port RX flex if error: 0
Port RX Framing Error: 0
Port RX Synced: 0
Port RX Synced Error: 0
Port RX BIP Error: 0
Port RX CL49_82 convert error: 0
Port RX pcs bad code: 0
Port RX AXIS fifo overflow: 0
Port RX AXIS error: 0
Port RX invalid start: 0
Port RX flex fifo overflow: 0
Port RX flex fifo underflow: 0
Port 1:
Data rate: 10GE
GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
Rx min packet len: 64
Rx max packet len: 9600
FEC Operating Mode: FEC Disabled
TX realtime status: 0x00000000
Port TX local fault : 0
Port TX axis underflow : 0
Port TX axis error : 0
Port TX flexif error : 0
Port TX pcs bad code : 0
Port TX CL82 CL49 convert error: 0
Port TX flex fifo overflow : 0
Port TX flex fifo underflow : 0
RX realtime status: 0x00000003
Port RX Status: 1
Port RX Block Lock: 1
Port RX aligned: 0
Port RX misaligned: 0
Port RX aligned error: 0
Port RX High BER: 0
Port RX remote fault: 0
Port RX local fault: 0
Port RX internal local fault: 0
Port RX received local fault: 0
Port RX bad code: 0
Port RX bad preamble: 0
Port RX bad SFD: 0
Port RX got signal ordered set: 0
Port RX flex if error: 0
Port RX Framing Error: 0
Port RX Synced: 0
Port RX Synced Error: 0
Port RX BIP Error: 0
Port RX CL49_82 convert error: 0
Port RX pcs bad code: 0
Port RX AXIS fifo overflow: 0
Port RX AXIS error: 0
Port RX invalid start: 0
Port RX flex fifo overflow: 0
Port RX flex fifo underflow: 0
When both ports 0 and 1 were connected to the switch and were up at the switch end:
$ identify_pcie_fpga_design/display_identified_pcie_fpga_designs
Opening device 0000:01:00.0 (10ee:b044) with IOMMU group 12
Enabled bus master for 0000:01:00.0
Design VD100_10G_ether_dual:
PCI device 0000:01:00.0 rev 00 IOMMU group 12
DMA bridge bar 2 AXI Stream
Channel ID addr_alignment len_granularity num_address_bits
H2C 0 1 1 64
H2C 1 1 1 64
C2H 0 1 1 64
C2H 1 1 1 64
IIC registers at bar 0 offset 0x11000
MRMAC configuration revision: 0x00000001
Port 0:
Data rate: 10GE
GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
Rx min packet len: 64
Rx max packet len: 9600
FEC Operating Mode: FEC Disabled
TX realtime status: 0x00000000
Port TX local fault : 0
Port TX axis underflow : 0
Port TX axis error : 0
Port TX flexif error : 0
Port TX pcs bad code : 0
Port TX CL82 CL49 convert error: 0
Port TX flex fifo overflow : 0
Port TX flex fifo underflow : 0
RX realtime status: 0x00000003
Port RX Status: 1
Port RX Block Lock: 1
Port RX aligned: 0
Port RX misaligned: 0
Port RX aligned error: 0
Port RX High BER: 0
Port RX remote fault: 0
Port RX local fault: 0
Port RX internal local fault: 0
Port RX received local fault: 0
Port RX bad code: 0
Port RX bad preamble: 0
Port RX bad SFD: 0
Port RX got signal ordered set: 0
Port RX flex if error: 0
Port RX Framing Error: 0
Port RX Synced: 0
Port RX Synced Error: 0
Port RX BIP Error: 0
Port RX CL49_82 convert error: 0
Port RX pcs bad code: 0
Port RX AXIS fifo overflow: 0
Port RX AXIS error: 0
Port RX invalid start: 0
Port RX flex fifo overflow: 0
Port RX flex fifo underflow: 0
Port 1:
Data rate: 10GE
GT Quad Operating mode: 10GE Wide, 10.3125 Gb/s, 32 bits, 322.2656 MHz, NRZ
AXI4-Stream Mode: Independent, 32 bits, Non-Segmented
Rx min packet len: 64
Rx max packet len: 9600
FEC Operating Mode: FEC Disabled
TX realtime status: 0x00000000
Port TX local fault : 0
Port TX axis underflow : 0
Port TX axis error : 0
Port TX flexif error : 0
Port TX pcs bad code : 0
Port TX CL82 CL49 convert error: 0
Port TX flex fifo overflow : 0
Port TX flex fifo underflow : 0
RX realtime status: 0x00000003
Port RX Status: 1
Port RX Block Lock: 1
Port RX aligned: 0
Port RX misaligned: 0
Port RX aligned error: 0
Port RX High BER: 0
Port RX remote fault: 0
Port RX local fault: 0
Port RX internal local fault: 0
Port RX received local fault: 0
Port RX bad code: 0
Port RX bad preamble: 0
Port RX bad SFD: 0
Port RX got signal ordered set: 0
Port RX flex if error: 0
Port RX Framing Error: 0
Port RX Synced: 0
Port RX Synced Error: 0
Port RX BIP Error: 0
Port RX CL49_82 convert error: 0
Port RX pcs bad code: 0
Port RX AXIS fifo overflow: 0
Port RX AXIS error: 0
Port RX invalid start: 0
Port RX flex fifo overflow: 0
Port RX flex fifo underflow: 0
Use a 10G DAC cable from VD100 port 0 to ConnectX-4 port 0.
With FEC Disabled in the VD100_10G_ether_dual and the ConnectX-4 set to auto FEC then both ends reported the link was up.
The following doesn't allow the link to come up.
- FEC disabled in MRMAC and auto in the ConnectX-4
- Enable FEC first on the MRMAC:
$ mrmac_ethernet/mrmac_configuration ctl_fec_mode=12 --reset - Enable FEC on the ConnectX-4:
$ sudo ethtool --set-fec ens1f0 encoding baser - Both ends report the link is down
The following allows the link to come up.
- FEC disabled in MRMAC and auto in the ConnectX-4
- Enable FEC first on the ConnectX-4:
$ sudo ethtool --set-fec ens1f0 encoding baser - Enable FEC on the MRMAC:
$ mrmac_ethernet/mrmac_configuration ctl_fec_mode=12 --reset - Both ends report the link is up. For the ConnectX-4:
$ ethtool --show-fec ens1f0 FEC parameters for ens1f0: Configured FEC encodings: BaseR Active FEC encoding: BaseR
In this state of FEC enabled started the switch test on the ConnectX-4:
$ sudo ibv_switch_test/ibv_raw_packet_switch_test -i rocep33s0f0 -n 1 -p 1-2 -r 5000
Bit rate on interface to injection switch = 10000 (Mbps)
Requested bit rate to be generated on each switch port under test = 5000.00 (Mbps)
Not limiting frame rate, as bit-rate on interface to injection switch doesn't exceed the total across all switch ports under test
Writing per-port counts to 20260419T221154_per_port_counts_linux.csv
Using interface rocep33s0f0 port 1
Test interval = 10 (secs)
Frame debug enabled = No
Press Ctrl-C to stop test at end of next test interval
22:12:04.542
Tx Test Rx Test Rx Unexpected Rx Flooded Rx Other missed frames tx rate (Hz) per port Mbps
8105565 0 0 0 2 8102253 810556.5 4999.51
Summary of missed frames : '.' none missed 'S' some missed 'A' all missed
Source Destination ports --->
port
12
1 A
2 A
Total test intervals with failures = 1
As expected, no frames were received by the switch test as nothing at the MRMAC end to return them.
And MRMAC statistics reported valid frames and FEC code words being received:
22:12:29.862 collection number 12
VD100_10G_ether_dual port 0 statistics (over 10.000 secs):
TX_CYCLE_COUNT : 6445236000
RX_CYCLE_COUNT : 6445153329
RX_FEC_CW_0 : 48826919
RX_TOTAL_PACKETS : 8105408
RX_TOTAL_GOOD_PACKETS : 8105408
RX_TOTAL_BYTES : 12336431344
RX_TOTAL_GOOD_BYTES : 12336430976
RX_PACKET_1519_1522_BYTES: 8105408
RX_UNICAST : 8105408
RX_VLAN : 8105408
VD100_10G_ether_dual port 1 statistics (over 10.000 secs):
TX_CYCLE_COUNT : 6445235877
RX_CYCLE_COUNT : 6445235877
It is repeatable that FEC has to be first enabled at the ConnectX-4 end for the link to come up. Not sure if some interaction with auto-negotiation being the default at the ConnectX-4 end.