Hi all,
Regarding NSX-V VTEP Offload to NIC, as far as I see, there are few articles and information over the internet so any advices on this feature would be appreciated!!
I managed to find how to set up and what default setting is as below.
However I have no idea with configuration criteria of RSS/TSO/VXLAN Offload… Do you have any information on them?
In the first place, does someone leverage enabling these NIC offload features with NSX-V? If so, any considerations to introduce them?
VMware® NSX for vSphere (NSX) Network Virtualization Design Guide says to recommend to use VXLAN
Offload feature for Edge Cluster but I wonder if ESXs on Prod VM HA cluster does not requires to enable them?
It is recommended to use hosts with higher core densities to support the required
Edge VM sizing (2 vCPU to 4 vCPU). Higher clock speed should also be considered
for VXLAN offload while NIC choices made based on the need for higher bandwidth.
Host Specification Guideline for Edge Cluster
- Highly recommended to use NIC that supports VXLAN
TSO offload and RSS support for desired line rate performance.
[root@esxi:~] esxcli system module parameters list -m vmnic10
Name Type Value Description
-------------------- ------------- ----- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DRSS array of uint Number of RSS Queues associated with default queue: 0 (Disabled), 2 (min), 4 (max). Default: Disabled
RSS array of uint Number of RSS Queues: 0 (Auto) or 1 to 4 (fixed queue number). Default: Auto
debug_mask uint Enabled debug mask (default: 0)
dropless_fc uint Pause on exhausted host ring. 0: Disable (Default) 1: Disable
enable_dcbx uint Enable DCBX. 0: Disable 1: Enable (Default)
enable_fwdump uint Enable/Disable the firmware dump file. Set to 1 to enable firmware dump, Set to 0 to disable firmware dump [Default]
enable_lro uint Enable/Disable the TPA (LRO) featureSet to 0 to disable TPA, Set to 1 to enable TPA [Default]
enable_vxlan_filters uint Enable/Disable the VXLAN Rx filters.Set to 0 to disable VXLAN Rx filtersSet to 1 to enable VXLAN Rx filtersDefault: 0
fairness_threshold uint When set to 1 will enable the fairness threshold; 0 by Default
hw_vlan uint Enable/Disable VLAN removal/insertion by hardware. 0: Disabled and 1: Enabled. Default: 1
max_vfs array of uint Number of VFs to be enabled for each pci function. Default:0, Valid values: 0:Disable, 1 to 64
mtu uint MTU when the driver is loaded. Range: 0-9000. (default: 1500)
offload_flags uint Offload flags: 1 (cso) 2(tso) 4(vxlan offload) 8(Geneve offload). Default: 15
rss_engine_nr array of uint Number of RSS Engines: 0 (Disabled) or 1 to 4 (fixed NUmber of RSS Engines). Default: 1
rx_filters int Define the number of RX filters per NetQueue-1: use the default number of RX filters based on availability:0: Disable use of multiple RX filters; 1,2,3,...: Will force the number of RX filters to use for NetQueue; Default: -1
rxqueue_nr uint Number of Rx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
rxring_bd_nr uint Number of RX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
tx_to_delay uint Time interval (in seconds) to wait for before considering a Tx queue stuck and triggering a tx timeout. Default:5, Disable:0 (use netdev tx timeout), Minimum:5
txqueue_nr uint Number of Tx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
txring_bd_nr uint Number of TX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
[root@esxi:~] esxcfg-module -i qfle3
esxcfg-module module information
input file: /usr/lib/vmware/vmkmod/qfle3
License: ThirdParty:QLogic_Proprietary
Version: 1.0.86.0-1OEM.670.0.0.8169922
Name-space:
Required name-spaces:
com.vmware.vmkapi@v2_5_0_0
Parameters:
DRSS: array of uint
Number of RSS Queues associated with default queue: 0 (Disabled), 2 (min), 4 (max). Default: Disabled
RSS: array of uint
Number of RSS Queues: 0 (Auto) or 1 to 4 (fixed queue number). Default: Auto
debug_mask: uint
Enabled debug mask (default: 0)
dropless_fc: uint
Pause on exhausted host ring. 0: Disable (Default) 1: Disable
enable_dcbx: uint
Enable DCBX. 0: Disable 1: Enable (Default)
enable_fwdump: uint
Enable/Disable the firmware dump file. Set to 1 to enable firmware dump, Set to 0 to disable firmware dump [Default]
enable_lro: uint
Enable/Disable the TPA (LRO) featureSet to 0 to disable TPA, Set to 1 to enable TPA [Default]
enable_vxlan_filters: uint
Enable/Disable the VXLAN Rx filters.Set to 0 to disable VXLAN Rx filtersSet to 1 to enable VXLAN Rx filtersDefault: 0
fairness_threshold: uint
When set to 1 will enable the fairness threshold; 0 by Default
hw_vlan: uint
Enable/Disable VLAN removal/insertion by hardware. 0: Disabled and 1: Enabled. Default: 1
max_vfs: array of uint
Number of VFs to be enabled for each pci function. Default:0, Valid values: 0:Disable, 1 to 64
mtu: uint
MTU when the driver is loaded. Range: 0-9000. (default: 1500)
offload_flags: uint
Offload flags: 1 (cso) 2(tso) 4(vxlan offload) 8(Geneve offload). Default: 15
rss_engine_nr: array of uint
Number of RSS Engines: 0 (Disabled) or 1 to 4 (fixed NUmber of RSS Engines). Default: 1
rx_filters: int
Define the number of RX filters per NetQueue-1: use the default number of RX filters based on availability:0: Disable use of multiple RX filters; 1,2,3,...: Will force the number of RX filters to use for NetQueue; Default: -1
rxqueue_nr: uint
Number of Rx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
rxring_bd_nr: uint
Number of RX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
tx_to_delay: uint
Time interval (in seconds) to wait for before considering a Tx queue stuck and triggering a tx timeout. Default:5, Disable:0 (use netdev tx timeout), Minimum:5
txqueue_nr: uint
Number of Tx Queues: 0 (Auto) or 1 to 10 (fixed queue number). Default: Auto
txring_bd_nr: uint
Number of TX BD Buffers: 4096(min) 16384(max), must be power of two. Default: 4096
[root@esxi:~] localcli network nic software list
NIC IPv4 CSO IPv4 TSO Scatter Gather Offset Based Offload VXLAN Encap Geneve Offload IPv6 TSO IPv6 TSO Ext IPv6 CSO IPv6 CSO Ext High DMA Scatter Gather MP VLAN Tagging VLAN Untagging
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vmnic10 off off off off off off off off off off off off off off