Why vSAN health decrease after enable RDMA

[Bab]

I am using VSAN 8u3 ESA . Also my network adapter is Broadcom BCM57414 that support RDMA .  Now with enable RDMA my health status decrease from 100 to 50 and with disable it another increase . 

1 - Do I have to apply any specific configuration with esxcli for that network adapter ?

2 - Would you please say what commands need to apply on cisco physical switch ?

[Allan Solomon Mejia]

Yes, this behavior strongly suggests the RDMA/RoCE configuration isn't complete end-to-end. With vSAN 8.0 U3 ESA, simply enabling RDMA on the ESXi side isn't enough.

 

For the BCM57414, check these first:

 

• Verify the BCM57414 + ESXi 8.0 U3 + bnxtnet/bnxtroce driver + firmware combination against the Broadcom Compatibility Guide. vSAN 8.0 U3 has specific health checks for both RDMA NIC certification and the driver/firmware combination.
• Check the ESXi-side DCB/PFC state:
  • esxcli network nic dcb status get -n vmnicX
  • esxcli software vib list | grep -i bnx
  • esxcfg-module -g bnxtnet

• For RoCEv2, the physical network also needs to be configured correctly. In particular, check DCB/PFC on the Cisco switch ports carrying the vSAN RDMA traffic. RDMA NICs depend on proper switch-side PFC configuration.
• Make sure the DCB mode matches on both sides. For example, if ESXi is using IEEE DCB while the switch is using CEE, PFC can remain disabled and vSAN will report RDMA Configuration Health warnings.
• Also verify MTU/VLAN consistency and LLDP/DCBX configuration across the vSAN path.

I don't recommend applying random esxcli changes yet. First identify which Skyline Health checks changed from green to warning/red after RDMA was enabled. That will tell you whether this is an HCL/driver issue or an actual DCB/PFC configuration problem.

 

One additional caution with the BCM57414: an RDMA issue on hosts with more than 1 TB RAM when using bnxtnet versions 234.0.147.0 or earlier; the documented fix is in bnxtroce 235.1.140.0.

 

If you can post the failed vSAN Skyline Health checks plus the output of:

• esxcli network nic dcb status get -n vmnicX
• esxcli software vib list | grep -i bnx

 

We should be able to narrow this down pretty quickly.

[Bab]

So thanks . This is my output for one of my esxi hosts . All of them are same .  vmnic6,vmnic8 are using for vSAN also teaming is fix on Route base on original virtual port 

[root@host10:~] esxcli network nic dcb status get -n vmnic6
   Nic Name: vmnic6
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: false
         PG Traffic Classes: 3
         PFC Traffic Classes: 0
   PFC Enabled: false
   PFC Configuration: 0 0 0 0 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 50 50 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 255 255 255 255 255 255
         Priority Assignment Per TC: 0 0 0 0 0 1 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 0
         PFC Configuration: 0 0 0 0 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3
[root@hostv10:~] esxcli network nic dcb status get -n vmnic8
   Nic Name: vmnic8
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: false
         PG Traffic Classes: 3
         PFC Traffic Classes: 0
   PFC Enabled: false
   PFC Configuration: 0 0 0 0 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 50 50 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 255 255 255 255 255 255
         Priority Assignment Per TC: 0 0 0 0 0 1 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 0
         PFC Configuration: 0 0 0 0 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 0 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3

----------------------------

[root@host10:~] esxcli rdma device list
Name     Driver    State    MTU  Speed    Paired Uplink  Description
-------  --------  ------  ----  -------  -------------  -----------
vmrdma0  bnxtroce  Active  1024  25 Gbps  vmnic2         Broadcom RoCE device
vmrdma1  bnxtroce  Down    1024  0        vmnic3         Broadcom RoCE device
vmrdma2  bnxtroce  Active  1024  25 Gbps  vmnic0         Broadcom RoCE device
vmrdma3  bnxtroce  Active  1024  25 Gbps  vmnic1         Broadcom RoCE device
vmrdma4  bnxtroce  Active  4096  25 Gbps  vmnic6         Broadcom RoCE device
vmrdma5  bnxtroce  Down    1024  0        vmnic7         Broadcom RoCE device
vmrdma6  bnxtroce  Active  1024  25 Gbps  vmnic4         Broadcom RoCE device
vmrdma7  bnxtroce  Down    1024  0        vmnic5         Broadcom RoCE device
vmrdma8  bnxtroce  Active  4096  25 Gbps  vmnic8         Broadcom RoCE device
vmrdma9  bnxtroce  Down    1024  0        vmnic9         Broadcom RoCE device

---------------------------------------

[root@host10:~] esxcli software vib list | grep -i bnx
bnxtnet                        235.1.140.0-1OEM.800.1.0.20613240    BCM     VMwareCertified   2026-09-07    host
bnxtroce                       235.1.140.0-1OEM.800.1.0.20613240    BCM     VMwareCertified   2026-09-07    host

[Bab]

Also I have to say . After disable RDMA on the vSAN health score increase to 100%

[Allan Solomon Mejia]

Thanks for sharing the outputs. They point to the likely cause.

 

The RDMA devices paired with vmnic6 and vmnic8 are active, and both bnxtnet and bnxtroce are already running version 235.1.140.0, so the earlier driver issue doesn't appear to apply here.

 

The important part is the DCB output:

 

Priority Flow Control: false

PFC Traffic Classes: 0

PFC Enabled: false

PFC Configuration: 0 0 0 0 0 0 0 0

 

ESXi detects the RoCEv2 application on UDP port 4791 using priority 3, but PFC isn't operational on either vSAN uplink. That would explain why vSAN health drops when RDMA is enabled and returns to 100% when RDMA is disabled.

 

I'd check the Cisco switch configuration next and confirm that:

 

• QoS and DCB are enabled globally.
• PFC is enabled for priority 3 on every switch port carrying vSAN traffic.
• RoCEv2 traffic using UDP 4791 is mapped consistently to priority 3.
• DCBX is enabled and operating in IEEE mode on both the switch and ESXi.
• The configuration is consistent across every switch hop.
• VLAN and physical-switch MTU settings match the vSAN design.

 

The 4096 value shown by esxcli rdma device list is the RDMA path MTU and isn't, by itself, evidence of an MTU problem.

 

After correcting the switch-side DCB/PFC configuration, rerun:

 

esxcli network nic dcb status get -n vmnic6

esxcli network nic dcb status get -n vmnic8

 

You should no longer see PFC disabled with zero configured traffic classes. Also check the exact failed Skyline Health tests afterward.

 

The precise Cisco commands depend on the switch platform and NX-OS or IOS-XE version, so share the Cisco model and software version before applying a configuration.

[Bab]

My cisco switch is 93180fx3  with OS :  nxos64-cs.10.5.5.M.bin 

Also this is my configurations : 

interface Vlan1

interface port-channel1
description VPC-KEEP
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface port-channel70
description Connect-To-N5K-Core
switchport
switchport mode trunk
switchport trunk allowed vlan 800,1100,1101,1102,1103
spanning-tree port type normal
vpc 70

interface Ethernet1/1
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/2
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/3
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/4
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/5
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/6
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/7
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/8
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/9
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/10
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/11
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/12
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/13
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/14
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

interface Ethernet1/15
lldp dcbx version ieee
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type normal
flowcontrol receive on
flowcontrol send on
mtu 9216
service-policy type qos input ROCE-QOS
no shutdown

 

I am really confused what configurations need to apply on esxi side or cisco switch ?

[Brandon Stricker]

Hi Bab,

We had a similar issue with PFC, Broadcom NICs and Cisco switches. On your ports where your ESXi hosts are connected, try changing:

priority-flow-control mode on

to

priority-flow-control mode auto

You may need to reboot the hosts or at least disable/re-enable the ports. Rerun the vSAN health check afterward.

[Bab]

So Thanks . I have changed PFC to auto on physical switch now this is my output from vmnics that are assign for vSAN . This output is same for other esxi hosts : unfortunatelly health status did not change and still showing 50% after retest 

[root@hostv10:~] esxcli network nic dcb status get -n vmnic6
   Nic Name: vmnic6
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3
[root@hostv10:~] esxcli network nic dcb status get -n vmnic8
   Nic Name: vmnic8
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3

[Bab]

Also I have applied follow command just on one esxi hosts in the cluster 

[root@hostv10:~] esxcli system module parameters set -m bnxtnet -p disable_roce=0
[root@hostv10:~] esxcli system module parameters set -m bnxtroce -p disable_rocev2=0

1 - Do I have to change it to default parameters or have to apply on other esxi hosts in the vsan cluster ?

2- If need to change for default parameters . How can change it ?

[Bab]

Another I have to say currently this is my physical switch configuration : 

class-map type network-qos ROCE
  match qos-group 3
policy-map type network-qos ROCE-NETWORK-QOS
  class type network-qos ROCE
    pause pfc-cos 3

class-map type qos match-any ROCE
  match cos 3

policy-map type qos ROCE-NETWORK-QOS
  class ROCE
    set qos-group 3
    set cos 3
system qos
  service-policy type network-qos ROCE-NETWORK-QOS

no lldp tlv-select dcbxp
lldp tlv-select dcbxp egress-queuing

interface Ethernet1/1
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/2
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/3
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/4
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/5
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/6
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/7
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/8
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/9
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/10
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/11
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/12
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/13
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/14
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

interface Ethernet1/15
  lldp dcbx version ieee
  switchport
  switchport mode trunk
  priority-flow-control watch-dog-interval on
  spanning-tree port type normal
  mtu 9216
  service-policy type qos input ROCE-NETWORK-QOS
  no shutdown

Is my switch configurations correct ?

[Allan Solomon Mejia]

Thanks; knowing this is a Nexus 93180FX3 running NX-OS 10.5.5 helps.

 

Your switch ports already have:

 

lldp dcbx version ieee

priority-flow-control mode on

mtu 9216

service-policy type qos input ROCE-QOS

 

However, don't add more ESXi or switch commands yet. The key thing we need to verify is whether your ROCE-QOS policy and the system network-QoS policy actually map the vSAN/RDMA priority to a no-drop class.

 

Cisco documents that PFC requires both the interface PFC configuration and a network-QoS policy defining the applicable QoS group as no-drop. priority-flow-control mode on only enables PFC at the interface; it doesn't by itself create the required lossless traffic class.

 

This is especially relevant because your earlier ESXi output showed:

 

User Priority: 0x3

PFC Enabled: false

PFC Traffic Classes: 0

 

In other words, ESXi sees the RDMA application at priority 3, but it isn't currently reporting PFC as enabled.

 

Could you post the output of these commands from the Nexus?

 

show policy-map type qos

show policy-map type network-qos

show policy-map system type network-qos

show interface priority-flow-control

show lldp dcbx interface ethernet1/1

 

Also include:

 

show running-config | section "class-map"

show running-config | section "policy-map"

show running-config | section "system qos"

 

I'm particularly looking for whether CoS/priority 3 → QoS group → no-drop is configured consistently. DCBX PFC negotiation requires an exact match of the no-drop CoS between peers.

 

One other point: you currently have both:

 

priority-flow-control mode on

flowcontrol receive on

flowcontrol send on

 

The latter commands are link-level flow control (LLFC), while priority-flow-control is PFC. Cisco distinguishes these mechanisms, so don't assume that enabling ordinary flow control satisfies the RoCE PFC requirement.

 

So at this stage, I don't think the answer is simply "configure something else on ESXi." Let's first verify the ROCE-QOS classification, system network-QoS/no-drop class, PFC state, and DCBX advertisement. Once you post those outputs, we can see exactly where priority 3 stops being treated as lossless.

[Bab]

I don't know why some of my posts does not appears : 

After change PFC mode from "on" to "auto" on physical switch this is my vmnic output on esxi . As you can see PFC status changed to true . but did not any changes in environment 

[root@hostv13:~] esxcli network nic dcb status get -n vmnic6
   Nic Name: vmnic6
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3
[root@hostv13:~] esxcli network nic dcb status get -n vmnic8
   Nic Name: vmnic8
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3

------------------------------

Also this is switch output : 

show policy-map type qos

  Type qos policy-maps
  ====================

  policy-map type qos ROCE-QOS 
    class  ROCE
      set qos-group 3
      set cos 3
  policy-map type qos ROCE-NETWORK-QOS 
    class  ROCE
      set qos-group 3
      set cos 3
  policy-map type qos default-mpls-in-policy 
    class  c-dflt-mpls-qosgrp1
      set qos-group 1
    class  c-dflt-mpls-qosgrp2
      set qos-group 2
    class  c-dflt-mpls-qosgrp3
      set qos-group 3
    class  c-dflt-mpls-qosgrp4
      set qos-group 4
    class  c-dflt-mpls-qosgrp5
      set qos-group 5
    class  c-dflt-mpls-qosgrp6
      set qos-group 6
    class  c-dflt-mpls-qosgrp7
      set qos-group 7
    class  class-default
      set qos-group 0
  policy-map type qos default-ndb-out-policy 
    class  class-ndb-default
      police cir 0 bps conform transmit violate drop 
  policy-map type qos default-vxlan-in-policy 
    class  class-default
  policy-map type qos default-vxlan-in-tnl-dscp-policy 
    class  c-dflt-vxlan-tnl-dscp0
      set cos 0
    class  c-dflt-vxlan-tnl-dscp8
      set cos 1
    class  c-dflt-vxlan-tnl-dscp16
      set cos 2
    class  c-dflt-vxlan-tnl-dscp26
      set cos 3
    class  c-dflt-vxlan-tnl-dscp32
      set cos 4
    class  c-dflt-vxlan-tnl-dscp46
      set cos 5
    class  c-dflt-vxlan-tnl-dscp48
      set cos 6
    class  c-dflt-vxlan-tnl-dscp56
      set cos 7

---------------------------------------------------------------
#show policy-map type network-qos

  Type network-qos policy-maps
  ============================
  policy-map type network-qos Lossless-Policy
    class type network-qos c-nq-default
      match qos-group 0
      mtu 1500
  policy-map type network-qos ROCE-NETWORK-QOS
    class type network-qos ROCE
      match qos-group 3
      mtu 1500
      pause pfc-cos 3
  policy-map type network-qos default-nq-policy
    class type network-qos c-nq3
      match qos-group 3
      mtu 1500
    class type network-qos c-nq2
      match qos-group 2
      mtu 1500
    class type network-qos c-nq1
      match qos-group 1
      mtu 1500
    class type network-qos c-nq-default
      match qos-group 0
      mtu 1500
  policy-map type network-qos default-8q-nq-policy
    class type network-qos c-8q-nq7
      match qos-group 7
      mtu 1500
    class type network-qos c-8q-nq6
      match qos-group 6
      mtu 1500
    class type network-qos c-8q-nq5
      match qos-group 5
      mtu 1500
    class type network-qos c-8q-nq4
      match qos-group 4
      mtu 1500
    class type network-qos c-8q-nq3
      match qos-group 3
      mtu 1500
    class type network-qos c-8q-nq2
      match qos-group 2
      mtu 1500
    class type network-qos c-8q-nq1
      match qos-group 1
      mtu 1500
    class type network-qos c-8q-nq-default
      match qos-group 0
      mtu 1500

------------------------------------------------------------------------
# show policy-map system type network-qos

  Type network-qos policy-maps
  ============================
  policy-map type network-qos ROCE-NETWORK-QOS
    class type network-qos ROCE
      match qos-group 3
      mtu 1500
      pause pfc-cos 3

------------------------------------------------------------------------

#show interface priority-flow-control

slot  1
=======

============================================================
Port               Mode Oper(VL bmap)  RxPPP      TxPPP     
============================================================

Ethernet1/1        Auto On  (8)       2          0          
Ethernet1/2        Auto On  (8)       2          0          
Ethernet1/3        Auto On  (8)       2          0          
Ethernet1/4        Auto On  (8)       2          0          
Ethernet1/5        Auto On  (8)       2          0          
Ethernet1/6        Auto On  (8)       2          0          
Ethernet1/7        Auto On  (8)       0          0          
Ethernet1/8        Auto On  (8)       0          0          
Ethernet1/9        Auto On  (8)       2          0          
Ethernet1/10       Auto On  (8)       0          0          
Ethernet1/11       Auto On  (8)       0          0          
Ethernet1/12       Auto On  (8)       2          0          
Ethernet1/13       Auto On  (8)       0          0          
Ethernet1/14       Auto On  (8)       0          0          
Ethernet1/15       Auto On  (8)       2          0          
Ethernet1/16       Auto Off           0          0          
Ethernet1/17       Auto Off           0          0          
Ethernet1/18       Auto Off           0          0          
Ethernet1/19       Auto Off           0          0          
Ethernet1/20       Auto Off           0          0          
Ethernet1/21       Auto Off           0          0          
Ethernet1/22       Auto Off           0          0          
Ethernet1/23       Auto Off           0          0          
Ethernet1/24       Auto Off           0          0          
Ethernet1/25       Auto Off           0          0          
Ethernet1/26       Auto Off           0          0          
Ethernet1/27       Auto Off           0          0          
Ethernet1/28       Auto Off           0          0          
Ethernet1/29       Auto Off           0          0          
Ethernet1/30       Auto Off           0          0          
Ethernet1/31       Auto Off           0          0          
Ethernet1/32       Auto Off           0          0          
Ethernet1/33       Auto Off           0          0          
Ethernet1/34       Auto Off           0          0          
Ethernet1/35       Auto Off           0          0          
Ethernet1/36       Auto Off           0          0          
Ethernet1/37       Auto Off           0          0          
Ethernet1/38       Auto Off           0          0          
Ethernet1/39       Auto Off           0          0          
Ethernet1/40       Auto Off           0          0          
Ethernet1/41       Auto Off           0          0          
Ethernet1/42       Auto Off           0          0          
Ethernet1/43       Auto Off           0          0          
Ethernet1/44       Auto Off           0          0          
Ethernet1/45       Auto Off           0          0          
Ethernet1/46       Auto Off           0          0          
Ethernet1/47       Auto Off           0          0          
Ethernet1/48       Auto Off           0          0          
Ethernet1/49       Auto On  (8)       0          0          
Ethernet1/50       Auto On  (8)       0          0          
Ethernet1/51       Auto On  (8)       0          0          
Ethernet1/52       Auto On  (8)       0          0          
Ethernet1/53       Auto On  (8)       0          0          
Ethernet1/54       Auto On  (8)       0          0          

slot 27
=======
--------------------------------------------------------------------------

# show lldp dcbx interface ethernet1/1
Protocol Versions:-  Configured: IEEE 802.1, Detected: IEEE 802.1.

Local TLV Information:
Type/
Subtype    Version    En/Will/Adv Config
011/000     000        Y/N/Y      0808
010/000     000        Y/N/Y      0000010000 64000000 00000000 02020000 00000000
009/000     000        Y/N/Y      0200010000 64000000 00000000 02020000 00000000

Peer's TLV Information:
Type/      Max/Oper
Subtype    Version    En/Will/Err Config
009/000     000/000    Y/Y/N      8300010000 64000000 00000000 02020000 00000000
011/000     000/000    Y/Y/N      8108
012/000     000/000    Y/N/N      006312b7
-------------------------------------------------------------

#show running-config | section "class-map"
class-map type network-qos ROCE
  match qos-group 3

class-map type qos match-any ROCE
  match cos 3

policy-map type network-qos Lossless-Policy
  class type network-qos c-nq-default
policy-map type network-qos ROCE-NETWORK-QOS
  class type network-qos ROCE
    pause pfc-cos 3

policy-map type qos ROCE-NETWORK-QOS
  class ROCE
    set qos-group 3
    set cos 3

-------------------------------------------------------
# show running-config | section "system qos"
system qos
  service-policy type network-qos ROCE-NETWORK-QOS

[Allan Solomon Mejia]

Thanks for the updated outputs. The situation now looks much better.

 

After changing the switch ports from priority-flow-control mode on to priority-flow-control mode auto

 

ESXi now reports:

 

• Priority Flow Control: true
• PFC Enabled: true
• PFC Configuration: 0 0 0 1 0 0 0 0
• RoCEv2 UDP port 4791 mapped to user priority 3

 

This confirms that PFC for priority 3 is now being advertised and negotiated successfully through IEEE DCBX. Previously, ESXi detected the RoCE application mapping but showed PFC as disabled, which was the likely reason the vSAN RDMA health checks failed.

 

The switch configuration also appears consistent:

 

• The QoS policy classifies RoCE traffic into QoS group 3 and CoS 3.
• The network QoS policy applies pause pfc-cos 3.
• The network QoS policy is applied globally under system qos.
• IEEE DCBX is configured on the host-facing ports.
• The interfaces use MTU 9216.

 

The zero PFC frame counters are not necessarily a problem. PFC pause frames are generated only when congestion reaches the configured threshold, so they may remain at zero in a lightly loaded environment.

 

Now, retest the exact vSAN Skyline Health checks that previously failed. Also verify both vSAN paths with:

 

esxcli network nic dcb status get -n vmnic6

esxcli network nic dcb status get -n vmnic8

esxcli rdma device list

 

If both vmnic6 and vmnic8 continue to show PFC enabled for priority 3, but vSAN health still remains at 50%, please share the names and details of the remaining failed health checks. At that point, the issue may be a different RDMA validation item rather than PFC negotiation.

 

One configuration item is worth reviewing: the host-facing interfaces have both PFC and ordinary Ethernet link-level flow control configured:

 

flowcontrol receive on

flowcontrol send on

 

Those commands are separate from PFC. For a RoCE design using priority-based pause on CoS 3, confirm with Cisco whether standard link-level flow control should be removed from these interfaces. Do not change it during production without validating the complete network design first.

[Bab]

This is output and same for other hosts on vmnic6,vmnic8 : I have to say exactly after disable RDMA from vSAN service health increase to 100% . 

[root@hostv14:~] esxcli network nic dcb status get -n vmnic6
   Nic Name: vmnic6
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3
[root@hostv14:~] esxcli network nic dcb status get -n vmnic8
   Nic Name: vmnic8
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3

[root@hostv14:~] esxcli rdma device list
Name     Driver    State    MTU  Speed    Paired Uplink  Description
-------  --------  ------  ----  -------  -------------  -----------
vmrdma0  bnxtroce  Active  1024  25 Gbps  vmnic2         Broadcom RoCE device
vmrdma1  bnxtroce  Down    1024  0        vmnic3         Broadcom RoCE device
vmrdma2  bnxtroce  Active  1024  25 Gbps  vmnic0         Broadcom RoCE device
vmrdma3  bnxtroce  Active  1024  25 Gbps  vmnic1         Broadcom RoCE device
vmrdma4  bnxtroce  Active  4096  25 Gbps  vmnic6         Broadcom RoCE device
vmrdma5  bnxtroce  Down    1024  0        vmnic7         Broadcom RoCE device
vmrdma6  bnxtroce  Active  1024  25 Gbps  vmnic4         Broadcom RoCE device
vmrdma7  bnxtroce  Down    1024  0        vmnic5         Broadcom RoCE device
vmrdma8  bnxtroce  Active  4096  25 Gbps  vmnic8         Broadcom RoCE device
vmrdma9  bnxtroce  Down    1024  0        vmnic9         Broadcom RoCE device

This is a screenshot from skyline health . 

What is the solution for this level ? really cannot find out the issue raise from physical switch or esxi configuration . 

Also follow screenshot is showing skyline health after disable RDMA as you can see is 100% health 

Does it mean have to remove follow options to configure switch for vSAN RDMA ? 

flowcontrol receive on

flowcontrol send on

[Allan Solomon Mejia]

The ESXi output shows DCB is enabled in IEEE mode and PFC is enabled for priority 3 on vmnic6 and vmnic8. However, this confirms only the host-side state; it doesn't prove that the physical network is configured correctly end-to-end. It requires consistent RDMA/PFC configuration across the NICs, switch access ports, and inter-switch links.

 

Don't remove flowcontrol receive on or flowcontrol send on based only on these results. Those are switch-vendor commands, and it specifically recommends validating the switch configuration with the switch vendor or networking team.

 

Verify the following:

 

• The switches use IEEE DCB mode, matching the ESXi NICs. A mismatch between IEEE and CEE can prevent correct PFC negotiation.
• PFC is enabled for priority 3 across every vSAN path, including access ports and switch-to-switch links.
• The configured NIC model, driver, and firmware combination is certified for vSAN over RDMA.
• VLAN, MTU, and vSAN VMkernel connectivity remain consistent on every host.

 

The cluster-partition alerts appearing only when RDMA is enabled strongly indicate loss or disruption of vSAN network communication. This check detects multiple vSAN network partitions but does not identify the root cause.

 

The available output is insufficient to determine whether the fault is specifically in the switch or ESXi configuration. Compare the detailed RDMA Configuration Health result for each vmnic and have the switch vendor validate the complete DCB/PFC policy.

[Bab]

This is exactly my esxi info :

you said verifying :

1 - 

• The switches use IEEE DCB mode, matching the ESXi NICs. A mismatch between IEEE and CEE can prevent correct PFC negotiation.

[root@hostv10:~] esxcli network nic list
Name    PCI Device    Driver   Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  -------  ------------  -----------  -----  ------  -----------------  ----  -----------
vmnic0  0000:23:00.0  bnxtnet  Up            Up           25000  Full    14:23:f3:28:71:30  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic1  0000:23:00.1  bnxtnet  Up            Up           25000  Full    14:23:f3:28:71:31  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic2  0000:11:00.0  bnxtnet  Up            Up           25000  Full    14:23:f3:28:6e:90  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic3  0000:11:00.1  bnxtnet  Up            Down             0  Half    14:23:f3:28:6e:91  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic4  0000:89:00.0  bnxtnet  Up            Up           25000  Full    14:23:f3:28:72:20  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic5  0000:89:00.1  bnxtnet  Up            Down             0  Half    14:23:f3:28:72:21  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic6  0000:37:00.0  bnxtnet  Up            Up           25000  Full    40:5b:7f:b8:24:14  9000  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic7  0000:37:00.1  bnxtnet  Up            Down             0  Half    40:5b:7f:b8:24:15  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic8  0000:af:00.0  bnxtnet  Up            Up           25000  Full    40:5b:7f:b8:10:ac  9000  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic9  0000:af:00.1  bnxtnet  Up            Down             0  Half    40:5b:7f:b8:10:ad  1500  Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller

-----------

[root@hostv10:~] esxcli network nic get -n vmnic6
   Advertised Auto Negotiation: true
   Advertised Link Modes: 1000BaseSR/Full, 10000BaseSR/Full, 25000BaseSR/Full, Auto
   Auto Negotiation: true
   Backing DPUId: N/A
   Cable Type: FIBRE
   Current Message Level: 0
   Driver Info:
         Bus Info: 0000:37:00:0
         Driver: bnxtnet
         Firmware Version: 237.1.141.0 /pkg 237.1.148.0
         Version: 235.1.140.0
   Link Detected: true
   Link Status: Up
   Name: vmnic6
   PHYAddress: 1
   Pause Autonegotiate: true
   Pause RX: false
   Pause TX: false
   Supported Ports: FIBRE
   Supports Auto Negotiation: true
   Supports Pause: true
   Supports Wakeon: true
   Transceiver: internal
   Virtual Address: 00:50:56:5e:a3:39
   Wakeon: MagicPacket(tm)
[root@hostv10:~] esxcli network nic get -n vmnic8
   Advertised Auto Negotiation: true
   Advertised Link Modes: 1000BaseSR/Full, 10000BaseSR/Full, 25000BaseSR/Full, Auto
   Auto Negotiation: true
   Backing DPUId: N/A
   Cable Type: FIBRE
   Current Message Level: 0
   Driver Info:
         Bus Info: 0000:af:00:0
         Driver: bnxtnet
         Firmware Version: 237.1.141.0 /pkg 237.1.148.0
         Version: 235.1.140.0
   Link Detected: true
   Link Status: Up
   Name: vmnic8
   PHYAddress: 1
   Pause Autonegotiate: true
   Pause RX: false
   Pause TX: false
   Supported Ports: FIBRE
   Supports Auto Negotiation: true
   Supports Pause: true
   Supports Wakeon: true
   Transceiver: internal
   Virtual Address: 00:50:56:52:35:2f
   Wakeon: MagicPacket(tm)

---------------

[root@hostv10:~] esxcli network nic dcb status get -n vmnic6
   Nic Name: vmnic6
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3
[root@hostv10:~] esxcli network nic dcb status get -n vmnic8
   Nic Name: vmnic8
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 3
         PFC Traffic Classes: 1
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 3
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 100 0 0 0 0 0 0 0
         RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
         TSA Assignment Table Per TC: 2 2 0 0 0 0 0 0
         Priority Assignment Per TC: 0 0 0 1 0 0 0 0
         Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
         Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
         Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
   IEEE PFC Configuration:
         Number Of Traffic Classes: 1
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 2 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps:
         App Type: UDP port number
         Protocol ID: 0x12b7
         User Priority: 0x3

--------------------

[root@hostv10:~] esxcfg-module -g bnxtnet
bnxtnet enabled = 1 options = 'disable_roce=0'

------------------

[root@hostv10:~] esxcli system module parameters list -m bnxtnet
Name                          Type           Value  Description
----------------------------  -------------  -----  -----------
DRSS                          array of int          Number of RSS Queues in Default Queue RSS pool. When Multi-RSS is active, This won't take effect[Default: 4, Max: 16]
RSS                           array of int          Number of RSS queues in a netqueue RSS pool or multi-rss engine. [Default: -1(auto-configured), Max: 16]
async_cmd_cmpl_timeout        uint                  For debug purposes, currently set to 10000 msec's, user can increase it. [Default: 10000, Min: 10000]
debug                         uint                  Debug msglevel: Default is 0 for Release builds
disable_dcb                   bool                  Disable the DCB support. 0: enable DCB support, 1: disable DCB support. [Default: 0]
disable_fwdmp                 bool                  For debug purposes, disable firmware dump feature when set to value of 1. [Default: 0]
disable_geneve_filter         bool                  For debug purposes, disable Geneve filter support feature when set to value of 1. [Default: 0]
disable_geneve_oam_support    bool                  For debug purposes, disable Geneve OAM frame support feature when set to value of 1. [Default: 1]
disable_q_feat_pair           bool                  For debug purposes, disable queue pairing feature when set to value of 1. [Default: 0]
disable_q_feat_preempt        bool                  For debug purposes, disable FEAT_PREEMPTIBLE when set to value of 1. [Default: 0]
disable_roce                  bool           0      Disable the RoCE support. 0: Enable RoCE support, 1: Disable RoCE support. [Default: 0]
disable_shared_rings          bool                  Disable sharing of Tx and Rx rings support. 0: Enable sharing, 1: Disable sharing. [Default: 0]
disable_tpa                   array of uint         Disable the TPA(LRO) feature. 0: enable TPA, 1..N: disable TPA. [Default: 0]
disable_vxlan_filter          bool                  For debug purposes, disable VXLAN filter support feature when set to value of 1. [Default: 0]
enable_default_queue_filters  int                   Allow filters on the default queue. -1: auto, 0: disallow, 1: allow. [Default: -1, which enables the feature when NPAR mode and/or VFs are enabled, and disables if otherwise]
enable_dma_mappers            bool                  Enable DMA mappers support. 0: disable, 1: enable. [Default: 0]
enable_dr_asserts             bool                  For debug purposes, set to 1 to enable driver assert on failure paths, set to 0 to disable driver asserts. [Default: 0]
enable_geneve_ofld            bool                  Enable Geneve TSO/CSO offload support. 0: disable Geneve offload, 1: enable Geneve offload. [Default: 1]
enable_host_dcbd              bool                  Enable host DCBX agent. 0: disable host DCBX agent, 1: enable host DCBX agent. [Default: 0]
enable_multi_rss              bool                  Enable Multi-RSS support. 0: disable, 1: enable. [Default: 1]
enable_r_writes               bool                  For debug purposes, set to 1 to enable r writes, set to 0 to disable r writes. [Default: 0]
enable_vxlan_ofld             bool                  Enable VXLAN TSO/CSO offload support. 0: disable, 1: enable. [Default: 1]
force_hwq                     array of int          Max number of hardware queues: -1: auto-configured, 1: single queue, 2..N: enable this many hardware queues. [Default: -1]
int_mode                      uint                  Force interrupt mode. 0: MSIX; other modes deprecated. [Default: 0]
max_vfs                       array of int          Number of Virtual Functions: 0: disable, N: enable this many VFs. [Default: 0]
multi_rx_filters              int                   Define the number of RX filters per NetQueue: -1: use the default number of RX filters, 0,1: disable use of multiple RX filters, so single filter per queue, 2..N: force the number of RX filters to use for a NetQueue. [Default: -1]
num_rss_pools                 uint                  Max Number of RSS pools. [Default: 4]. This will take effect when enable_multi_rss is set
psod_on_tx_tmo                bool                  For debug purposes, set to 1 to force PSOD on tx timeout, set to 0 to disable PSOD on tx timeout. [Default: 0]
roce_pri                      uint                  RoCE app priority in firmwawre based dcb mode. Valid range is 0-7. [Default: 0x3]
rsvd_netq                     array of int          Number of hardware queues reserved as regular (non-RSS) RX queues.[Default: -1(auto-configured)]

1-1 : According to above output is showing roce is enabled on firmware  but enable_host_dcbd is showing is disabled  . Do I have to change enable_host_dcbd  value to 1  ?

you said for verifying  :

• PFC is enabled for priority 3 across every vSAN path, including access ports and switch-to-switch links.

2 - What does it mean ? Does it your mean have to apply PFC enable for priority 3 on physical switch globally and also on each switch port for vmnic6 , vmnic8 ?

• The configured NIC model, driver, and firmware combination is certified for vSAN over RDMA.

3 - As you can see I sent all vmnic drivers and firmware for vmnic6 , vmic8 all of them are using same version and it is certified with vSAN . Pleae confirm it 

• VLAN, MTU, and vSAN VMkernel connectivity remain consistent on every host.

for all nodes in the cluster for vmnic6,vmnic8 MTU = 9000 and same VLAN for vSAN vmkernel . 

I am really confused what is the issue . Would yo please say what is the next to solve it ?

[Allan Solomon Mejia]

The output confirms that both vmnic6 and vmnic8 are already operating in IEEE DCB mode with PFC enabled for priority 3:

 

Mode: 3 - IEEE Mode

PFC Enabled: true

PFC Configuration: 0 0 0 1 0 0 0 0

DCB App User Priority: 0x3

 

Do not change enable_host_dcbd to 1 solely because it currently shows 0. I cannot find verified Broadcom documentation requiring that module parameter for this configuration. I recommend validating Broadcom NIC parameters with the hardware vendor and configuring the NIC and switch together.

 

PFC priority 3 must be configured end-to-end on every physical link carrying the vSAN RDMA VLAN. This includes the switch ports connected to vmnic6 and vmnic8, and any switch-to-switch links in that path. It does not mean enabling PFC globally on unrelated switch ports. It requires a lossless Layer 2 path using IEEE DCB and PFC priority 3.

 

Next, verify every host and both vSAN uplinks:

 

esxcli vsan network list

esxcli rdma device list

esxcli network nic dcb status get -n vmnic6

esxcli network nic dcb status get -n vmnic8

 

Both RDMA devices should be active, and each associated uplink should report IEEE mode with PFC priority 3. Then retest the vSAN RDMA health check. If it still fails, provide the exact Skyline Health error because the current output does not show a missing-PFC condition.