Unable to pass jumbo frames

[call_percula]

ESXi 5.5.1474528

Dell M620 blades with 10g to Juniper EX 4500 stack to EQL 6510 arrays

I have a dedicated dvS for iSCSI with the MTU on the switch set to 9000. I have two kernel nics in two port groups with MTU 9000

If I vmkping -I vmk1 -d -s 8972 192.168.100.50 (SAN group IP, or if I use a controller IP) I get...

sendto() failed (Message too long)

If I back it down to 1472 I'm able to ping

I am able to send and receive jumbo packets from the switch to the SAN.

root@JUN-4500-01-IO> ping do-not-fragment size 8972 192.168.100.50

PING 192.168.100.50 (192.168.100.50): 8972 data bytes

8980 bytes from 192.168.100.50: icmp_seq=0 ttl=255 time=10.598 ms

8980 bytes from 192.168.100.50: icmp_seq=1 ttl=255 time=6.103 ms

Here are the esxcli outputs

~ # esxcli network nic list

Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description

------  -------------  ------  ----  -----  ------  -----------------  ----  --------------------------------------------------------------

vmnic0  0000:001:00.0  bnx2x   Up    10000  Full    a4:1f:72:36:26:e9  1500  Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet

vmnic1  0000:001:00.1  bnx2x   Up    10000  Full    a4:1f:72:36:26:ec  9000  Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet

vmnic2  0000:003:00.0  bnx2x   Up    10000  Full    a4:1f:72:36:26:ed  9000  Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet

vmnic3  0000:003:00.1  bnx2x   Up    10000  Full    a4:1f:72:36:26:f0  1500  Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet

~ # esxcli network ip interface list

[snip the other nics out]

vmk1

   Name: vmk1

   MAC Address: 00:50:56:6b:17:15

   Enabled: true

   Portset: DvsPortset-0

   Portgroup: N/A

   Netstack Instance: defaultTcpipStack

   VDS Name: iSCSI

   VDS UUID: 81 f4 18 50 5e 58 c8 6c-47 5a 4e 3a 0d 4e 35 cd

   VDS Port: 130

   VDS Connection: 1353293863

   MTU: 9000

   TSO MSS: 65535

   Port ID: 50331656

vmk2

   Name: vmk2

   MAC Address: 00:50:56:6d:b9:4c

   Enabled: true

   Portset: DvsPortset-0

   Portgroup: N/A

   Netstack Instance: defaultTcpipStack

   VDS Name: iSCSI

   VDS UUID: 81 f4 18 50 5e 58 c8 6c-47 5a 4e 3a 0d 4e 35 cd

   VDS Port: 258

   VDS Connection: 1383667712

   MTU: 9000

   TSO MSS: 65535

   Port ID: 50331657

~ # esxcli network vswitch dvs vmware list

[Snip other vdS out]

iSCSI

   Name: iSCSI

   VDS ID: 81 f4 18 50 5e 58 c8 6c-47 5a 4e 3a 0d 4e 35 cd

   Class: etherswitch

   Num Ports: 4352

   Used Ports: 7

   Configured Ports: 512

   MTU: 9000

   CDP Status: both

   Beacon Timeout: -1

   Uplinks: vmnic2, vmnic1

   VMware Branded: true

   DVPort:

         Client: vmnic1

         DVPortgroup ID: dvportgroup-762

         In Use: true

         Port ID: 128

         Client: vmnic2

         DVPortgroup ID: dvportgroup-762

         In Use: true

         Port ID: 129

         Client: vmk1

         DVPortgroup ID: dvportgroup-764

         In Use: true

         Port ID: 130

         Client: vmk2

         DVPortgroup ID: dvportgroup-765

         In Use: true

         Port ID: 258

[zXi_Gamer]

Can you provide the output of esxcfg-vswitch -l

[MKguy]

1. I suppose your iSCSI vmkernel NIC and the storage are on the same subnet? Please also provide the output of:

esxcli network ip interface ipv4 get

esxcli network ip route ipv4 list

2. Can you ping the host on it's own iSCSI vmkernel IP?

3. There also used to be some issues with interface selection of vmkping in the past if I remember correctly, try this instead of vmkping:

esxcli network diag ping --df --interface vmk1 --ipv4 --size 8972 --host 192.168.100.50

4. Also make sure you're running a recent NIC driver and NIC firmware on your host. I also have some hosts with BCM57810-based (HP-branded) 10Gbe NICs and jumbo frames running fine with the recent bnx2x driver version 1.78.27.v50.1 and firmware 7.8.79:

ethtool -i vmnic1

[grasshopper]

Great advice above.  Also just curious since it was not explicitly stated, have you tried pinging at 8000?  If that works, then ensure that the Juniper physical switch is configured at 9216 (same holds true on Cisco switches).

[call_percula]

I am not sure exactly what happen, but... I ended up rebooting the first host in the cluster and after it was able to pass jumbo packets.

Now on to the real fun of dealing with the issue that brought this to light in the first place...

Thanks all for the help!