ESXi

 View Only
Expand all | Collapse all

vmkping -d -s 9000 sendto () (message too long)

  • 1.  vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 18, 2014 08:26 PM

    I am getting an error when doing a jumbo packet ping to my iscsi storage

    everything has been set to 9000mtu end to end

    vmkping -s 9000 storageip

    responds fine

    but when i put the -d flag in I get

    sendto () failed (message too long)

    any idea?

    I tried vmkping -I vmkernel_nic as well but same problem



  • 2.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 18, 2014 08:42 PM

    You need to subtract some header data bytes when you use the ping command. Use 8972 instead. See http://rickardnobel.se/troubleshoot-jumbo-frames-with-vmkping/ for details

    André



  • 3.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 18, 2014 08:55 PM

    still doesnt work

    1 packets transmitted, 0 packets received, 100% packet loss

    /vmfs/volumes/504d1fa5-1c455cec-0519-0024e85c1297/pa-vc # vmkping -I vmk5 -d -s 8972 10.60.1.106

    PING 10.60.1.106 (10.60.1.106): 8972 data bytes

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    sendto() failed (Message too long)



  • 4.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 18, 2014 09:00 PM

    In this case you need to check which part in the chain isn't configured properly.

    port groups -> vSwitch -> physical switch -> storage system

    André



  • 5.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 19, 2014 08:59 AM

    If you are using 5.5, then you can use  pktcap-uw to identify where the drop is happening



  • 6.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 19, 2014 10:00 AM

    Packet tracing won't work, and the physical infrastructure can't be blamed (at least yet) because the packets are dropped by the host's own networking stack before they actually reach the wire.

    I've seen that sometimes a host needs an actual reboot after enabling jumbo frames.If it still doesn't work after a reboot, try this:


    1. Please post the output of the following commands on the local shell:

    # esxcli network nic list

    # esxcli network ip interface list

    # esxcli network ip interface ipv4 get

    # esxcli network ip route ipv4 list

    # esxcli network vswitch dvs vmware list

    # esxcli network vswitch standard list

    # esxcli network nic list | awk {'print $1'} | grep vmnic | while read nic; do esxcli network nic get -n "$nic" ; done

    # vmware -v

    2. Try some smaller payload size values like 8000, 4000, 2000.

    3. Can you vmkping the host on it's own localhost and vmkernel interface IP? E.g.:

    # vmkping -I vmk5 -d -s 8972 [IP of vmk5]

    # vmkping -d -s 8972 127.0.0.1

    4. Try the tests with esxcli network diag ping utility as well, e.g.:

    esxcli network diag ping --df --interface vmk5 --ipv4 --size 8972 --host 10.60.1.106


    5. Also make sure your NIC drivers and firmware are up to date.



  • 7.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 26, 2014 12:53 AM

    vmkping -d -s 8972 127.0.0.1 responds fine


    but


    vmkping -I vmk5 -d -s 8972 [IP of vmk5]fails



    esxcli network diag ping --df --interface vmk5 --ipv4 --size 8972 --host 10.60.1.106


    fails



  • 8.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 26, 2014 05:48 PM

    by default the MTU size is 1500 bytes and there is Jumbo Frame enabled yet, that's why when run command Ping -d -s 9000 giving you error sendto() failed (Message too long).



  • 9.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 26, 2014 06:42 PM

    Please don't mind me asking this again, but you didn't respond to my previous reply. Can you confirm Jumbo frames (MTU 9000) is configured for the VMkernel port group as well as for the vSwitch?

    André



  • 10.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Mar 26, 2014 08:25 PM

    hi

    yes jumbo frames are enabled on the host vmk ports and the vswitch

    DVS Name Num Ports Used Ports Configured Ports MTU Uplinks

    dvS-UCS-iSCSI 256 7 256 9000 vmnic9,vmnic8

    vmnic8 0000:10:00.00 enic Up 10000Mbps Full 00:25:b5:11:0a:cf 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

    vmnic9 0000:11:00.00 enic Up 10000Mbps Full 00:25:b5:11:0b:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

    vmk4 101 IPv4 10.60.1.219 255.255.0.0 10.60.255.255 00:50:56:6c:aa:d6 9000 65535 true STATIC

    vmk5 132 IPv4 10.60.1.220 255.255.0.0 10.60.255.255 00:50:56:6c:6c:34 9000 65535 true STATIC



  • 11.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Aug 23, 2014 06:42 PM

    Hi guys,

    I got this thread searching in google for problem in question/subject. Straight to the output:

    ~ # vmware -v

    VMware ESXi 5.5.0 build-1623387

    ~ # esxcli network ip interface list

    vmk0

       Name: vmk0

       MAC Address: 00:50:56:63:22:4c

       Enabled: true

       Portset: DvsPortset-0

       Portgroup: N/A

       Netstack Instance: defaultTcpipStack

       VDS Name: ASW1

       VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

       VDS Port: 2229

       VDS Connection: 25747893

       MTU: 1500

       TSO MSS: 65535

       Port ID: 33554488

    vmk1

       Name: vmk1

       MAC Address: 00:50:56:62:41:cc

       Enabled: true

       Portset: DvsPortset-0

       Portgroup: N/A

       Netstack Instance: defaultTcpipStack

       VDS Name: ASW1

       VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

       VDS Port: 2230

       VDS Connection: 418560376

       MTU: 9000

       TSO MSS: 65535

       Port ID: 33554504

    vmk2

       Name: vmk2

       MAC Address: 00:50:56:68:ad:97

       Enabled: true

       Portset: DvsPortset-0

       Portgroup: N/A

       Netstack Instance: defaultTcpipStack

       VDS Name: ASW1

       VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

       VDS Port: 2233

       VDS Connection: 442667658

       MTU: 9000

       TSO MSS: 65535

       Port ID: 33554505

    ~ # esxcli network ip interface ipv4 get

    Name  IPv4 Address   IPv4 Netmask   IPv4 Broadcast   Address Type  DHCP DNS

    ----  -------------  -------------  ---------------  ------------  --------

    vmk0  192.168.100.4  255.255.255.0  192.168.100.255  STATIC           false

    vmk1  192.168.100.6  255.255.255.0  192.168.100.255  STATIC           false

    vmk2  192.168.100.7  255.255.255.0  192.168.100.255  STATIC           false

    ~ # esxcli network ip route ipv4 list

    Network        Netmask        Gateway        Interface  Source

    -------------  -------------  -------------  ---------  ------

    default        0.0.0.0        192.168.100.1  vmk0       MANUAL

    192.168.100.0  255.255.255.0  0.0.0.0        vmk0       MANUAL

    ~ # esxcli network vswitch dvs vmware list

    ASW1

       Name: ASW1

       VDS ID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

       Class: cisco_nexus_1000v

       Num Ports: 1024

       Used Ports: 45

       Configured Ports: 1024

       MTU: 9000

       CDP Status: listen

       Beacon Timeout: -1

       Uplinks: vmnic2, vmnic3

       VMware Branded: false

       DVPort:

             Client:

             DVPortgroup ID: dvportgroup-268

             In Use: false

             Port ID: 96

       

    [cut]

    ~ # esxcli network vswitch standard list

    ~ # esxcli network nic list | awk {'print $1'} | grep vmnic | while read nic; do esxcli network nic get -n "$nic" ; done

       Advertised Auto Negotiation: true

       Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

       Auto Negotiation: true

       Cable Type: Twisted Pair

       Current Message Level: 7

       Driver Info:

             Bus Info: 0000:05:00.0

             Driver: igb

             Firmware Version: 1.59, 0x800008f8

             Version: 5.0.5.1

       Link Detected: false

       Link Status: Down

       Name: vmnic0

       PHYAddress: 1

       Pause Autonegotiate: true

       Pause RX: true

       Pause TX: true

       Supported Ports: TP

       Supports Auto Negotiation: true

       Supports Pause: true

       Supports Wakeon: true

       Transceiver: internal

       Wakeon: MagicPacket(tm)

       Advertised Auto Negotiation: true

       Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

       Auto Negotiation: true

       Cable Type: Twisted Pair

       Current Message Level: 7

       Driver Info:

             Bus Info: 0000:05:00.1

             Driver: igb

             Firmware Version: 1.59, 0x800008f8

             Version: 5.0.5.1

       Link Detected: false

       Link Status: Down

       Name: vmnic1

       PHYAddress: 1

       Pause Autonegotiate: true

       Pause RX: true

       Pause TX: true

       Supported Ports: TP

       Supports Auto Negotiation: true

       Supports Pause: true

       Supports Wakeon: true

       Transceiver: internal

       Wakeon: Disabled

       Advertised Auto Negotiation: true

       Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

       Auto Negotiation: true

       Cable Type: Twisted Pair

       Current Message Level: 7

       Driver Info:

             Bus Info: 0000:05:00.2

             Driver: igb

             Firmware Version: 1.59, 0x800008f8

             Version: 5.0.5.1

       Link Detected: true

       Link Status: Up

       Name: vmnic2

       PHYAddress: 1

       Pause Autonegotiate: true

       Pause RX: true

       Pause TX: true

       Supported Ports: TP

       Supports Auto Negotiation: true

       Supports Pause: true

       Supports Wakeon: true

       Transceiver: internal

       Wakeon: Disabled

       Advertised Auto Negotiation: true

       Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

       Auto Negotiation: true

       Cable Type: Twisted Pair

       Current Message Level: 7

       Driver Info:

             Bus Info: 0000:05:00.3

             Driver: igb

             Firmware Version: 1.59, 0x800008f8

             Version: 5.0.5.1

       Link Detected: true

       Link Status: Up

       Name: vmnic3

       PHYAddress: 1

       Pause Autonegotiate: true

       Pause RX: true

       Pause TX: true

       Supported Ports: TP

       Supports Auto Negotiation: true

       Supports Pause: true

       Supports Wakeon: true

       Transceiver: internal

       ~ # esxcfg-vswitch -l

    DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks

    ASW1             1024        45          1024              9000    vmnic2,vmnic3

    ~ # vmkping -4 -d 192.168.100.2 -I vmk1 -s 1600

    PING 192.168.100.2 (192.168.100.2): 1600 data bytes

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    --- 192.168.100.2 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ # vmkping -4 -d 192.168.100.2 -I vmk2 -s 1600

    PING 192.168.100.2 (192.168.100.2): 1600 data bytes

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    --- 192.168.100.2 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ #  vmkping 192.168.100.6 -s 7600 -d -I vmk1

    PING 192.168.100.6 (192.168.100.6): 7600 data bytes

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    --- 192.168.100.6 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ #

    ~ #  vmkping 192.168.100.6 -s 7600 -d -I vmk2

    PING 192.168.100.6 (192.168.100.6): 7600 data bytes

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    sendto() failed (Message too long)

    --- 192.168.100.6 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ #

    It looks like the MTU is properly configured everywhere, but for some reason vmkernel is not able to use it :|

    My setup is as follow:

    [iSCSI-TARGET]------[NEXUS3064]-------[ [NEXUS1000v] [SUPERMICRO] ]

    here is some output from 1000V:

    port-profile type ethernet system-port-channel-uplink

      vmware port-group

      switchport trunk allowed vlan 50,100

      switchport mode trunk

      mtu 9000

      channel-group auto mode active

      no shutdown

      system vlan 100

      state enabled

    !

    port-profile system-port-channel-uplink

    type: Ethernet

    description:

    status: enabled

    max-ports: 32

    min-ports: 1

    inherit:

    config attributes:

      switchport trunk allowed vlan 50,100

      switchport mode trunk

      mtu 9000

      channel-group auto mode active

      no shutdown

    evaluated config attributes:

      switchport trunk allowed vlan 50,100

      switchport mode trunk

      mtu 9000

      channel-group auto mode active

      no shutdown

    assigned interfaces:

      port-channel2

      Ethernet3/3

      Ethernet3/4

    port-group: system-port-channel-uplink

    system vlans: 100

    capability l3control: no

    capability iscsi-multipath: no

    capability vxlan: no

    capability l3-vservice: no

    port-profile role: none

    port-binding: static

    !

    ASW1# show int ethernet 3/3

    Ethernet3/3 is up

      Hardware: Ethernet, address: 0050.5653.1a8e (bia 0050.5653.1a8e)

      Port-Profile is system-port-channel-uplink

      MTU 9000 bytes

      Encapsulation ARPA

      Port mode is trunk

      full-duplex, 1000 Mb/s

      5 minute input rate 268104 bits/second, 195 packets/second

      5 minute output rate 310099816 bits/second, 1472 packets/second

      Rx

        209833845 Input Packets 211147151 Unicast Packets

        22289 Multicast Packets 5553 Broadcast Packets

        90787232006 Bytes

      Tx

        209377590 Output Packets 207623415 Unicast Packets

        58439 Multicast Packets 20585 Broadcast Packets 65636 Flood Packets

        530214333212 Bytes

        1 Input Packet Drops 0 Output Packet Drops

    ASW1# show int ethernet 3/4

    Ethernet3/4 is up

      Hardware: Ethernet, address: 0050.5653.1a8f (bia 0050.5653.1a8f)

      Port-Profile is system-port-channel-uplink

      MTU 9000 bytes

      Encapsulation ARPA

      Port mode is trunk

      full-duplex, 1000 Mb/s

      5 minute input rate 3574688 bits/second, 5062 packets/second

      5 minute output rate 307317560 bits/second, 1260 packets/second

      Rx

        206359322 Input Packets 207121928 Unicast Packets

        291654 Multicast Packets 1324 Broadcast Packets

        48693934052 Bytes

      Tx

        221563934 Output Packets 219762972 Unicast Packets

        50337 Multicast Packets 36333 Broadcast Packets 73025 Flood Packets

        457044944110 Bytes

        117 Input Packet Drops 0 Output Packet Drops

    ASW1# show int po2

    port-channel2 is up

      Hardware: Port-Channel, address: 0050.5653.1a8e (bia 0050.5653.1a8e)

      Description: === LINK TO N3064 ===

      MTU 9000 bytes, BW 2000000 Kbit, DLY 10 usec,

         reliability 255/255, txload 1/255, rxload 1/255

      Encapsulation ARPA

      Port mode is trunk

      full-duplex, 1000 Mb/s

      Beacon is turned off

      Input flow-control is off, output flow-control is off

      Switchport monitor is off

      Members in this channel: Eth3/3, Eth3/4

      Last clearing of "show interface" counters never

      300 seconds input rate 3842696 bits/sec, 5257 packets/sec

      300 seconds output rate 614845280 bits/sec, 2723 packets/sec

      Rx

        118063294 unicast packets  37583 multicast packets  1363 broadcast packets

        118102240 input packets  85807288965 bytes

        80 input packet drops

      Tx

        36317177 unicast packets  7133 multicast packets  17137 broadcast packets

        36341448 output packets  839871823663 bytes

        22599 flood packets

        0 output packet drops

      1 interface resets

    I have configured VMGuest machine (Linux) with MTU9000 and managed to ping the physical Nexus3k switch with bigger packets:

    root@debian2:~# ping 192.168.100.2 -s 8000 -M do

    PING 192.168.100.2 (192.168.100.2) 8000(8028) bytes of data.

    8008 bytes from 192.168.100.2: icmp_req=1 ttl=255 time=1.24 ms

    8008 bytes from 192.168.100.2: icmp_req=2 ttl=255 time=1.09 ms

    8008 bytes from 192.168.100.2: icmp_req=3 ttl=255 time=0.965 ms

    So it looks like VMguest interfaces are OK, 1000v is OK, Nexus3064 is OK, but for some reason vmk1 & vmk2 interfaces cannot use mtu bigger than 1500 ..

    I am really confused here :smileysad: Any help would be more than welcome!


    Sincerely,

    D.



  • 12.  RE: vmkping -d -s 9000 sendto () (message too long)

    Posted Aug 23, 2014 09:02 PM

    Hi chaps, I managed to find what was the problem... It was due to all interfaces being part of the same broadcast domain (vmk0,1 and 2). For some reason kernel decides to use vmk0 interface (perhaps owing to routing table?! but the remote host shouldn't be even routed as it is part of the same bcast domain..) which has mtu 1500... So I split iSCSI ifaces (vmk1 & 2) to another subnet and that fixed it.. Still strange though, why does ESXi kernel uses vmk0 when I have explicitly pointed vmk1 & 2.. ? Anyway! Thanks for your time and hope this post will help someone else :-)