ESXi

 View Only
Expand all | Collapse all

Guest Dropped Packets (Received) ?

  • 1.  Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 02:35 PM

    We have noticed that our new monitoring server (n-central) which checks all of our vmware esxi 4.x & 5.x servers, complains that the threshold for "Guest Dropped Packets (Received)" on our virtual machines has failed. Our vmware servers seem to be running OK, just a little concerned what this service status shows.

    Any ideas? Thanks, Oliver



  • 2.  RE: Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 02:42 PM

    Hello Oliver,

    As per the n-central , This service tracks the Number of data packets lost.

    Guest Dropped Packets (Received)The number of data packets lost during the reception of data by the virtual machine.

    To troubleshoot this and confirm not an ESXi problem

    check the performance charts on ESXi and see if there are packet drops for the VM and also you could run esxtop.

    Thanks,
    Avinash



  • 3.  RE: Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 03:07 PM

    Hello Avinash,

    Thanks for your feedback. Which performance chart should I look at (for the vm itself, right?) & which measurements? I am not aware of esxtop - how do I use this tool? Finally, is this a serve problem?



  • 4.  RE: Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 03:14 PM

    Hi,

    Well , we do not yet know if its a VM problem or server problem or the monitoring tool.?

    Highlight the VM -- and select the advanced charts, the below link gives out all the steps

    VMware vSphere 4 - ESX and vCenter Server

    esxtop will confirm if its a server problem or VM and this should be helpful

    http://www.vmworld.net/wp-content/uploads/2012/05/Esxtop_Troubleshooting_eng.pdf

    Thanks,

    Avinash



  • 5.  RE: Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 03:47 PM

    Hi,

    I manage the ESXi servers using vSphere. Using the advanced charts, I can see that there is packet loss for "Guest Dropped Packets (Received)" - that means it's not a monitoring tool issue.

    I have also used esxtop to monitor the esxi. Here the result:

    Yes, it seems that we are receiving packet loss for "Guest Dropped Packets (Received)". What could be causing this? Is this severe? All my ESXi servers, in different network topologies, seem to be experiencing this. Any ideas?

    Message was edited by: EVERSYS



  • 6.  RE: Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 04:32 PM

    correct so its not a monitoring tool problem , you may need to use the esxtop and check the logs as well


    >> Not sure how to use esxtop - I assume through putty? Unfortunately, the vmware host does not accept a putty connection


    A : yes you can SSH using a putty connection, The host may have ssh disabled and hence your not able to login using the ssh,


    you can enable it from vsphere client,

    host -- configuration --software --security profile -- properties -- ssh -- start



    Thanks,

    Avinash



  • 7.  RE: Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 04:34 PM

    ok. so if you see across all the hosts then I would Investigate the Physical switch and see if there are any port flapping.??? or do you see any packet drops on the switch level.

    If switch is clean then it could and problem of the Host Nic drivers.



  • 8.  RE: Guest Dropped Packets (Received) ?

    Posted Jan 31, 2014 05:26 PM

    I see this is a general problem across different networks with different topologies & different switches. Also vmware esxi 4.x & 5.x. Could it be the esxi network (vSwitch or Adapter) setup?



  • 9.  RE: Guest Dropped Packets (Received) ?

    Posted Feb 03, 2014 10:06 AM

    If you see it across multiple networks, you may need to Investigate your network setup- install a wireshark and try a packet capture



  • 10.  RE: Guest Dropped Packets (Received) ?

    Posted Feb 05, 2014 09:16 AM


  • 11.  RE: Guest Dropped Packets (Received) ?

    Posted Feb 05, 2014 09:50 AM

    I've seen this a few times myself, it always boiled down to a cosmetic display issue and never actually dropped packets or caused issues.

    I'd suggest updating to ESXi 5.1 U2 first before modifying anything within the guests, as it should be resolved there:

    https://www.vmware.com/support/vsphere5/doc/vsphere-esxi-51u2-release-notes.html

    Network packets drop is incorrectly reported in esxtop between two virtual machines on the same ESXi host and vSwitch

    When two virtual machines are configured with e1000 driver on the same vSwitch on a host, the network traffic between the two virtual machines might report significant packet drop in esxtop. This happens because during reporting there is no accounting for split packets when you enable TSO from guest.

    This issue is resolved in this release.



  • 12.  RE: Guest Dropped Packets (Received) ?

    Posted Feb 05, 2014 12:50 PM

    I have ESXi, 5.0.0, 504890 - how best do I update to 5.1 U2? I have no vCenter & just use vSphere to manage the ESXi. Are these any issues or warnings I need to be aware of before I update?

    Thanks



  • 13.  RE: Guest Dropped Packets (Received) ?

    Posted Feb 05, 2014 12:55 PM

    Hello,

    In ESXi 5.1 Most of the problems are with the vCenter and SSO and if you do not have a VC, No worries. I do not recall any major Issues and it does handle storage problems like APD and PDL better in ESXi 5.1 Update 2.

    Thanks



  • 14.  RE: Guest Dropped Packets (Received) ?

    Posted Feb 25, 2014 05:04 PM

    We too are seeing the exact same issue out of our N-Central product...... The N-Central is at the latest rev and so is ESX as of 2-25-14. vSphere and ESXTOP indicate that the host is dropping packets. This does appear to be a know issue form VMWare:

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2056468

    They say that is was resolved in ESXi 5.1 Update 2 but we are running ESX 5.5 - They also recommend changing NIC from the E1000 to something better but we have tried this as well as tweaking the buffer to no avail. This seem to me to be a REAL issue. The N-Central is set to alert you if you have more than 5 dropped packets and I have multiple VM's sitting at thousands of dropped packets and I'd be willing to bet if people would turn on that monitoring profile under Advanced Charts as detailed in post 3 above, they'd see similar results. Out of seven customers that we manage/monitor their 5.x ESX environment, ALL of them are showing excessive (thousands and thousands) of Guest Dropped Packets (Received) but oddly enough the Guest Dropped Packets (Transmitted) are typically fine?!?!?!

    I am taking this on this week and will share any findings back to this post, but has anyone else gotten across this bridge?

    Dave



  • 15.  RE: Guest Dropped Packets (Received) ?

    Posted Feb 25, 2014 05:51 PM

    They say that is was resolved in ESXi 5.1 Update 2 but we are running ESX 5.5

    Yes, the erroneously reported dropped frames issue was fixed in 5.1 U2 which was released on January 16th 2014. However, it's still not fixed for 5.5, which got it's latest updates on December 22nd 2013. It will probably be fixed for 5.5 with the next patches/Update1 which can be expected in the next few weeks (with the VSAN release).

    So unless you're actually seeing real packet loss or problems with your applications, you're most likely hit by this cosmetic, but not otherwise impacting bug and don't need to worry.

    As a matter of fact you just can't trust whatever the dropped packets counter reports until we get the patch for ESXi 5.5.



  • 16.  RE: Guest Dropped Packets (Received) ?

    Posted Jul 15, 2014 03:48 PM

    Just wanted to find out if this issue has now been fixed in N-Central or if you know of any workarounds? Thanks for the feedback



  • 17.  RE: Guest Dropped Packets (Received) ?

    Posted Jun 15, 2015 08:44 PM

    Hi ,  we are running 5.5 U2 [build 2718055] and the "issue" is still here.

    We are seeing it mostly on HP hosts (c7000), with Flex 10

    Are any you guys that are experiencing this issue using the Nexus 1Kv?



  • 18.  RE: Guest Dropped Packets (Received) ?

    Posted Sep 24, 2015 04:49 PM

    I am seeing guest VM 's performance chart has large number "Received packets dropped" on our environment now. Those VMs have different Windows/Linux OS, located on different ESXi 5.5, ESXi 6.0 servers.

    It doesn't look like a real network problem.


    1. esxtop on ESXi hosts shows 0% DRPRX for all vSwitches.

    2. for a newly created Centos 7 VM with almost non network traffic, the performance chart shows 31 packets dropped out of 31 received :

    but inside this VM:

    ifconfig shows no dropped packets:

    eno16777728: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

            inet 10.xx.xx.xxx  netmask 255.255.254.0  broadcast 10.4.117.255

            inet6 fe80::250:56ff:feb6:494d  prefixlen 64  scopeid 0x20<link>

            ether 00:50:56:b6:49:4d  txqueuelen 1000  (Ethernet)

            RX packets 11291  bytes 8235123 (7.8 MiB)

            RX errors 0  dropped 0  overruns 0  frame 0

            TX packets 2781  bytes 201323 (196.6 KiB)

            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

    there is no retransmission:

    netstat -s | grep retran

        0 segments retransmited

    It looks like this issue: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052917

    But since it's showing on our new ESXi 6 with very little CPU load, I doubt it's the same one.



  • 19.  RE: Guest Dropped Packets (Received) ?

    Posted Sep 24, 2015 06:12 PM

    I found a great blog : http://blog.igics.com/2014/11/esxi-network-troubleshooting.html

    This helped me to kind of understand my problem. In our case, seems it's not real packet drop.

    I tried to get the port data for one of my VM with almost no traffic :

    esxcli network port stats get -p 33554476 | grep -i recei | grep -i pack

       Packets received: 810

       Broadcast packets received: 767

       Multicast packets received: 6

       Unicast packets received: 37

       Receive packets dropped: 768

      esxcli network port stats get -p 33554476 | grep -i recei

       Packets received: 1606

       Broadcast packets received: 1538

       Multicast packets received: 9

       Unicast packets received: 59

       Receive packets dropped: 1534

    Now I see the "Packets received" include "Broadcast packets received", and "Broadcast packets received" is close to "Receive packets dropped" , so most of the dropped packets are broadcast packets.

    I tried to generate some real traffic, ssh into the VM and run tcpdump ( so it generate huge traffic on ssh ), then on the ESXi host check the port status:

    esxcli network port stats get -p 33554476 | grep -i recei | grep -i packe

       Packets received: 203001

       Broadcast packets received: 3594

       Multicast packets received: 24

       Unicast packets received: 199383

       Receive packets dropped: 3584

    esxcli network port stats get -p 33554476 | grep -i recei | grep -i packe

       Packets received: 238509

       Broadcast packets received: 3601

       Multicast packets received: 24

       Unicast packets received: 234884

       Receive packets dropped: 3591

    esxcli network port stats get -p 33554476 | grep -i recei | grep -i packe

       Packets received: 265753

       Broadcast packets received: 3606

       Multicast packets received: 24

       Unicast packets received: 262123

       Receive packets dropped: 3596

    The "Packets received" is getting bigger quickly, but dropped packets and broadcast packets stay the same pace and increase slowly.



  • 20.  RE: Guest Dropped Packets (Received) ?

    Posted Sep 24, 2015 06:33 PM

    did more troubleshooting, seems those dropped broadcast packets are "who has".

    I run

    esxcli network port stats get -p 33554476 | grep -i recei | grep -i packe;sleep 10 ; esxcli network port stats get -p 33554476 | grep -i recei

    and found in 10 seconds, "Broadcast packets received" increased 16, and at the same time I run tcpdump in the guest OS, captured 16 "who has" broadcast.

    Interesting how vmware define the received packets got "dropped", since tcpdump can see it.



  • 21.  RE: Guest Dropped Packets (Received) ?

    Posted Sep 25, 2015 12:30 AM

    Can you check Network Load Balancing in vSwitch.

    If it's configured for IP hash, try changing it to Virtual Port ID. It should fix packet drop issue.



  • 22.  RE: Guest Dropped Packets (Received) ?

    Posted Sep 28, 2015 02:23 PM

    checked the load balancing configuration and it's configured with "Route based on the originating virtual port ID" ( I guess this is the default setting? ), two adapters are configured as active and standby for this vswitch.

    For a not very active VM port:

    esxcli network port stats get -p 33554441

    Packet statistics for port 33554441

       Packets received: 512363

       Packets sent: 11019

       Bytes received: 44523546

       Bytes sent: 856446

       Broadcast packets received: 499440

       Broadcast packets sent: 10

       Multicast packets received: 3112

       Multicast packets sent: 0

       Unicast packets received: 9811

       Unicast packets sent: 11009

       Receive packets dropped: 486956

       Transmit packets dropped: 0

    It shows most broadcast packets are dropped, but in the VM level, when I run tcpdump, it actually shows those broadcast packets, so maybe this is a bug in ESXi ?



  • 23.  RE: Guest Dropped Packets (Received) ?

    Posted Sep 28, 2015 03:37 PM

    I changed a VM's NIC from e1000 to VMXNET3; changed its ethernet0.emuRxMode to 1, it didn't make any difference.

    When I booted up this VM, esxcli shows:

    esxcli network port stats get -p 33554460 | grep -i rece

       Packets received: 1755

       Bytes received: 147212

       Broadcast packets received: 1688

       Multicast packets received: 9

       Unicast packets received: 58

       Receive packets dropped: 1642

    Inside this VM, ethtool shows:

    ethtool -S eno16780032 | grep  -i "pkts rx"

    ucast pkts rx: 58

    mcast pkts rx: 9

    bcast pkts rx: 1690


    and there is no dropped packets ( ethtool run seconds later than the esxcli command ) .


    So the VM actually received all the broadcast/multicast packets, but somehow ESXi shows they were dropped.







  • 24.  RE: Guest Dropped Packets (Received) ?

    Posted Oct 07, 2015 09:55 PM

    Hi,

    What version of NIC you are using in your Guest?

    Have you tried upgrading the version to vmxnet?



  • 25.  RE: Guest Dropped Packets (Received) ?

    Posted Oct 20, 2015 08:20 AM

    Same expirience here.

    Cisco UCS Blades, Nexus 1000V, ESX 5.5 Build 2403361

    VMs are mixed. Windows and Linux with E1000 and VMXNET3.



  • 26.  RE: Guest Dropped Packets (Received) ?

    Posted Mar 11, 2016 04:07 PM

    For anybody who sees packet drop on esxcli but not esxtop, it's not real packet drop, but a ESXi bug:

    VMware KB: vCenter Server 5.1/5.5/6.0 performance charts report dropped network packets

    Seems it's fixed on 5.1 and 5.5 now, but still no fix on 6.0.