Vm loosing network packets

[wizzorama]

Hi,

I have a serious issue but can´t find any solution. I have a ESXi 5.1 running 3 VMs. The Host is a dual XEON machine with 24Gb Memory. Datastore with the VMS are running on SSD. additional Datastore for backup. Networkconnection 1NIC with 1GB connection and 1 NIC with 1MB connection. the 3 vms are 1 Windows 7 and 2 Win2k8 server. the windows 7 machien is not loosing any packets at all while the win 2k8 servers are loosing arround 15-30% packets.  I just upgraded the host yesterady to 5.1 was facing the problem after moving from one building to another one. No changes to the config of the server. Same switch same clients. I also have the issue that if I´m connected to the VM via the vsphere console I have "interrupts".

Any ideas.

Thx and greetz

Michael

[TomHowarth]

Can you confirm your second NIC's speed please.  also can you explain how your vSwitching is set up.

[wizzorama]

Hi,

yes 2. nic is on 100 MBit I just connected it yesterday to test.. this one was not connected before I tried to get a work arround with the 2. NIC but without success. The virtual switch is more or less default no changes there. I tried to change the connection speed from autonegotiate to 1000VD or 100 VD, but no changes. I also created a 2. vSwitch with the 100 MBit NIC and moved one VM to that switch for testing.. no difference. same sh... The host itslef is not loosing any packets.

What config parameters do you need?

Thx Michael

[zXi_Gamer]

Just to confirm if its a VMware driver in the Win2k8 problem or an infrastructure problem, perform the small experiment.

Have Win7 and Win2k8 in same vSwitch with NO uplinks. Give static IPs and do a ping from both the machines one by one. [I am pretty sure that there must not be any packet drops]. If you see any packet drops, just try to update the tools in Win2k8 if you are using vmxnet3 driver or e1000e driver.

Second step of the experiment. Power off Win7 and have Win2k8 in the vSwitch and ping from a machine which is not in the ESX server.

HTH,

zXi

[wizzorama]

Ok.. workingon it..now I lost the connection to the management ...grrr.. ping is still running but can´t connect to the management via vSphere.... send you the info as soon as possible..

Btw. do you know why there is "monitoring network "10.90.2.6-10.90.2.6" is in there when the nic has the IP 10.90.2.5?

Thx

Michael

[TomHowarth]

is your management network on the 100MB link? there are no 100MB NICs on the HCL for vSphere 5.1 AFAIK.

[wizzorama]

Hi,

no the management network is on the GB nic. Actually all VMs are on the GB link as well..the 100 MBIt connection was just used for testing yesterday if it could help solving th eissue..but not success on that. So still same problem.

What I tried as well is the thing with pinging internaly on a vswitch without uplink. Same resuilt the Win7 Vm pinging the 2k8 Vm is loosing packets :-(

I´m also experiencing connection issues using the VM console to the  Win2k 8 servers. Connecting to the Windows 7 Vm is working fine and smoothly.

All VM tools are updated.

Any suggesions... my users are almost killing me as they can´t work properly .-(

[wizzorama]

Strange effekt is that the Windows 7  VM pinging the Win2k8 server is loosing packets... while the other way arround the wind8 server pinging the win7 machine is not loosing any packets

[zXi_Gamer]

Chill my friend, now we are able to narrow down to the drivers in the GOS instead of the infrastructure itself. This is half way solution reached :smileyhappy:

BTW, can you mention the drivers used in the Win7 and Win2k8.

If you have time and if you have not read this, please do so:

http://www.yellow-bricks.com/2010/02/02/e1000-and-dropped-rx-packets/   [Coutesy Duncan Epping]

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010071

[wizzorama]

hi,

thx for the hints I went trough them already yesterday, unfortunately wiuth no success.

Just installed  a fresh Win2k8 server  running fine on that host so it seems to be a patch or something like that :-( Any ideas?

[wizzorama]

Just checked driver  versions on the fresh installed box and the productive sucking box, same version :-(

[TomHowarth]

Was the production Sucking box a P2V conversion or a new build?

[wizzorama]

Hi,

fresh install, and was working fine until 2 days ago. I just uninstalled all windows patches since 01.11.2012 ... no changes :-(

Ah in the meanhwile I conected the 2. NIC also to a GBIT Lan, and splittetd management and Vm network...no changes.. but actually it canßt be the vswitch anymore, as the fresh installed naked win2k8 is not loosing any packages ...

[TomHowarth]

what did those patches deal with?

[wizzorama]

MS security and normal patch stuff.. I didnt check one by one ìt was just a Idea out of my frustrated "no more ideas" situation :-(. No idea left anymore in my head... but the situation is really bad now.. still need to do all the work from yesterday and today on that productive machine, but impossible like the box is running :-(

[zXi_Gamer]

actually it canßt be the vswitch anymore, as the fresh installed naked win2k8 is not loosing any packages .

We cant work around naked though :smileyhappy: I tried out the experiment with Win7 and a Win2k8 and the ping was proper. I was using R2. These were not running any load. But when I started iozone on Win2k8, I was able to see 1 or 2 packet drops for every 10 mins or so.

When you say naked Win2k8, do you mean without VMware tools also?? Also, I had asked for the drivers present in the Windows machines. vmxnet3 or e1000e?

[wizzorama]

Hi,

naked means nothing installed apart from the OS and I installed the VMware tools, because without it really suxx :-).

The driver for both are the E1000. I also tried vmxnet3 without any changes. Alltough it keeps the information of old nix in the "background" so if I delete the nic and then later on recreate the same nix the settings for that nic are still there.

Do you think it makes sense to create a fresh vm assigning the old HDDs?

[zXi_Gamer]

Do you think it makes sense to create a fresh vm assigning the old HDDs?

I highly doubt it. The VM created so, will still be having the same effect. Power off the VM, remove all the nics in the VM. Power on the VM and find if you are able to see the nics listed. Connect only one nic and check for the ping effect.

There is a hotfix from MS for win2k8 losing packets.

[wizzorama]

Do you have the KB of the hotfix from MS? Or do you have a link for it?

[zXi_Gamer]

http://support.microsoft.com/kb/974909 plus more read in here

[wizzorama]

Hm this KB is stuck to MS product virtualisation product I don´t have any Hyper V installation nor roles.

I also have the problem that the affected VMs also hangs using the vSphere console. Which should not be related to any netwok ard dirver / issues. Does anybody know about something liek that?

[Gkeerthy]

can you check the teaming policy... vlan flapping issue...kind of thing...are you using LAG/ether channel... etc..

[abirhasan]

You can alos tracert your destinations IP and find out the path which takin more time to reach the destinations IP.

[wizzorama]

Hi @ll,

finaly its fixed. Still doesn't make sence to me but its working fine now. What I did delete all files from the vm directory except the vmdk files. Create a new virtual machine using the same name as the old one, and link the original virtual HDDs to this new machine. Power up and guess what. No problems anymore. The vm is working really smooth

again.

Thx for your help, and maybe this topic wille help somebody else as we'll. Sometimes the Solutions are not understandable.

Thx and greetz

Michael