How to simulate network latency between two ESXI hosts?

[shavi72]

Hi,

We have a scenario where we want to simulate a WAN between two co-located ESXi 5.x hosts that are part of a HA+DRS cluster. The idea is to generate variable network latencies like 10 ms to 1000 ms between the hosts. I have tried WANEM tool as follows:

1. Setup WANEM and configured a latency of 500 ms and packet drop %age of 10 in WANEM Web UI
2. Added routes to both ESXi hosts using esxcfg-route command and specified WANEM IP as the gateway.

esxcfg-route -a <management n/w ip of host2> 255.255.255.255 <wanem ip>

esxcfg-route -a <management n/w ip of host1> 255.255.255.255 <wanem ip>

   3. Tried a ping between the two ESXI hosts --> Result: No additional n/w latency was observed between the two hosts

Queries:

1. Am I missing something in the above setup?
2. Is there a better way/tool of achieving the above? Our requirement is to first verify this with standard vSwitch and later also try with vDS

Thanks, Vivek

[tomtom901]

Are your hosts in the same subnet? If so, this will be handeled at L2 (stuff like arp etc) instead of the static route you've configured. If the WanEM supports it, you could choose to configure the hosts in different subnets and let your WanEM route between the subnets.

[shavi72]

Hey, Thanks for the response!

My ESXi hosts are in the same subnet.

WANEM supports configurations where two hosts are in different broadcast domains - http://openmaniak.com/wanem_network.php

In our case it is not possible to configure the hosts in different subnets.

I was able to use WANEM for introducing n/w latency between two Windows hosts which are in same subnet; after adding the WANEM IP as the gateway using route command

From windows host 1 run

"route add <ip of windows host 2> mask 255.255.255.255 <wanem ip>"

However with ESXi, the latency does not happen. Is it something on ESX networkng stack that prevents the WANEM to be used as the gateway? Should I tweak some vSwitch settings?

Any further inputs would be very helpful and appreciated

Thanks,

Vivek

[tomtom901]

Does a traceroute provided the wanted output? You can do this via SSH.

[shavi72]

After configuring latency of 500ms in WANEM UI and adding the route on host 1; traceroute shows the WANEM IP as the gateway and briefly shows the latency as 500 ms; but later falls back to default (< 1ms)

~ # traceroute  <IP of ESXi host2>

traceroute to <IP of ESXi host2> (<IP of ESXi host2>), 30 hops max, 40 byte packets

1  <IP of WANEM> (<IP of WANEM>)  1002.266 ms  501.323 ms  501.873 ms

2  <IP of ESXi host2> (<IP of ESXi host2>)  501.895 ms  0.989 ms  1.013 ms

Additional observation:- when I try ping second time for host1 to host2 - it shows below error

~ # ping <IP of ESXi host2>

PING <IP of ESXi host2> (<IP of ESXi host2>): 56 data bytes

sendto() failed (Invalid argument)

Seems like adding the route causes some problem with the ping itself. Am I messing up with ESXi routes? Is there a better/different way to achieve this?

Thanks,

Vivek

[admin]

To determine whether the poor performance is due to network latency:

1. Test the maximum bandwidth from the virtual machine with the Iperf tool. This tool is available from http://sourceforge.net/projects/iperf/.
   
   Note: VMware does not endorse or recommend any particular third-party utility.
   
   
2. 1. While using Iperf, change the TCP windows size to 64 K. Performance also depends also on this value. To change the TCP windows size:
      
      
      1. On the server side, enter this command:
         
         iperf -s

         
      2. On the client side, enter this command:
         
         iperf.exe -c sqlsed -P 1 -i 1 -p 5001 -w 64K -f m -t 10 900M

For more information, see http://openmaniak.com/iperf.php.

1. Run Iperf with a machine outside the ESX/ESXi host. Compare the results with what you expect you should have, depending on your physical environment.
2. Run Iperf with another machine outside the ESX/ESXi host on the same VLAN on the same physical switch. If the performance is good, and the issue can only be reproduced with a machine at another geographical location, then the issue is related to your network environment.
3. Run Iperf between 2 VMs on the same ESX server/portgroup/vswitch. If the result is good, you can exclude a CPU, memory or storage issue.

If you identify a bottleneck on the network:

1. Work through the steps in Troubleshooting network performance issues (1004087).
2. If you are using iSCSI storage and jumbo frames, ensure that everything is properly configured. For more information, see:
   
   
   • iSCSI and Jumbo Frames configuration on ESX/ESXi (1007654)
   • Enabling IOAT and Jumbo frames (1003712)
   • Configuring Jumbo Frames on a vNetwork Distributed Switch (1010711)
   • Enabling Jumbo Frames for VMkernel ports in a virtual distributed switch (1038827)
     
     
3. If you are using Network I/O Control, ensure that the shares and limits are properly configured for your traffic. For more information, see Network I/O Resource Management in vSphere 4.1 with vDS (1022585).
4. Ensure that traffic shaping is correctly configured. For more information, see Traffic Shaping Policy in the ESX/ESXi Configuration Guide.

[shavi72]

The above seems to be coming straight out of some VMware KB article. Though the info is useful, unfortunately it does not answer my query

Any help much appreciated?

Thank you,

Vivek