network throughput

[bluesherpa]

Does this sound like an issue with ESXi and, if so, any ideas on how to fix it?

It seems like I am getting slow file transfer rates to my ESXi VMs - an average of 13MB/s. I'm trying to figure out how to get better transfer rates.

My setup:

File server with Ubuntu, 6gb ram, 2.4ghz dual core, an Areca Raid Controller (1230 w/1gb cache) and 8 Seagate 2TB drives configured in one Raid 5 volume, and two ethernet bonded (802.3ab) ports.

Dell PowerConnect 2724 switch - 24 gigabit ports with 802.3ab support.

ESXi server with an i7 920 processor, 12gb of ddr3 (1600), and 3 Western Digital 1TB black drives (10k rpm), Intel Pro/1000 PT Dual Port Server Adapter.

My tests:

baseline - how quickly a 700mb file will transfer from the source, the file server, to the destination. I know... IOmeter, etc. is better for measuring performance, but please bear with me, since I don't have IOmeter installed nor do I know how to use it yet.

test 1 - destination is a desktop connected to the switch at gigabit, full-duplex. The file transfers in 10 seconds. The desktop has a network utility showing that reports an average transfer rate of 65MB/s

test 2 - destination is a VM running on the desktop (VMware Fusion). The file transfers in 15 seconds.

test 3 - destination is a VM running on the ESXi server. The file transfers in 55 seconds.

test 4 - copy (not move) the file from one folder to another inside the ESXi VM. The file copies in 10 seconds.

Additional Notes:

- ESXi has vmnet0, both ethernet ports are added, teaming is set to Hash based on IP

- CPU utilization on the file server never exceeds 10% on any of the tests

- I ran all of the tests with both Windows 98 and Linux (Ubuntu) VMs

- All of the VMs have the VMware client installed

Conclussions:

It seems like ESXi is somehow misconfigured, because a file transfer rate of 13MB/s seems slow given that the file server can feed the same file to other destination 5 times faster. I'm trying to get ESXi to use 802.3ab bonding for greater throughput, but it seems like I should get more than 13MB/s even with a single gigabit port.

What can I do achieve greater / faster network throughput? ie. Make the file copy transfer more quickly from the file server to one of the ESXi VMs? Do I need to configure something differently? Do I need to install quad gigabit or 10g? Do I have to go with iSCSI? Or is there a configuration item that just needs to be tweaked to achieve better performance?

[LarryBlanco2]

Quick ?.. you mention 802.3ab do you mean 802.3ad?

802.3ab just specifies the gigabit standard. 802.3ad specifies link agrregation.

The 2724 only supports XOR based balancing which will not work very well with the ESX boxes. Shame on Dell for putting that in the tech specs for this switch.

If you want to start using 802.3ad you'll need to move up a notch to the 2824 switch.

That switch support static lacp with the ability to do IP Hash or MAC hash or a combination of both.

http://support.euro.dell.com/support/edocs/network/pc28xx/en/ug/pdf/ug_en.pdf

Once you upgrade your switch be sure to enable jumbo frames on the switch as well as your esxi box.

Hope that helps you and gets you a new switch.

Larry B.

[bluesherpa]

Hi Larry,

Yes - 802.3ad - thanks for the clarification.

I was able to get the server to transfer at 55MB/s. The trick was to upgrade to using vmxnet3.

I would still like to upgrade the switch to 2824 like you've recommended. Of course, I automatically wonder if there is even a better option in the same price range. I checked and it looks like I can pick one up for under $200. Any other favorites that you think would work better than the 2724 and in the $200 price range?

Thanks!

[LarryBlanco2]

As far as switches goes, I do like the Dell's. I think they provide really good features for their relatively low prices. So my pick would be to stick with those.

Thanks and take care.

LB.

[bluesherpa]

Great =) Thanks for the help, Larry =)