ESXi

 View Only
Expand all | Collapse all

ESXi 5 iSCSI speed query

  • 1.  ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 01:50 AM

    G'day Guys,

    I'm progressing well with my vmware (vSphere 5) deployment i have been playing around with for months and am at the stage where I am serioulsy considering 'going live' and start replacing production servers. I ran up a Windows 2008R2 server to be used as a Fileserver as a test and went to copy about 50gb of data to it to check the speeds across the network and was a bit alarmed to see that I was getting, at best, around 12mb across the LAN from a physical box to the VM.

    Physically, I have a Dell R710, 48gb Ram, 2x 6 core Xeons, 3x 4 port 1gb nics (12x 1gb nics all up) and all setup and working well to conenct back to a MD3200i SAN which the VM's reside on - all setup and working as per vmware specifications. I have 8 paths per LUN, MRU setup (as Dell MD series SANs don't work well with RR?? is that still the case??). I've setup Jumbo frames on all switches boh physical (Dell PowerConnects) and all the vSwitches are set to MTU 9000. Each VM Network uses 5 nics and on the physical switch I have set these ports as a LAG (no LACP and set to Trunk) and in the vSwitch I have IP Hash set.

    The Virtual envronment runs through a Dell Power Connect 6248 which has a 4 port LAG (set to Trunk) across to a Dell PowerConnect 5448 to which all the physical servers are connected.

    So, with that info...have I missed/screwed something in the setup that would result in such poor network speeds that anyone can see?

    Any advice or suggestions are greatly appreciated.



  • 2.  RE: ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 01:53 AM

    As a further example...I just copied an Exchange 2010 iso to it....and it was sitting around 11mb transfer speed....when I looked in the Resource tab for that host I could see that of the vmnics that are setup for the VM Network (vmnics 1,2,7,8 and 9) the only stats I could see were for vmnics 2 and 9.....1,7 and 8 were all 0's.

    So I went and checked the Paths for that LUN and I can see 8, 4 active - 4 standby. 1 set as Active (i/o) due to using MRU. Should I be seeing data coming through on all 4 active connections??



  • 3.  RE: ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 04:34 AM

    Hi,

    It's fair enough to expect higher, but:

    Eight paths is far more than is useful. I would reduce this. You may be hurting yourself in the management in the way that doesn't help you.

    You shouldn't expect simultaneous throughput on all NICs. Running multiple paths will allow you to balance your LUNs across different paths - it won't give you 4Gb of communication to a single LUN.

    Unless I missed it, you didn't mention how your SAN was configured. If it's full of SATA disks I'm surprised you performed this well.

    Usual advise, firmware on the SAN and patches to ESXi are worth looking at. There's one in particular for iSCSI under ESXi 5.0.



  • 4.  RE: ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 05:24 AM

    G'day Josh, cheers for the response.

    I should have mentioned that the MD3200i has 12x 600gb 15k SAS drives split into 2x 6Disk RAID 10 arrays - we have pretty modest data needs so I went for speed and redundancy. The array also has 2 controllers - hence the 8 paths per LUN. Each cotnroller has 4 ports for a total of 8. Under 'Manage Paths' each LUN shows 4 active (only one says (i/o) ) and 4 standby which equates to all 4 ports on Controller 0 are active and all teh Controller 1 ports are on Standby.

    Both the MD3200i and ESXi are fully patched and updated.



  • 5.  RE: ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 08:12 AM

    newtovms wrote:

    I've setup Jumbo frames on all switches boh physical (Dell PowerConnects) and all the vSwitches are set to MTU 9000.

    Have you also configured the VMkernel interface used by iSCSI for larger frame size?

    newtovms wrote:

    Each VM Network uses 5 nics and on the physical switch I have set these ports as a LAG (no LACP and set to Trunk) and in the vSwitch I have IP Hash set.

    The Virtual envronment runs through a Dell Power Connect 6248 which has a 4 port LAG (set to Trunk) across to a Dell PowerConnect 5448 to which all the physical servers are connected.

    Because of the way the IP hash load balancing work on ESXi vSwitches you will gain very little on the iSCSI network and there are several risks for incorrect configuration which could cause many issues. You could consider switching to Port ID based load balancing.



  • 6.  RE: ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 01:48 PM

    Thanks Ric,

    Yes, I have enabled Jumbo frames on the switches and in ESXi. In ESXi Have quadruple checked that each vSwitch has the MTU set to 9000. Going into the properties of vSwitch1 and vSwitch2 I have checked that all teh iSCSI vmKernel Ports are set to a MTU of 9000 and confirmed again that vSwitch3 which has the vmnetwork is also set to MTU 9000



  • 7.  RE: ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 02:05 PM

    Then it is most likely not a jumbo frames issues. If you like you can do a final verification end-to-end something like this.

    Any thoughts on the configuration of IP Hash vs Port id?



  • 8.  RE: ESXi 5 iSCSI speed query

    Posted Nov 24, 2011 04:31 PM

    As I am remoting in from home to teh servers right now (as its 2am here) I'll make those changes tomorrow an see how we go. At the moment I've decided to split up the iSCSI vmkernels to 1 vmkernel per vswitch after eading a few articles with peopel having similar issues. So instead of iSCSI1 and iSCSI2 being in vSwitch 1 I now have a vswitch5 with ISCSIA and a vswitch6 with iSCSIB etc I've also switched from MRU to RR in the Paths management....



  • 9.  RE: ESXi 5 iSCSI speed query

    Posted Nov 25, 2011 05:54 PM

    Are your nics's binded to your iSCSI software HBA ? I'm not sure if its the same in ESX5i but I knew in 4 if you didn't have the additional nics binded to vmhba33 or whatever your iSCSI software HBA is called they wouldn't load balance properly.

    This will bind the vmk0 VMkernel port to the iSCSI Software Adapter vmhba33. We then proceed

    to bind all of the other vmk# to the same vmhba.

    esxcli swiscsi nic add –n vmk1 –d vmhba33

    esxcli swiscsi nic add –n vmk2 –d vmhba33

    esxcli swiscsi nic add –n vmk3 –d vmhba33

    esxcli swiscsi nic add –n vmk4 –d vmhba33

    esxcli swiscsi nic add –n vmk5 –d vmhba33

    To verify that all of the vmk# are bound properly to the vmhba run the following command:

    esxcli swiscsi nic list –d vmhba33

    This will list all of the information for VMkernel ports that are assigned to the iSCSI Software

    Adapter.

    Not sure if this needs to be done anymore but worth taking a look into.  This is from one of Dell's Equallogic series setup which is all iSCSI connected SANs



  • 10.  RE: ESXi 5 iSCSI speed query

    Posted Nov 26, 2011 08:01 AM

    JESX35 wrote:

    Are your nics's binded to your iSCSI software HBA ? I'm not sure if its the same in ESX5i but I knew in 4 if you didn't have the additional nics binded to vmhba33 or whatever your iSCSI software HBA is called they wouldn't load balance properly

    In ESXi 5 there is a graphical interface for doing this. You still have to make sure each vmk interface has a single VMNIC as active and the other as unused (could be done through different vSwitches as above) but the binding of the Software iSCSI component to the VMkernel vmk is done in the GUI.

    However, I do not think this should be done with the IP hash load balancing policy used here.

    Newtovms, could you post how your virtual networking is setup. Perhaps a screen shot of this, with the Vmkernel and the vSwitches?



  • 11.  RE: ESXi 5 iSCSI speed query

    Posted Nov 28, 2011 05:33 AM

    Back after the weekend. I'll grab some screen shots in the next hour or two and pop them up.

    Cheers for responses