HP 2312i SAN vdisk and LUN performance configuration questions

[Sly]

We are in the process of upgrading our hosts from ESX 4.0 classic to ESXi 4.1 embedded one at a time and during this process we will be reformatting one of our vdisks on our HP 2312i SAN. Here are the questions related to this that I would appreciate your help with:

1) When we recreate the LUNs on this vdisk, can I use a HW initiator instead of a SW initiator? I see on the Hardware Compatibility Matrix that it looks like this is possible using "qla4xxx version 5.01.00.vm1". If so, is this beneficial and how do I do so on ESXi embedded? Will I see performance improvements? Are there any down sides to doing this?

2) Is it better to format the LUN with a block size of 4MB? We have been using 1MB.

3) With this SAN does Round Robin path selection improve multipathing over the default MRU selection?

Thank you for your help.

[redsi]

Sly,

I'm not familiar with your HP hardware but here's the answer to your question #2:

yes, If you're using Gigabit+ Ethernet and configure esxi 4.x jumbo frames set to above 4000 (preferably 9000) and your managed physical switche(s) / routers in the middle connecting the iSCSI to the host hardware are also all configured for jumbo framing set to 4000+ then you will see improved performance to the iSCSI as less packets and operations are required to move your data.

As an example for you, in my test lab with iSCSI+ QNAP TS-659 Pro RAID 5 with 6 x 2TB Seagate Constellation ES and 2 LACP bonded 1Gb NICs.

I see average 100 MB/sec individual transfers over 1 Gbit Ethernet with Jumbo frames @ 9000 and 4MB disk formatting versus 40-50 MB/sec without jumbo frames (using default 1500) and 1MB format.

see http://kb.vmware.com/kb/1007654 for info on enabling jumbo frames in esx/esxi 4.x

remember that the jumbo framing is end to end, so all devices in the path from vmachine/host to iSCSI hardware need to be set to use jumbo framing @ 4000+ (preferably 9000 on 1 Gb+ Ethernet)

-redsi

[Sly]

Redsi,

We did enable jumbo frames and it should be enabled across the entire iSCSI storage network. I see that 4MB block size seems to be the most popular choice, based on your comment, is it better to make your block size 8MB? Do people ever try to align their MTU with their block size?

Sly

[logiboy123]

1) Using a HW Initiator requires that your host has an iSCSI HBA or a NIC that is capable of TCP/IP offloading and Jumbo Frames if you want to use an MTU higher then 1500.

For example I recently created a vSphere environment uses DL380 G7's. When I created the environment I was using Hardware Initiators as the first four inbuilt NIC's on a DL380 G7 are capable of this, however when I turned on Jumbo Frames with an MTU greater then 1500 I lost all connection to the iSCSI SAN. Turns out the inbuilt NIC's could be HW Initiators, but couldn't do Jumbo Frames. So to sum up; it depends entirely on the hardware your hosts are running on.

2) Depends on the size of your LUN's and the size of the files you want to put on them. Block size to file size limits are:

1MB = 256GB file size Limit

2MB = 512GB file size Limit

4MB = 1024GB file size Limit

8MB = 2.48GB file size Limit

3) Generically speaking multipath IO is the way I go with most iSCSI SAN configurations. Especially with the P4000. I would look for some documentation from HP around best practise setup for VMware. You can also configure it with multipath IO round robin and see if you get a performance increase. A nice walkthrough I found is here:

Regards,

Paul Kelly

[Sly]

Paul,

My hosts are HP DL380 G6 and G7 boxes, each with 4 onboard broadcom NICs. In each host, I have added an additional Intel ET Quad Port NIC (82576 chipset), for a total of 8 NICs per host. Based on the documentation for this Intel card, those two features may be available, I am just not sure they can both be on at the same time. For most things, I have a MB-based NIC paired with the Intel-based NIC for failover purposes, so if one goes the other will take over. Based on what you indicated, I may want to make my Intel-based NICs my default iSCSI connectors.

How can I test out making one of my Intel NICs into a HW iSCSI initiator? Is their a procedure documented?

Do I have to disable the other SW iSCSI initiator on the same host before enabling the HW-based initiator?

Thank you for your help,

Sly

[logiboy123]

You can have hardware and software initiators active at the same time.

If you go to your configuration tab on your host, then storage adapters; you will get a list of all the adapters. Each adapter will be assigned a vmhbaXX address. If you do not have any hardware initiators then the software initiator will be vmhba33 by default I think. So if you do have hardware initiators that ESX recognizes, then they will appear in this list.

If you read the walkthrough I posted in my earlier reply you will find a guide showing how to;

1) Create the iSCSI switch

2) Bind the iSCSI adapter to each pNIC

3) Modify the MTU size

4) Configure load balancing

Even though this walkthrough is for a P4000, on the ESX side of the instructions it will still hold true no matter what your storage device is.

I'm glad I could be of assistance!

Regards,

Paul Kelly

If you found this or any other post helpful please consider the use of the Helpful/Correct buttons to award points

[Sly]

All:

   Thank you for your replies.  Here are my findings as we went through this upgrade:

• We are only using SW iSCSI adapters.  Even though offloading maybe supported by the Intel and broadcom NICs, the research I have done indicates that the only truly supported HW inititators for vSphere are the QLogic-based iSCSI HBAs.  On an upside there is a new driver for our Intel 82576-based NICs thas supports netqueue, which should offload much of the CPU interrupts to handle data sorting to the VMs, to the NIC.
  
• On this SAN, I can set controllers to a specific vdisk and multipathing to a given controller is handled at the LUN level.  When I turn on Round Robin on all LUNs on all hosts, I can see activity on all 4 controller paths simultaneously.  I was able to get about 2Gbps speed with a single VM with multiple disks on multiple LUNS.
  
• I went with a 4MB block size.  It seems to be an optimal setting, although from my reading, VMware has optimized their storage format and I/O to the point where block size really should only be based on the largest VM size(including snapshots) you would ever have.
  

-Sly