improving iSCSI performance: Delayed ACK, IOs per path, Large Receive Offload

[mpadfield]

I've raised a ticket with VMware for these 3 settings and I'll post the response, but I'd also appreciate any comments anyone makes.

We have two Dell R620 servers with ESXi 5.0 U1, directly connected to a Dell MD3220i iSCSI SAN. The SAN has 2 controllers and each server has 2 NICs connected to each controller. Round Robin and jumbo frames are enabled.

Performance isn't great, and I'm wondering if these settings would help...

Reducing IOs per path:

The default is 1000, I've read people saying they changed it to 1, or 3, or 8, or 8800 bytes, also that VMware don't support 1.

Disabling Delayed ACK:

It's enabled by default and sounds helpful, but lots of people say they turned it off, some saying it didn't help. VMware KB 1002598 says it may be necessary to do this with some SANs.

Disabling Large Receive Offload:
I found less info but some people advise it.

These steps aren't included in Dell's build doc but it is basic: http://i.dell.com/sites/content/shared-content/data-sheets/en/Documents/PowerVault_MD_iSCSI_Deployment_Guide_for_VMware_ESX50_Server_Software.pdf

[chriswahl]

I'd first examine the array to ensure that access to the spindles is optimal before working attempting to tweak advanced network configuration values - these are typically fine to be left at default unless you have an edge case that demands an exception (rare).

Raid type, storage grouping, LUN placement, etc. all have great impact on overall performance. This is a good tool for SWAG estimates http://www.wesworld.net/raidcalculator.html

[mpadfield]

Chriswahl, thanks for the comment and url, I like it a lot actually and will use it for purely the SAN side of things. In my case with our little SAN (it's got 12 x 300GB 10k disks) I tested raid 5 and raid 10, and went for raid 5 because it was similar for reads, not much slower for writes and I wanted the extra space, and I've no experience with raid 6. Hopefully I won't regret that. I tested raid 5 across 3 disks then 10 disks (the max we can have assuming 2 disks for spare), and 10 disks was very roughly a third faster so I'm thinking we'll go with that.

Re the iscsi networking, we have seen that some changes to the defaults have made significant improvements, although I accept your point and this guy's point What's the point of setting IOPS=1? that the improvement may only apply to a single test VM and in a real world with a bigger load the defaults may be fine. We'll finish testing tomorrow so I'll put what we went for then.

[Duncan Epping]

mpadfield wrote:

Re the iscsi networking, we have seen that some changes to the defaults have made significant improvements, although I accept your point and this guy's point What's the point of setting IOPS=1? that the improvement may only apply to a single test VM and in a real world with a bigger load the defaults may be fine. We'll finish testing tomorrow so I'll put what we went for then.

LOL, "this guy's" --> that is me :-)

[mpadfield]

That's still making me laugh now, if you want me to quote your blog to you again let me know! Good thing you didn't disagree with yourself and say: who's the fool that wrote that, should be shot!

[mpadfield]

In case anyone has the same setup and is interested, the conclusions we came to were:

Disabling Delayed Ack:
We're leaving it enabled. Disabling it didn't seem to make a significant change to performance. I'd read that a reboot was needed after disabling it so we did that. KB 1002598 and http://www.dell.com/support/troubleshooting/us/en/19/KCS/KcsArticles/ArticleView?

c=us&l=en&s=dhs&docid=599631 say it may need disabling if there's network congestion but we didn't notice any problem when running perf tests on both our hosts simultaneously which is probably about as busy as they'll get.

IOs per path:
We tested 1, 3, 7, 8, 9, 20, 100 IOs, and 8800 bytes. They were all similar for random reads and writes with small IO sizes and numbers of outstanding IOs. We thought this was because the bottleneck was the disk performance. With sequential, and random with large IO sizes and outstanding IOs, the max speed varied from about 130 MB/s to 229 MB/s for reads, and about 3/4 of that for writes (I don't have the numbers in front of me). The fastest were 8 IOs and 8800 bytes, with nothing to choose between them. We went for 8. The disk usage of our VMs will be varied but mostly random, so I'm not sure what difference it's going to make really. And there's also depping's point that there might not be any difference in real usage anyway, above.

Large Receive Offload:
We didn't bother testing it, wasn't widely mentioned on forums, and we couldn't see how 2 x 1gig links were going to get much faster than we'd got.

So the changes/decisions we made were:
- used the max number of disks possible for the storage group
- enabled jumbo frames, set the mtu on the port group and vswitch, don't do it on just the port group like me and wonder why performance is identical!
- didn't disable Delayed Ack or Large Receive Offload
- set the default PSP for VMW_SATP_ALUA to VMW_PSP_RR and made sure the LUNs were using round robin
- set the IOs per path to 8
- didn't use iscsi port binding which the Dell doc above incorrectly recommends

[Duncan Epping]

quantify "isn't great" and explain what your array setup looks like... difficult to provide any opinion based on the info above.

[mpadfield]

depping, thanks for showing an interest and if you've got any comments I'd be grateful. It's a very little setup:

2 x Dell R620 servers

1 x Dell Powervault MD3220i with 2 controllers each with 4 NICs. It contains 12 x 300GB 10k disks. 2 disks are spare, the other 10 are a single raid 5 storage group.

There are no switches involved, we've got a pair of patch cables going direct between each host and each controller, ie 4 cables from each host. We think now that not using switches was a mistake and unhelpful advice from the Dell guy who sold it to us. It seems to be not best practice, and when I call Dell and say we're not using switches they're amazed!

On Dell's advice we've changed the subnets. The default is for port 0 on both controllers to be in one subnet, port 1 on both controllers to be in another etc, ie 4 subnets in total. Dell said each port needs its own subnet with direct connect so we now have 8.

The Dell doc (in my first post) advises setting up iscsi port binding but this is wrong. It's only appropriate if all the vmkernels and target are on the same subnet, as per VMware KB: Considerations for using software iSCSI port binding in ESX/ESXi.    This isn't the case with the subnets left as the default or as we changed them. Dell agreed that the doc's wrong, and said they already knew but didn't know a date when they'd fix it (thanks Dell, nice one I didn't need those few hours).

The Dell doc doesn't mention the settings I'm posting about, iops per path, delayed ack, large receive offload. I raised a case with vmware who said they don't make recommendations, they should come from the SAN vendor, and also that they do support changing these settings. I was concerned about that because on this forum someone says vmware told him they don't support iops=1 What's the point of setting IOPS=1? , maybe they didn't when he posted but they do now. I phoned Dell who said they had no recommendations about these settings and I should do my own tests. I've been messing around with them since. We'll decide tomorrow what we think and I'll post it here but we can change it if you or anyone advise otherwise, so I'd still be grateful for opinions.

[Duncan Epping]

mpadfield wrote:

The Dell doc doesn't mention the settings I'm posting about, iops per path, delayed ack, large receive offload. I raised a case with vmware who said they don't make recommendations, they should come from the SAN vendor, and also that they do support changing these settings. I was concerned about that because on this forum someone says vmware told him they don't support iops=1 What's the point of setting IOPS=1? , maybe they didn't when he posted but they do now. I phoned Dell who said they had no recommendations about these settings and I should do my own tests. I've been messing around with them since. We'll decide tomorrow what we think and I'll post it here but we can change it if you or anyone advise otherwise, so I'd still be grateful for opinions.

I wouldn't worry too much about changing them from a support perspective. Only when an issue arises VMware support may ask you to reset it back to 1000.