Expanding AF vSAN capacity on R730xd

[carlo_COT]

We have six R730xd servers in a stretched cluster.  Three at location A, and three at location B.  We are adding 27 new WS2016 VMs for a new project, and want to increase RAM and storage.

Currently each server has 2 disk groups, consisting of 1 Intel DC P3700 800GB NVME drive and 4 Intel DC S3610 1.6TB drives.

1.  Increase capacity

I want to add 2 drives to each server.  The S3610 is discontinued.  Should I add S4510 drives?

2.  Cache drives

I (kind of) followed the 10% cache-capacity ratio when I did one 800GB cache drive to four 1.6TB capacity drives.  Should I just add the drives to the existing disk groups (1 drive added to each disk group), or create a new disk group?  If I add a new disk group, the P3700 drives are also discontinued, so what replaces them?

3.  Increase performance

I've read that Optane drives can make a big improvement in performance.  Should I replace all of my P3700 drives with P4800X drives?  Here's the thing.  Since my servers are single processor, I had to add a Dell P31H2 expander card to be able to add 4 NVME drives to the R730xd.  Would it work, or will I get the full benefit of the Opteron drives if I use them with this expander card?

Thanks in advance.

[TheBobkin]

Hello Carlo,

"1.  Increase capacity

I want to add 2 drives to each server.  The S3610 is discontinued.  Should I add S4510 drives?"

The general recommendation when adding replacing devices used for vSAN is to use devices with the same or higher performance capabilities - the rationale behind this is that if you use slower devices then you can potentially slow down your whole cluster as they will essentially end up 'waiting' on these slower devices to do their share of the work (e.g. if you had one replica of data on a fast device and the other replica on a slow device). Thus I wouldn't recommend S4510 as they are a lower performance class than s3610:

VMware Compatibility Guide - ssd

VMware Compatibility Guide - ssd

"2.  Cache drives

I (kind of) followed the 10% cache-capacity ratio when I did one 800GB cache drive to four 1.6TB capacity drives.  Should I just add the drives to the existing disk groups (1 drive added to each disk group), or create a new disk group?  If I add a new disk group, the P3700 drives are also discontinued, so what replaces them?"

In All-Flash clusters it is more dependant on the workload IO profile than the % cache:capacity ratios (though bigger ratio is preferable and also note Write-buffer is 600GB max per cache-tier device so using something like a 2TB cache-tier device is a bit wasteful and won't add any benefit aside from lifespan over say a 800GB device):

Designing vSAN Disk groups - All Flash Cache Ratio Update - Virtual Blocks

If possible, additional Disk-Groups are always going to be more beneficial than adding to existing ones - better for improving performance (more IO paths and cache), improves resilience in the event of cache-device or deduped Disk-Group failure (as you have more disks left available for component placement).

If you can afford P4800x then do consider these as they are completely different level than P3xxx devices.

"Since my servers are single processor, I had to add a Dell P31H2 expander card to be able to add 4 NVME drives to the R730xd.  Would it work, or will I get the full benefit of the Opteron drives if I use them with this expander card?"

This is potentially a pain-point - the use of SAS-expanders with vSAN nodes are only supported in ReadyNodes with specific expanders and configurations, I am not aware if we currently have a support statement with regard to PCI-e splitters but am guessing they would be in the same boat. At a bare minimum using these is going to split the X provided by the lane so I would imagine these won't fully utilise the potential of the devices (if it is hitting cap with reduced X that is).

Bob

[carlo_COT]

Bob,

Many thanks for such a quick and detailed response.

Instead of the S4510, it looks like a more comparable drive to the S3610 capacity drive is the D3-S4610.  Correct?

VMware Compatibility Guide - ssd

Sounds like you are saying I could add a 1.92TB drive to each disk group.  Better performance would be to add a cache drive and the two drives in a new disk group on each server.  Best performance would be to replace the my current P3700s with P4800X drives. 

Regarding adding a new disk group:

The P3700, S3700 and S3710 are all discontinued, leaving the P4800X drives as the only drives on the HCL that are rated for vSAN caching tier, endurance class of D, and performance class of E or better.  Should I still buy P3700 drives for new disk groups?

VMware Compatibility Guide - vsan

Also, who do I ask to find out if the expander card would be supported for the Optane P4800X drives?

Thank you!

[TheBobkin]

Hello Carlo,

"Instead of the S4510, it looks like a more comparable drive to the S3610 capacity drive is the D3-S4610.  Correct?"

Yes, S4610 would be a more comparable device.

"Sounds like you are saying I could add a 1.92TB drive to each disk group.  Better performance would be to add a cache drive and the two drives in a new disk group on each server.  Best performance would be to replace the my current P3700s with P4800X drives."

Correct, in order of throughput improvement gains those would be the 3 options. As I said, while adding a new Disk-Group will improve the overall throughput, using drastically better performance devices in just this new Disk-Group likely won't garner as much gains as one would think due to the points outlined in my previous comment. Switching current Cache-tier devices for something more modern like P4800x and adding the new Capacity-tier drives to the existing Disk-Groups might be a happy medium here which keeps everything balanced and configuration-consistent across Disk-Groups - I would advise going for the 800GB versions over 400GB if possible but whether this is necessary is entirely dependant on your workload.

"The P3700, S3700 and S3710 are all discontinued, leaving the P4800X drives as the only drives on the HCL that are rated for vSAN caching tier, endurance class of D, and performance class of E or better.  Should I still buy P3700 drives for new disk groups?"

If the option is there to do a minor(ish) hardware refresh in general it is always better to go with something that you know will be supported and available for replacement for some time to come, this can avoid potential headaches down the road.

"Also, who do I ask to find out if the expander card would be supported for the Optane P4800X drives?"

If you have VMware PSO or pre-sales contact available then this would be likely best approach, you could contact us at GSS but note that we generally operate on a break/fix basis so we would likely have to engage our backline for clarification of this, my suspicion though is that this falls under expanders are expanders and thus only supported with ReadyNodes of specific types and configurations. With functionality there could even be a distinction in single-slot expanders vs dual-slot expanders but as I said this is unclear as I've never had to troubleshoot them in the wild (other than for non-vSAN purposes e.g. Dell BOSS). Note as well that we don't appear to certify any form of expanders on the ESXi HCL other than specifically as part of ReadyNode configurations and thus hardware vendor should be able to clarify this point at the least.

Bob