Highly available vSAN design - 8 hosts

[_steez]

Hello everyone,

I have a question about the vSAN and high availability.

Im confused with the data protection methods used in vSAN - FTT and RAIDs.

We would like to deploy vSAN among our 8 hosts which are distributed between 2 racks (4+4) and our main goal is to stay operation after a rack failure.

1. When reading documentation, the maximum amount of failures tolerated is 3. Does that mean that the VM object will have 3 copies randomly distributed between 3/8 hosts and when those 3 hosts fail only that VM will not be available? Will other VMs that were not residing on failed hosts still be available?
2. Is it possible to select where to place replica copies (for example each copy in each rack), because you can never know which host will fail and it would be nice to know that the other copy is in other rack.
3. Also im confused about FTTs and RAID. If im correct FTT means how many hosts can go out before cluster becomes unavailable. Why in documentation FTTs are used with RAID-1, where that RAID 1 comes in play. Does it mean that FTT=1 is RAID-1, and FTT=2/3 is equal RAID5/6? Why in examples only RAID-1 used and not R5 or R6, which offers better protection?

We were also thinking about making a stretched cluster between each racks, that way we could set primary FTT to 1 and secondary probably FTT also to 1.

What could be the optimal way how achieve high availability and best space consumption between 2 racks (4+4)?

Sorry for such a long post, im new to this concept and it is hard to grasp. If there is something that wasn't understandable let me know I'll try to rephrase.

[TheBobkin]

  If your main goal is to stay operational during a rack-failure then you are going to have to go with a Stretched-Cluster configuration here - nothing else will protect all the data against 4 of 8 nodes becoming available here as even if you had something crazy like FTM=RAID1, FTT=3 (4 copies of the data + 3 witness components = massive footprint!) this would be distributed across both racks and you would have maybe something like 50% odds of any data Object (e.g. vmdk, namespace) staying available (and VMs need access to ALL of their Objects or they are marked as Inaccessible). If you went with something like 4 Fault Domains (2+2 + 2+2) you would be in the exact same boat.

 

Regarding FTM(Fault Tolerance Method)=RAID1, regardless of the FTT (e.g. 1, 2, 3) vSAN stores each of these as complete replicas (no parity etc.) each on an individual Fault Domains (FD) - a FD is by default each node in the cluster (unless configured otherwise e.g. for rack-awareness or as a Stretched-cluster). FTT=2,FTM=RAID6 doesn't offer any more protection than FTT=2,FTM=RAID1 - they are both FTT=2 they just store the data in different ways (and have significant differences in footprint with the trade-off being performance).

 

Stretched cluster will require vSAN Enterprise/Enterprise+ licensing and also require a 3rd site to run the Witness on (this can be a standalone host, another cluster, cloud, max latency to it RTT=200ms and doesn't require a beefy connection) but will offer you a wealth of choices with regard to how you store your different data on this cluster - you can assign Storage Policies build for high redundancy but also performance (PFFT=1,SFTT=1,SFTM=RAID1), others that don't require as high performance but high redundancy and lower space usage(PFFT=1,SFTT=1,SFTM=RAID5), good redundancy and performance, lower space usage(PFFT=1,SFTT=0), data local to only one site (but won't be available if that rack fails) with FTM RAID1/RAID5 depending on needs.

 

"Sorry for such a long post, im new to this concept and it is hard to grasp. If there is something that wasn't understandable let me know I'll try to rephrase."
If you think that is long you should read some of my wall-of-text replies on here, thanks for keeping it brief and concise!

[niyijr]

Very useful feedback  I may be working on a similar arrangement to  in the near future.

---

wall-of-text replies

---

 

[TheBobkin]

, I am not really sure many people realise how many bricks it take to make a proper wall-of-text but here is an example of just my out-of-work-hours brick by brick written on almost solely vSAN Community from the last 4 or so years (~1/3 is copy of what replying to but near-zero log output etc.) :

Sure, I could probably explain some things with one line or with an analogy vs a WoT, but the way I see it is that it is not just answering the questions the person asked but laying out the fundamentals of why the question is there (and for the wider community to access) - you need to lay a foundation under that WoT and buttress it after all, maybe even get planning permission...

 

So, do please stack a WoT if you have the bricks, if I have time and the questions/concerns are legit then I will almost surely reply with a wall of answers .

[niyijr]

Haha! Classic. 

[_steez]

Thank you for the detailed answer and insights.

This clears some confusion, we were leaning towards a stretched cluster deployment but were unsure whether we have checked all possible scenarious with simple cluster. It is nice to have a suggestion from a pro.

[_steez]

Hello 

I still have some concerns when comparing stretched cluster vs cluster with 4 fault domains (2+2+2+2).

Aren't they essentialy the same in this deployment? Let me explain.

When using stretched cluster you can set PFFT=1 which will mean that one copy will be stored in the other rack, by setting SFFT=1 we ensure that second copy will be stored locally. So far so good we have a protection from single or all server failure in the rack.

But how about 4 failure domain case with FTT=2. Each rack can have 2 failure domains, with FTT=2 you can achieve the same result as the streched cluster. VM will reside in FD1, with FTT=2 2 copies will be distributed between FD2, 3, 4 guarantying that one copy will be copied to other rack. Only downside may be that all copies might get copied to other rack.

Is there anything that im missing here? What are the downsides to this deployment?

[TheBobkin]

 , A 4 Fault-Domain configuration (e.g. 2+2 + 2+2) can only provide a maximum FTT=1 (e.g. FTT=1,FTM=RAID1 requires 3 FDs, FTT=1,FTM=RAID5 requires 4 FDs) - if a site/room that housed 2 of these FDs failed then all of the data Objects would be Inaccesible if stored as FTM=RAID5 and by the laws of odds ~50% Inaccessible with FTM=RAID1.

Stretched clusters use nested FDs to provided higher FTT - PFTT=1 is basically RAID1 across the sites (and with Witness for quorum) and then the SFTT is the protection at the nested level within a site - basically it is a RAID1 of RAID1/RAID5 data, good examples of component placement diagrams can be found on Duncan Epping and Cormac Hogan blogs e.g.:

https://www.yellow-bricks.com/2018/03/19/vsan-stretched-cluster-pftt-and-sftt-what-happens-when-a-full-site-fails-and-multiple-hosts-fail/

 

 

[_steez]

Thanks, I was missing the part - maximum of FTT=1.