Best vSAN topology for two sites?

[PonF]

We are planning a new vSAN Environment and I wonder what you think of these three alternatives below? See also attached pictures for an illustration.
We have two different main production sites on each side of town, around 10km apart (fiber path probably longer). We will have around 5 hosts (6 if we must) on both sites (10-12 in total) and we have our own fiber ring between the sites with 100Gbit/s.

The applications running on the VM's serves the local production site, but the application also have its own redundancy so we will distribute primary and secondary VM/servers between the sites. That way the appliaction should work if one site goes down, but we still also want to use the VMware redundancy, migration and maybe HA functions. With the help of a few spare host (normally used in lab environment) we will to be able to run all machines on one single site (if for some reason the whole site will be unavalible for a long time).

Minimum is FTT=1 but we aim at FTT=2. I think we will use ESA.

1. One large Standard cluster.
   Since we have our own fiber you could see everything like one site? That way we can share resources how we want together with HA. We can use fault domains to spread out the components. There could be a problem with quorum if one site goes down or similar, but maybe if we use a third "site" for a couple of hosts we can avoid this. The third site is actually inside Site 2 but it has its own server room and another UPS/power supply.
2. Stretched cluster.
   We could for example have 5 hosts with Raid 5 on both sites and then use site mirror. The witness host will be placed at the "third site" mentioned in the first alternative.
3. Two separate Standard cluster.
   To have FTT=2 and save space we would need 6 hosts and Raid 6. Or if we have the space we could go for Raid 1 FTT=2 but I dont know if we have that much storage/license. We would not be able to use HA but can migrate between the sites if we connect vMotion between the sites.

1:

2:

3:

[PonF]

I used vmkping on current setup where we only have vMotion between the sites, and RTT seems to be max 0,5ms so at least that part should not be a problem.

[Geogee]

Hello,

if you have an application based redundancy and doesn't require to failover VMs to another site, than I will use 2 separate clusters, one on each site. You will not need to have any witness host.

But You will not be able to migrate VMs or start that VMs on another site after site failure. You will not have data on the "recovery" site.

If you need that, You will have to go with stretched cluster, and you are able to define, which VMs will have data on both sites, or which will have data only on one of sites. Everything only by assigning the correct storage policy.

If you have standard cluster, you don't have ability to specify, where the data will be placed and losing one of the sites can break up all the VMs. 

------------------------------
Jiri Viktorin
VCIX-DCV, VCIX-NV, VCF-VCI, vExpert
------------------------------

[PonF]

@Geogee

But if we use Raid6 and 6 fault domains in one large cluster (like my option 1 picture), then we could loose one site and the VM would still be okay? (4 of  6 components okay).

Only thing is that site 3 is withing site 2, but on another building...

I guess maybe we then are forced with Raid6 policy on all VM's to ensure they are evenly distributed to all three sites. Can you see any other downsides or benifits with one large cluster?

[Duncan Epping]

Keep in mind, that if a host fails in this fault domain scenario vSAN will try to rebuild the capacity within the fault domain itself. So it would be smart to take that into consideration as well if you want to increase resilience while minimizing risks; https://blogs.vmware.com/virtualblocks/2019/12/20/design-operation-considerations-using-vsan-fault-domains/

[PonF]

@Duncan Epping

If you mean we need disk capacity for this I understand, but you could aslo accept that it will not be possible to rebuild the component and because we have FTT2 that could be okay. With two separate clusters (example 3)  we could also not rebuild a host failure with raid 6 since we only have 6 hosts.

I think separate standard clusters is a common setup, but can you see any benifits using one large cluster like my example 1 instead? Or any obvious problems? (one would be if both fiber is lost).

[PonF]

If we would go with alternative 1 (one large cluster) and we loose Site 1 (either by a long power outage or double network failure). 
Does anyone see a problem to either manually, or preferably with HA , to start the VMs on the other hosts in site 2 and 3?