Stretched vSAN cross-connect - questionable design?

[FredrikK77]

Hi, we’re researching a vSAN stretched cluster design where we plan to convert a 4-node vSAN cluster into a 6-node stretched vSAN cluster with 3 nodes in each server room. Both server rooms have redundant power and local access and core switches. This switching infrastructure is connected using VSL / DAD across the server rooms. The server rooms are close to each other so we could cross-connect the ESXi management and virtual machine traffic uplinks, meaning hosts in room A has active uplink to switch infra in room A and passive uplink to switch infra in room B and vice versa.  The vSAN vmkernel adapters of all hosts are connected using a separate isolated L2 network which is stretched across the two server rooms. The witness is located in a third room with L3 connectivity to the other two server rooms. vSAN traffic separation is enabled so that witness traffic is going over the ESXi management uplinks. Inter-site links are redundant and with adequate bandwidth and latency. In our design reasoning, we think that cross-connecting the ESXi uplinks between the server rooms will be a more redundant setup as we would not experience a virtual machine outage (HA restarts) in case the switching infra in one server room goes down. Can you see any flaws in this design that we are missing other than that the fault domain boundaries become a bit blurry and troubleshooting can get more complex as a consequence?

[TheBobkin]

@FredrikK77

"The vSAN vmkernel adapters of all hosts are connected using a separate isolated L2 network which is stretched across the two server room"

Can you elaborate what you mean by this?

 

"In our design reasoning, we think that cross-connecting the ESXi uplinks between the server rooms will be a more redundant setup as we would not experience a virtual machine outage (HA restarts) in case the switching infra in one server room goes down."

 

You mentioned cross-connecting the rooms uplinks to switches for Management+VM traffic but if this is not also the plan for vsan-traffic then if the switches in a Fault Domain/site/room go down then they lose vSAN network, all those nodes get isolated from the cluster, all VMs become inaccessible to them and thus killed and restarted in the other site - I don't understand what you think your plan would prevent here if not also done for vsan-traffic.

[FredrikK77]

@TheBobkin Thanks for your feedback. Sometimes thoughts doesn't translate well into text, sorry about that :)

To elaborate: Our vSAN network is built on completely dedicated vSAN switches physically separated from the regular network infra which is carrying management, vsan witness and vm traffic. We have two vSAN switches in each server room for local redundancy and we have full mesh between all 4 vSAN switches using MLAG.

Also, to add some more context, we are using vSAN ESA v8.

Scenario A: Access or core switches in room A goes down but vSAN switches are intact and operational. In this case the vm's continue to run on the hosts in room A but they are inaccessible due to uplinks to the core/access being down. HA (which is running on the vSAN vmkernel adapters) will not restart them unless we also configure more isolation addresses. But this means there is an outage during vm restart. So either way, we will have an outage. Witness will lose access to hosts in room A.

Scenario B:  Access or core switches in room A goes down but vSAN switches are intact and operational. In this case the vm's continue to run on the hosts in room A and they are accessible due to uplinks to the core/access in room B. HA doesn't need to restart any vm and consequently we have no outage. Witness can also reach all hosts in this scenario.

Does it make sense?