NSX Design Guide Query -vMotion Subnet

[Techstarts]

I'm reading a NSX Design Guide. I'm trying to understand below table especially in context with Racks diagram.

The Guide suggests that all L2 should be terminated in ToR. However I observed nodes are spread across Racks.

In this case, if VM needs to be vMotion between the Racks, it has to go through L3 is it right?

Referring above table, I understand I have to create following VLANs and I have to enable routing between these segments except VXLAN

Compute Cluster A	Rack 1 (ToR)	Rack 2 (ToR)
Management VLAN 100	10.100.1.x/24	10.100.2.x/24
vMotion VLAN 101	10.101.1.x/24	10.101.2.x/24
VXLAN VLAN 102	10.103.1.x/24	10.103.2.x/24

The thing I finding hard to absorb is the vMotion subnet? vMotion subnet for hosts in Rack1 and Rack2 is different. It means vMotion traffic will be routed? is it correct? I know it is supported but it was not recommend. Has the recommendation changed?

[smitmartijn]

Hi,

Ever since the introduction of long distance vMotion in vSphere 6.0, vMotion over routed connections is supported. I don't think there's an official stance on which to use in which situation, just as long as you meet the requirements. Also, in the recommended spine-leaf topology design, it doesn't matter much whether the traffic is routed or switches, as the traffic will follow the same path (first to a spine then to the leaf the traffic is meant to).

Here's a bit more information from the KB:

vSphere vMotion Networking Requirements

Place vMotion Traffic on the vMotion TCP/IP Stack of an ESXi Host

Hope that helps,

[Sree]

It is not all a thumb rule to terminate VMotion at TOR . Like mentioned in this thread- Just ensure you are aware of VMotion RTT . L2/L3/Stretched L2 doesn't matter.  Based on the network topology(Overall design,firewall decision ,DMZ etc)i usually decide VMotion should be L2/L3 -were to terminate etc. Especially in multisite design scenario, based on network zone -some will be l2,few will l3 and others would be terminating within the rack etc.

[lucasitteam]

Page: 119 from Design Guide

From a VMware support point of view, the historical recommendation has always been to deploy all the VMkernel interfaces used for vMotion as part of a common IP subnet. This is not possible when designing the network for virtualization using L3 in the access layer, as it is mandatory to select different subnets in different racks for those VMkernel interfaces. Until this design is supported by VMware, it is recommended that users go through the RPQ process and allow VMware validate designs on a case-by-case basis.

=> So, please check the support statement.

The design guide clearly states that in Routed Network Topology L2 is terminated ToR. Below reference from Design Guide Page:118

When deploying a routed data center fabric, each VLAN terminates at the leaf switch (i.e., Top-of-Rack device), so the leaf switch will provide an L3 interface for each VLAN.

Hope it helps

[bayupw]

Martijn has explained on the vMotion part in above reply.

You mentioned that "Referring above table, I understand I have to create following VLANs and I have to enable routing between these segments except VXLAN"

For VXLAN VLAN 102 (or 103 on the picture), please note that in Leaf-Spine where L3 is on leaf, the VLAN for VXLAN (VTEP) will be terminated on the leaf switches and the subnet will be different across rack, 10.103.1.0/24 for Rack 1, 10.103.2.0/24 and so on.

Therefore you will still need to do routing between the different VXLAN segment e.g. between rack 1 and rack 2.

The VLAN ID of VXLAN/VTEP need to be the same, the design guide says: "Keeping the same VLAN ID simplifies the configuration for every rack and only requires configuration once."

Another reason is when you are having vSphere cluster spanning across racks, the VXLAN/VTEP VLAN ID of a cluster must be common so you must use same VLAN ID across different racks but the VLAN ID for each leaf maps to a unique subnet

Some VMworld session on NSX vSphere design that might be useful:

NET5770 - Reference Design for SDDC with NSX & vSphere - Part 1

NET5792 - Reference Design for SDDC with NSX & vSphere - Part 2

[Techstarts]

Thanks Sreec​, lucasitteam​, bayupw​ and smitmartijn

The VLAN ID of VXLAN/VTEP need to be the same, the design guide says: "Keeping the same VLAN ID simplifies the configuration for every rack and only requires configuration once."

Another reason is when you are having vSphere cluster spanning across racks, the VXLAN/VTEP VLAN ID of a cluster must be common so you must use same VLAN ID across different racks but the VLAN ID for each leaf maps to a unique subnet

Since the VLAN ID for each leaf maps to a unique subnet, it implies I need do routing between these subnet in spite of they belonging to same VLAN ID. Eventually all units inside racks will be L2 and will depend on L3 to route between the racks. E.g. If ESXi-Rack1 wish to communicate with ESXi-Rack2, the flow of communication will be via leaf-Rack1--> Spine-Leaf --> Rack2?

Anyone has spoken with VMware on the Support statement raised on vMotion in different subnets

Thanks for the link bayupw​ I will definitely check them.

[bayupw]

I can see that the design guide says that routed vMotion is recommended to go through with RPQ (Request for Product Qualification) process.

It doesn't mention which vSphere version.

With vSphere 6.0, vMotion can use separate TCP/IP stack which means it will have different default gateway, routing table, etc.

Prior vSphere 6.0, a static route is required if vMotion is running on different network and my understanding is that this require RPQ if we want to use it in production Configuring static routes for vmkernel ports on an ESXi host (2001426) | VMware KB

I thought routed vMotion does not require RPQ anymore with vSphere 6.0, but I could be wrong.

May need to check with the person who wrote the design guide or check with VMware Support/GSS.

On the leaf-spine side, it depends on your physical network architecture.

Leaf-spine can be L3 or some vendors use L2 fabric such as MLAG/MC-LAG, SPB, FabricPath, TRILL.

If you are using VXLAN (not NSX) on the underlying network then it is normally L3 and in this case VLAN is local to rack/leaf switch

If they are on different subnet then yes you will need routing in spite they are on same VLAN ID.

When a source leaf (let say leaf rack1) need to communicate to destination leaf (rack2), the destination vMotion subnet would be reachable from source leaf via routing (e.g. BGP) through the spine switches.

References:

Cisco Nexus 9000 Data Center Service Provider Guide - Cisco

Arista EOS Central – VXLAN bridging with MLAG

[Sree]

Vmotion btwn different subnet is supported , i have worked with VMware PSO team few months back and implemented one such solution.

[Richard__R]

Something else to note around simplified configuration by re-using VLAN IDs is your VDS design. You can only create a single VXLAN transport dvPG per VDS with a single VLAN tag. So if you didn't do that you'd in theory need a VDS per rack.

I'm pretty sure routed vMotion has been supported for a while as the others have mentioned and I have certainly seen this in production, presumably without RPQ under vSphere 6. I I'm also not aware of any recommendations against doing it that way.