VTEP values ?

[GauravCCIE]

Hi Team

I have a very basic doubt regarding the assignment of VTEP id .

1) Lets say I have 2 DC ( lets say West DC and East DC )

2) I just have one rack in each DC

3) Each of these racks have one top-of-rack switch [ LEAF switch 1 and 2]

4) Leafs connect to spine and there is a pure IP/MPLS network from any ISP in middle

5) Workload"a" on west need to talk to workload "b" on east

With help of nsx , i have a logical switch

i assign name , transport zone , vni=5002

Hence workloads will be attached to the logical switch

All good till now

My question is : in case of software VTEP , where is vtep id assigned ? and by whom ?

I believe in case of software vtep , the tunnel end point is the physical server .. is it true ??

if thats true , who assigns the vtep id ? and how is it publically routed

if someone can fill the end to end data packet flow please

[rajeevsrikant]

Below diagram explains how VTEP communication happens in simple terms.

[bayupw]

For VXLAN to work, you would need to have MTU 1600 end to end from ESXi in West DC to ESXi in East DC and this includes the ISP in the middle.

The network device in the ISP need to be configured for MTU 1600 at the minimum.

There is VNI/VXLAN Network Identifier which is Segment ID and in your case is 5002

There is also VTEP IP address which is a VMkernel interface in ESXi.

This VTEP IP address across ESXi hosts (same site or different site) can be in different network.

The requirement is to have end to end MTU 1600 reachability

These VTEP IP addresses is assigned when NSX Admin configures VXLAN on the vSphere cluster.

This can be a statically assigned IP Pool (admin defines a range of IP Pool for the VTEP) or provided by DHCP.

Depends on what Replication Mode/Control Plane Mode are you using; when NSX Controllers are exist, ESXi hosts send VNI & VTEP information to NSX Controller which also known as VTEP report. The NSX Controller then sends a copy of this VTEP report table to each VTEP (ESXi hosts).

See below diagram

The NSX Controller also handle MAC table report. Similar to VNI, each ESXi hosts will send a copy of every learned MAC address in each VNI segment to the NSX Controller which also known as MAC report. The MAC report includes the VNI, MAC address and the VTEP IP that reported it.

No with this information, the ESXi hosts know which VTEP (ESXi host) has which VM (IP & MAC), see an example of MAC table report on ESXi-1 from below diagram.

Based on the table, if VM1 need to communicate with VM2, ESXi-1 need to send the packet to VTEP 10.20.10.11.

The diagram is taken from the VMware® NSX for vSphere Network Virtualization Design Guide ver 3.0

To understand more about this, read section 4.2 in the design guide

[GauravCCIE]

Thanks for the reply Bayu .

Hence can I summarize it as follows :

1) If the VTEP end points don't span the public internet , i may define any pool and by DHCP , I can get the VTEP IP addresses assigned to each ESXI host

2) If the VTEP end points are reachable via a public internet , I will need a public IP address as VTEP , This generally will be given to me by the service provider .

3) The VTEP is assigned on each ESXI host to one of vmkernel port.

---> For the MTU case , is there a equivalent of "mtu path discovery" in VM world ?

Also , do you have similar explanation for HARDWARE VTEP case as well? i.e the case where vtep tunnel end point is Leaf switch ( like cisco aci leaf ) ?

Thanks

GAURAV MADAN

[bayupw]

If you need to design NSX for multi-site, you can refer to this design guide: NSX-V Multi-site Options and Cross-VC NSX Design Guide

I would not recommend to use public IP of VTEP as this means exposing ESXi to the Internet.

Just to add a note that one ESXi can have more than one VTEP IP address, depend on configuration (multi-VTEP is normally use to achieve higher throughput)

All solution but L2VPN requires 1600 MTU.

I'm not aware of equivalent command in VMware for MTU, but you can use vmkping command to ping from the VTEP vmkernel, see these KBs:

Testing VMkernel network connectivity with the vmkping command (1003728) | VMware KB

NSX for vSphere VXLAN logical switch connectivity tests fail with the error: The VMKPing fault, [ setsockopt() LINUX_SOC…

Hardware VTEP integration requires NSX Enterprise Edition and compatible Hardware & firmware VMware Compatibility Guide - Hardware VXLAN Gateway

Hardware VTEP integration in VMware NSX-v is used to extend VXLAN logical switch to physical port in physical switch or also referred as Layer 2 bridging.

For example, you have a VM on a VXLAN Logical Switch VNI 5001 (let say the name of Logical Switch is Web_Tier Logical Switch) subnet 192.168.1.0/24 and you have a VLAN 101 that needs to be in the same subnet 192.168.1.0/24, then you need to bridge VXLAN 5001 to VLAN 101.

This can be performed in Software natively in NSX or in Hardware VTEP.

With Native bridging in NSX, you can bridge the Web_Tier logical switch to a PortGroup with VLAN 101

With Hardware VTEP bridging, you can bridge the Web_Tier logical switch to a physical switch | port | VLAN in physical switch

Some use cases for hardware bridging are for migrating from VLAN to VXLAN while maintaining IP, extend physical network services/appliances to VXLAN, extend virtual services to physical VLAN

To understand more about HW VTEP integration for VMware NSX-v, see this document: Hardware Layer 2 Gateways Integration with NSX

and blog post series by Dmitri Kalintsev here: HW VTEP | Telecom Occasionally