[VLAN backed segment] No traffic to top of rack switch

[ChrisOk]

Hello,

I´ve been using a pair of Tier-0 router towards a top of rack switch (ToR) since NSX-T 2.3, i.e. with 2 different VLANs (80, 81) with 1 IP/IPv6 address each. BGP between ToR switch and the Tier-0 routers, tagged VLAN between the ToR and the physical hosts, ...

In meantime NSX-T was upgraded to 3.2.1.2. Now I wanted to setup a second pair of Tier-0 routers on different edge routers and started from scratch:

• created a new edge VM
• configured a Tier-0 on top of it
• created a VLAN segment with VLAN ID 3291, added an interface to the Tier-0 router and used the former mentioned
• created VLAN 3291 on the top of rack switch and tagged alle ports toward the physical hosts, added an IP address,…
• On the edge VM, I added the VLAN segment to a new edge switch
• The VLAN segment is added to the edge VM and in vSphere I can see this segment added to the third vNIC of the VM. (vNIC 1: Management VLAN, vNIC 2: Overlay Network, vNIC 3: New segment with VLAN ID 3291)
• ...

After I reboot the VM of the new Tier-0 edge router I can even see the MAC address of the 3291 segment (which is shown within vSphere) on the ToR switch, at least for  ~ 2 minutes after which it vanishes / times out. Also I cannot ping the IP address. So something is missing.

• VLAN Segment with ID 3291
  • 

 

• Edge VM with "VLAN segment" added (
  • (On a side note: I have the option to choose a distributed port group of 3291 in this stage aswell which results in no traffic, either. I tried distributed port group first as the existing edge routers with VLAN 80 and 81 have distributed port groups configured)
• 
  
  • 

     

    
    

     

  
  

   

   

  • 

     

  
  

   

 

 

 

 

 

 

• IP for Interface on T0
  • 

     

• The interface shows no traffic received
  • 

 

 

 

 

 

 

 

 

I´m currently stucked, so any help or hints where to look next, would be great.

[Lalegre]

,

Seems you are doing everything correct, let´s do a quick test. This portgroup 3291 change it to Trunk Mode and tag the VLAN Segment from inside NSX-T. What happens if you do that?

[ChrisOk]

Configuring Trunk Mode is here: https://kb.vmware.com/s/article/1004074

How can I tag the VLAN Segment from inside NSX-T?

 

[Dragan Ilic]

Different options are available on design edge VM -- ToR overlay/VLAN configuration, as always with NSX. Based on your setup I think you can proceed with your config in following way:

- create NSX segment / VLAN TZ / trunk type - mean in field VLAN allow all VLANs ie 0-4094 - connect your edge VM nsxhostswitch uplink1 to this segment

- create another NSX segment for T0 interface BGP connection up to ToR / VLAN TZ / add 3291 VLAN on that segment

- test connectivity between T0 and ToR on that link

Be aware that this is completely possible setup and is used in smaller environments (troubleshooting is harder because edge VM is "sitting" on NSX segment etc) - for bigger and more scalable ones - edge VM uplinks are connected to trunk vDS PGs and you should use dedicated VLANs with appropriate failover teaming inside uplink profile for overlay and T0 - ToR interconnection.

HTH,

Dragan