Static route issues on NSX-T

[Rob80]

Hi all,

It seems that I can't find the way correctly connect Tier-0 to external network via static route on NSX-T 2.4. I have attached below my provisional network setup and I can't understand where the issue occur. Any help would be appreciated.

Thanks

[daphnissov]

Your diagram shows you have two separate uplinks from your edges on VLAN overlays to two different networks. Not sure why you're doing this as it doesn't make too much sense. The proper way to do this is configure those uplinks on your T0 to be in the same subnet. One port per edge. Configure a HA VIP across those two T0 uplinks. Your static route on your Meraki has its next hop set to that HA VIP. However, in order to route outbound traffic from NSX-T land, you will need to configure a default route (0.0.0.0/0) on your T0 directed to the next hop upstream. That is usually on the same subnet as the T0 uplink in the VLAN overlay.

[Rob80]

Daphnissov,

Currently I have setup each uplink on the same subnet and connected to one port per edge as described above but would appreciate guidance regarding HA VIP configuration.

Thanks

[daphnissov]

This should be covered in the official docs. You configure the HA VIP on your T0. After creating uplinks for each edge, you create the VIP but do not assign it to any interface. Leave the field blank and it will auto-assume both available uplinks. Goes without saying but the HA VIP needs to be in the same subnet as the individual uplink ports.

[Rob80]

I did search online and number of times on NSX-t 2.4 and can't find HA VIP configuration only router ports.

[daphnissov]

Configuration => HA VIP

[Rob80]

Hi,

HA VIP has now been configured with IP of 192.20.11.5 which I can ping from VM's.

I added the following addresses to nsx-t  static route

And another IP on MX

But still no connection to outside world.

[daphnissov]

Ok, I need to be able to see these screenshots you're posting as they're extremely small. Please also show your edge profiles and what interfaces are connected where. Also show your transport zone (overlay) profile and how your hosts are connected.

[Rob80]

Please find attached few screen shots of existing configuration.

[daphnissov]

Ok, this gives me a better idea. Here are questions for you to check out based on your images.

1. Your overlay profile is specifying VLAN 1020. This means the actual vmnics you're defining on your ESXi hosts are connected to upstream ports in trunk mode with 1020 allowed. Are you certain this is the case? If yes => Ping something else on this VLAN. Remember to pass the -S flag to vmkping to use the VXLAN TCP/IP stack. If no => set to proper VLAN. Use VLAN 0 if connected to an access port, or if this transport node (ESXi host) is virtual and connected to a virtual switch (in which case the VLAN tag is stripped off).
2. Your edge uplink profile profile is specifying an MTU of 1600. Normally this is left at 1500. Are you certain that you have MTU 1600 available on the VLANs in use by your edge uplinks to the physical infra? If yes => You're ok, but verify. If no => Change to 1500. To eliminate this variable, I'd change to 1500 regardless for testing.
3. Your T0 uplinks on the edges are in 192.20.11.0/24 with addresses of .2 and .3 for the two edges. What's the HA VIP address? What is the gateway address on this segment?
4. You are mixing configurations in the new policy-based API of 2.4 and the older format of 2.3. Anything you have under Networking => Tier-0 Gateways please remove. All configuration should be under Advanced Networking & Security.
5. From Advanced Networking & Security => Routers => T0, show Routing => Static Routes
6. Show from the same menu Route Redistribution
7. Show your edge cluster
8. Show your T1 => Configuration => Router Ports
9. Show your T1 => Routing => Route Advertisement

[Rob80]

I have attached file with screens to some of the lines below.

In terms of Tier-0, do I need to remove all and recreate on Advanced networking? Shall I do the same with Tier-1 or keep as it is.

Gateway for HA VIP is subnet on meraki router 192.20.11.1

[daphnissov]

Your vmkping command in your screenshot is not what I wrote. Flags are case sensitive. From the ESXi host, do it again and check the result:

vmkping -S vxlan <TEP> -d -s 1572 -c 10

From your ESXi host, you should be able to do a vmkping from the TEP to the:

• TEP of your edge transport nodes (both of them) on the same subnet

From a VM that is attached to a logical switch, you should be able to ping (do not alter MTU):

• Gateway (T1 downlink for this segment)
• T1 linked port to T0
• T0 uplink HA VIP address
• Gateway on the HA VIP network (which you say should be 192.20.11.1)

After you've corrected all these things, please indicate which of these ping tests pass and which fail.

Regarding your static route, it's wrong. The T0 static route needs to be the gateway for that segment which is the SVI on the Meraki. So it needs to be 192.20.11.1.

Your Meraki must have a static route setting the next hop for any networks you do not wish to NAT as the HA VIP address.

In terms of Tier-0, do I need to remove all and recreate on Advanced networking? Shall I do the same with Tier-1 or keep as it is.

Yes, to eliminate complexity, remove anything you have for routing and switching that's not under Advanced Networking & Security.

[Rob80]

NSX-T to edge node ping didn't succeed:

-- 192.20.11.5 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

[root@ESXi:~] vmkping -S vxlan 192.20.11.5 -d -s 1572 -c 10

PING 192.20.11.5 (192.20.11.5): 1572 data bytes

--- 192.20.11.5 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

[root@ESXi:~] vmkping -S vxlan 192.20.11.2 -d -s 1572 -c 10

PING 192.20.11.2 (192.20.11.2): 1572 data bytes

--- 192.20.11.2 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

[root@ESXi:~] vmkping -S vxlan 192.20.11.3 -d -s 1572 -c 10

PING 192.20.11.3 (192.20.11.3): 1572 data bytes

--- 192.20.11.3 ping statistics ---

10 packets transmitted, 0 packets received, 100% packet loss

Regarding below

• Gateway (T1 downlink for this segment) - success
• T1 linked port to T0 - success
• T0 uplink HA VIP address - success
• Gateway on the HA VIP network (which you say should be 192.20.11.1) - unsuccessful

[daphnissov]

If you only have one ESXi host and, for whatever reason, you can't ping the TEPs on your edges but a VM which is running on that host and connected to an NSX-T logical switch has that level of access then something isn't right with that vmkping command.

Based on what you say here

• Gateway (T1 downlink for this segment) - success
• T1 linked port to T0 - success
• T0 uplink HA VIP address - success
• Gateway on the HA VIP network (which you say should be 192.20.11.1) - unsuccessful

it sounds like you do not have your static routes configured correctly. So next show how you have configured your static route on your L3 switch.

[Rob80]

Basically I have mx65 doing layer 3 routing between vlans where esxi connected via layer 2 switch ms120-8lp as MTU of 1600 required as a minimum. I atatcehd below screens of current setup where I haven't added anything on switch for static routing.

MX 65

and for ms 120-8lp

[daphnissov]

Your static route appears to be wrong. If .5 is the HA VIP address, you need to direct any networks which you want to route into your T0. This would be any logical segments that exist behind a T1. From your diagram, that appeared to be 10.x.y.z/24 subnets. You will either need to summarize those routes into one, or set static routes for each network.

[IPv4toIPv6]

Hello Rob80,

Base by you PDF you have 2 uplinks in 2 different vlans.

Did you check if you have L2 connectivity

VLAN 2011 from MX65 Meraki --- Edge Uplink1

Example:

MX65 Meraki  IP 192.20.11.1

Edge Uplink1 IP 192.20.11.2

There suppose to be connectivity.

Ping form 192.20.11.1 to 192.20.11.2 suppose to be ok

VLAN 2012 from MX65 Meraki --- Edge Uplink2

MX65 Meraki  IP 192.20.12.1

Edge Uplink1 IP 192.20.12.2

There suppose to be connectivity.

Ping form 192.20.12.1 to 192.20.12.2 suppose to be ok

Please before going to the Routing configuration check the steps from above to be sure you have L2 connectivity :smileygrin:

Then go on edge routing and add destination 0.0.0.0/0  GW 192.20.11.1,192.20.12.1

And if you have a default route just delete'it.

My recommendation is still to use BGP

[Rob80]

Hi,

In order to use bgp I need router capable in supporting bgp as one of the prerequisites is AS which seems can be only setup when mx in vpn concentrator mode, but then I have no vlans or network.

Thanks

[Rob80]

I have made the following amendments on mx device

and wonder if I need to do similar on the switch.

[daphnissov]

Your App-Tier is really a 10.10.20.0/25 network?

Do a traceroute from a host external to any of these networks to a host that resides on one of them. What do you get?

[Rob80]

I have some issues

[daphnissov]

Your static routes are not configured properly. When you traceroute to 10.10.10.2 from your ESXi host, it send the packet to its management vmkernel port gateway. If that is the L3 device, it drops the frame because it has no entry in its routing table to send requests for 10.10.10.0/24. I cannot advise you how to accomplish this with a Meraki device so I'd recommend reading the official docs or opening a support request with Cisco if necessary.

[dejongraymond]

From the NSX Tier-0 Gateway the 0.0.0.0/0 default route pointing towards your router makes sure northbound traffic is achieved. Make sure, like said earlier, you create static routes for all the networks behind the Tier-0 Gateway on your Physical Router to make sure southbound traffic works and point the routes to the HA VIP.

On your router check L2 connectivity. Check if you see HA VIP address in MAC address table. If you are available next week I might be able to have a look together with you on a remote session.

I would also like to know if your tagging VLAN on your port group or in NSX Segment. Misconfiguration there could be a reason for not having L2 connectivity from Edge Node towards physical router.

[Rob80]

I Can ping physical router gateway but and all addresses up to HA VIP

  HA VIP shown below as connected and I can ping it from router only

Uplink 1 & 2 also available in MAC table and can be ping them from router only

   Edge tagging on VDS

Remote session would be great if possible.