Strange logical router behavior?

[amflacco]

Hello everybody and nice to meet you!

I'have a lot of questions but i'll begin with this one, because it's strange. I have installed a nested NSX-T Environment for routing only VLAN Segment. Everything gone almost well but i have a strange behavior. This is my configuration

 - 4 VLANs backend on NVDS Segment 

 - One T0 routing this VLANs with an interface on same subnet of pfsense (as external router) (Edge with uplink on same NVDS)

 - Pfsense has an interface ip 172.16.2.254 and T0 172.16.2.252

My VLANs Backend works fine , i can connect vm to internet, i can route each other and so on ... but a strange behavior happen.

Assume that a vlan backend vm has ip 10.0.200.10 with a def-gw 10.0.200.254 direct connected with T0. Login on edge and :

traceroute to 10.0.200.10 (10.0.200.10), 30 hops max, 60 byte packets.    --> TRACEROUTE  FROM T0

 1  172.16.2.254  0.331 ms  0.427 ms  0.356 ms.  -->  ASK TO PFSENSE

 2  172.16.2.252  2.250 ms  2.113 ms  3.661 ms.  ---> BACK IN T0

 3  10.0.200.10  4.041 ms  3.837 ms  3.404 ms. ---> CLIENT

What happen is that T0 (who as an interface direct connected on subnet VLAN with ip address 10.0.200.254)  ask to pfsense (172.16.2.254 that is T0 default gateway) which respond, hey, you've got the subnet direct connected!

That would mean that if i didn't create a static route on pfsense my T0 would forward out the request for this subnet, even if it has the subnet direct connected.

Again

traceroute to 10.0.200.254 (10.0.200.254), 30 hops max, 60 byte packets  ---> FROM T0

 1  172.16.2.254  1.340 ms  1.255 ms  1.150 ms --> TO PFSENSE

 2  10.0.200.254  1.862 ms  1.761 ms  1.721 ms ---> OK NOW DIRECT TO .254 ( AND 172.16.2.252 JUMPED?)

This happen with all segment.

I hope I have clearly explained what I mean. In my opinion something is wrong. What do you think about? Why T0 forwards the routing request to pfsense instead of processing it directly?

 

Thank a lot!

Alessandro

[shank89]

Ok, so first things first..

 

If you are indeed using VLAN backed segments, they are not routed through the T0.  Think of them as a standard VDS portgroup, that is they just tag the VLAN onto the pnic and send it out to your physical fabric (heavily summarised the process here).  When creating these segments, you do not have to assign a connected gateway.  Refer to the screen shot attached.

There could be several reasons as to why traffic is passing your PFsense (routing incorrectly configured, it is the default gateway and all unknown traffic is passed there etc.)

 

Can you provide a network map and some indication of what you are trying to achieve?  I assume it is routing housed completely within NSX-T where possible (overlay segments).

 

 

[amflacco]

Hello shank89,
thank you very much for your quick reply. I am convinced of and I underline what you say. What I'm trying to do is an experiment based on what I found a few days ago by reading the vmware documentation that is:
"If you have only VLAN-backed logical switches, you can connect the switches to VLAN ports on a tier-0 or tier-1 router so that NSX-T Data Center can provide layer-3 services." . There is no a particular design or objective but only test if: can NSX do L3 switching on VLAN backend segments managed by N-VDS? 
Indeed, apart from the strange behaviour I wrote in the previous post, the packets are routed correctly, as if there was a real router.
The attached network topology correctly reports the desired forcing. It works even if I connect the Tier-1 service ports and this one to Tier-0, let's call it what we want, but the result is that the packets are rotated correctly at the end.

On tier-0 there is the route 0.0.0.0/0 via pfsense (172.16.2.254) and on the pfsense the route 10.0.0.0/16 via 172.16.2.252.
I find it strange that being 10.0.100.0/24 a subnet connected to a T0 uplink, he forwards the request to pfsense.

This does not happen if the tracepath is done from the vm in 10.0.100.10 to 8.8.8.8 or from outside the NSX environment.

From 10.0.100.10 --> 8.8.8.8

[root@centostpl ~]# tracepath 8.8.8.8
1?: [LOCALHOST] pmtu 1500
1: gateway 0.858ms
1: gateway 0.788ms
2: 172.16.2.254 1.406ms
3: 10.6.75.254 1.416ms
4: 10.0.131.112 3.417ms
5: 10.0.218.14 5.678ms asymm 6
6: 10.251.35.186 4.824ms asymm 5
7: 10.0.143.153 5.645ms asymm 6
8: 10.0.128.141 6.573ms asymm 7
9: 10.254.12.253 20.971ms asymm 11
10: 93-63-100-105.ip27.fastwebnet.it 8.477ms asymm 9
11: 62-101-124-17.fastres.net 11.263ms asymm 8
12: 209.85.168.64 10.950ms asymm 14
13: no reply

From Out NSX ENV --> 10.0.100.10

sh-3.2# traceroute 10.0.100.10
traceroute to 10.0.100.10 (10.0.100.10), 64 hops max, 52 byte packets
1 172.16.10.1 (172.16.10.1) 14.245 ms 5.219 ms 5.514 ms
2 172.16.2.252 (172.16.2.252) 5.230 ms 6.230 ms 7.363 ms
3 10.0.100.10 (10.0.100.10) 7.813 ms !Z 6.237 ms 6.823 ms 

If there was some routing problem I should be able to look at it right?

Do you find this as curious as I do?

 

Thanks a lot

Alessandro

[Mauricio Queiroz Santos Amorim]

From what I understood the strange behavior only occurs on traceroute done directly from T0 router. Is this correct?

Since T0 router has several interfaces, it might be choosing the uplink interface as the source interface and sending it to the next-hop for that network (in this case acting as an end device). When the packet comes back it now acts as a router and delivers it.

Please try to do the traceroutes mannualy specifying the source to see if this is the case: traceroute x.x.x.x source x.x.x.x

[shank89]

Agree it's not straight forward to understand.

Have you had a look at the forwarding and routing table on the edge to see if anything strange is there?

[amflacco]

Hi shank89,

I attached the new topology.

From edge

edgw01> get routes
Thu Jan 28 2021 UTC 09:23:56.073
0.0.0.0/0 gateway 172.16.2.254 interface eth0
169.17.0.0/16 interface docker0
172.16.2.0/24 interface eth0

everything seems ok to me.

Thanks a lot for your support!

Alessandro

[Mauricio Queiroz Santos Amorim]

Remember the Edge is not the router itself, but a pool of resources for multiple SRs. Whenever you test from the edge you will use its interface to exit, which is on the management network. For testing as if it were from the router itself you have to do it from the respective vrf.

[amflacco]

Absolutely!

Thanks again !

[amflacco]

Here I'am again.

Yes mauricioamorim only if traceroute is done directly on edge. From what you say I understand, correct me if I'm wrong, that it is normal behavior.

Actually specifying the vrf does not pass through the pfsense ... 

Thanks a lot for your support.

Alessandro

[Gutkowski]

Nevermind guys, this was just a silly bug.

I accidentally used non unique IDs inside of 2 HTML-templates.

All working as expected now.

Keep up the great work.