VM Not reaching Internet via Edge ?!

[TryllZ]

Hi All,

My NSX network setup is as follows, there are no firewall restrictions anywhere.

Both VMs can ping each other which tells me T1 Gateway is working.

Both Edge nodes can ping firewall interface and internet as well.

Its the VMs that cannot reach the internet, the reply is coming from T0 interface.

Traceroute results with the following.

Any thoughts where the issue might be ?

[DanielKrieger]

How do you setup your route redistribution on your t0 and t1 route advertisement?

[TryllZ]

On T0 Gateway

On T1 Gateway

 

[DanielKrieger]

have you created return routes on the firewall for your segments or are you using a dynamic routing protocol? it looks like your firewall can't send the traffic back. It knows the edge nodes, because they are in a network which is connected to your firewall.

[TryllZ]

Thanks,

I'm using dynamic routing with BGP, all routes are advertising fine in the router, and edge nodes as well.

[DanielKrieger]

Is your TEP network functional? Can you do a traceflow under Plan-Troubleshoot? Can you ping the TEP IP addresses from your ESX server? 

ping ++netstack=vxlan <dst IP> -s 1600 -d 

Is your T0 activ-activ? If activ/actvi URPF Mode on none?

can you look your rounting table on the edge vm and look if the segments in the routing table of your sr t0

[TryllZ]

I'll do a traceflow once on the system.

I had checked, ESXi server can ping TEP addresses of all ESXi hosts in the TEP, will still recheck.

No, the T0 is not Active/Active when checked in edge CLI, its Active and Never Established (if I recall correctly). The firewall cannot ping the 2nd uplink interface IP addresses, 10.10.26.101, and 10.10.26.102. However, in the GUI the T0 HA is Active Active.

I had checked routing table in Edge, it had all the networks, including segments, I have all networks allowed in prefix list.

Will share the results in some time.

Thanks  

[TryllZ]

Results..

ESXi Pinging TEP addresses, pinging ESXi TEP IP is 10.10.25.51, 10.10.25.52

[root@d-esx-srv-cn5:~] vmkping -I vmk11 -s 9000 -d -S vxlan 10.10.23.57
PING 10.10.23.57 (10.10.23.57): 9000 data bytes
9008 bytes from 10.10.23.57: icmp_seq=0 ttl=64 time=5.915 ms

--- 10.10.23.57 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 5.915/5.915/5.915 ms

[root@d-esx-srv-cn5:~] vmkping -I vmk10 -s 9000 -d -S vxlan 10.10.23.57
PING 10.10.23.57 (10.10.23.57): 9000 data bytes
9008 bytes from 10.10.23.57: icmp_seq=0 ttl=64 time=5.392 ms

--- 10.10.23.57 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 5.392/5.392/5.392 ms

Traceflow from Edge Node interface

Traceflow from VM to internet

BGP Summary from Edge Node (State is Active but Uptime/Downtime is Never)

edge2(tier0_sr[1])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.102  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:22:42     UP  40      28      11     4
10.10.26.1                          65555       Activ never        DC  0       0       0      0
Sat Aug 19 2023 UTC 14:21:35.584

Edge Node Logical Router, and Routing Table

edge2> get logical-router
Sat Aug 19 2023 UTC 14:21:15.633
Logical Router
UUID                                   VRF    LR-ID  Name                              Type                        Ports   Neighbors
736a80e3-23f6-5a2d-81d6-bbefb2786666   0      0                                        TUNNEL                      4       6/5000
77f0d5e7-e687-48b2-83df-147cec4de28c   1      2054   SR-T0-GW                          SERVICE_ROUTER_TIER0        7       2/50000
b4a49245-8bb3-4c63-b455-40c56631a04f   3      2049   DR-T0-GW                          DISTRIBUTED_ROUTER_TIER0    5       2/50000
f6a2a880-335e-4092-9684-e4eeba1c70f1   4      2052   DR-T1-GW                          DISTRIBUTED_ROUTER_TIER1    6       4/50000

edge2(tier0_sr[1])> get route

Flags: t0c - Tier0-Connected, t0s - Tier0-Static, b - BGP, o - OSPF
t0n - Tier0-NAT, t1s - Tier1-Static, t1c - Tier1-Connected,
t1n: Tier1-NAT, t1l: Tier1-LB VIP, t1ls: Tier1-LB SNAT,
t1d: Tier1-DNS FORWARDER, t1ipsec: Tier1-IPSec, isr: Inter-SR,
> - selected route, * - FIB route

Total number of routes: 20

b  > * 10.10.13.0/24 [20/1] via 10.10.25.1, uplink-271, 00:23:27
b  > * 10.10.15.0/24 [20/1] via 10.10.25.1, uplink-271, 00:23:27
b  > * 10.10.23.0/24 [20/1] via 10.10.25.1, uplink-271, 00:23:27
b  > * 10.10.24.0/24 [20/1] via 10.10.25.1, uplink-271, 00:23:27
t0c> * 10.10.25.0/24 is directly connected, uplink-271, 00:23:32
isr> * 10.10.25.101/32 [200/0] via 169.254.0.130, inter-sr-278, 00:23:19
t0c> * 10.10.26.0/24 is directly connected, uplink-277, 00:23:32
isr> * 10.10.26.101/32 [200/0] via 169.254.0.130, inter-sr-278, 00:23:19
t1c> * 10.10.100.0/24 [3/0] via 100.64.0.1, linked-275, 00:23:24 <--- Segment Connected to 1st VM
t1c> * 10.10.200.0/24 [3/0] via 100.64.0.1, linked-275, 00:23:24 <--- Segment Connected to 2nd VM
t0c> * 100.64.0.0/31 is directly connected, linked-275, 00:23:32
t0c> * 169.254.0.0/25 is directly connected, downlink-280, 00:23:31
isr> * 169.254.0.128/25 is directly connected, inter-sr-278, 00:23:32
b  > * 192.168.1.0/24 [20/0] via 10.10.25.1, uplink-271, 00:13:18
b  > * 192.168.9.0/24 [20/1] via 10.10.25.1, uplink-271, 00:23:27
b  > * 192.168.11.0/24 [20/0] via 10.10.25.1, uplink-271, 00:13:18
t0c> * fc64:1a87:8e1f:3400::/64 is directly connected, linked-275, 00:23:33
t0c> * fe80::/64 is directly connected, linked-275, 00:23:33
Sat Aug 19 2023 UTC 14:22:22.305

URPF mode is Strict.

[TryllZ]

Where it says Dropped for No Route found, is this for incoming traffic or outgoing traffic, from Edge Node ?

[TryllZ]

Hi  

I think I understand why its not working.

I set static default route, and the VM's can reach internet. which tells me there is no default route set on the router for BGP to be advertised.

[DanielKrieger]

Yes that explains it. Our posts have just overlapped. You can specify in the BGP that the default route is passed along.

[TryllZ]

Great, thanks, will do so and test again..

On a similar note, why is the 2nd Edge Uplink on both Nodes in Active mode and not Established even though the HA is Active/Active in T0, any thoughts on that ?

[DanielKrieger]

 This are my FW Settings (PFSense Cluster) for my Neighbor.

The problem with the 2nd edge uplink could be manifold.

1. i would check if you can ping your firewall over the 2nd ip address of your edge node and vice versa.
2. check the bgp configuration, sometimes it's a simple number error of the IP or the update interface.
3. what does the NSX GUI show?
4. is the BFD profile correct, are the timers right?

PS: Kudos would be nice if I helped, because I still need them for my VMware Rewards profile

[TryllZ]

1. i would check if you can ping your firewall over the 2nd ip address of your edge node and vice versa.

I have checked it already, the firewall cnanot ping the 2nd interface on both edge nodes.

2. check the bgp configuration, sometimes it's a simple number error of the IP or the update interface.

All configurations are the same on both uplinks, the only thing being the 2 uplinks are connected to 2 interfaces on the same firewall.

3. what does the NSX GUI show?

For the 1st uplink it shows Success, I can see BGP exchange happening in both NSX and Firewall.

For the 2nd Uplink it shows Down.

4. is the BFD profile correct, are the timers right?

This is default and has been untouched.

[DanielKrieger]

Okay, then we have a problem in your setup. Which firewall are you using? Single or cluster?
Is your lab nested?
How are your VLANs configured?
The edge IP must be pingable, even if no BGP neighborhood is established. As long as your layer 2 is not clean, no BGP will work.

[TryllZ]

The Firewall is OPNSense, single, for now, I might gowith HA or setup 2 firewalls, not sure yet.

Yes this is a nested lab.

Sorry unsure how to answer "How is my VLAN configured", its with sub-interfaces on the firewall.

I'll add my network diagram in a while, that should make the picture clearer.

NSX VLANs are as follows, Host TEP (VLAN 23), Edge TEP's (VLAN 24), and Edge Uplinks (Uplink 1 VLAN 25, Uplink 2 VLAN26). Edgeup Uplink portgroups in the Distributed Switch are are carrying VLANs 25, 24 (Uplink 1), and 26, 24 (Uplink 2).

[DanielKrieger]

What are your security settings on your uplink dvPG?
You need to allow promiscuous mode, mac address changes and forged transmits for it to work cleanly.

 

[TryllZ]

I recall setting that to Allowed on the Baremetal, will need to double-check on the Edge Uplinks..

[TryllZ]

Thanks a lot  appreciate all the help..

[TryllZ]

I changed Security settings on both Uplink portgroups to Accept all the 3.

Added Prefix list and route maps on the router.

Strangely the router is showing the below in its routing table.

For the network 10.10.26.0 the next hop is 10.10.25.101 and 10.10.25.102, 65000 is the AS number for NSX BGP.

The firewall cannot ping the 2nd Uplinks on both Edge Nodes (10.10.26.101, 10.10.26.102), nor can the Edge Nodes ping the firewall interface 10.10.26.1.

I tested ping from within the Edge Node, of now is how the interface 10.10.26.101 responds compared to 10.10.26.102 with DUP ping responses.

On the 2nd Edge Node its the reverse, 10.10.26.101 responds with DUP while 10.10.26.101 responds normally.

Pretty sure I've messed up somewhere, could it be due to 1 router with multiple interfaces.

 

edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

^C
--- 10.10.26.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

edge1(tier0_sr[2])> ping 10.10.26.101
PING 10.10.26.101 (10.10.26.101): 56 data bytes
64 bytes from 10.10.26.101: icmp_seq=0 ttl=64 time=12.413 ms
^C
--- 10.10.26.101 ping statistics ---
2 packets transmitted, 1 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 12.413/12.413/12.413/0.000 ms

edge1(tier0_sr[2])> ping 10.10.26.102
PING 10.10.26.102 (10.10.26.102): 56 data bytes
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.513 ms
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.630 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.726 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.824 ms (DUP!)
64 bytes from 10.10.26.102: icmp_seq=0 ttl=64 time=21.919 ms (DUP!)
^C
--- 10.10.26.102 ping statistics ---
2 packets transmitted, 1 packets received, +35 duplicates, 50.0% packet loss
round-trip min/avg/max/stddev = 21.513/63.136/120.443/34.055 ms

 

 

[DanielKrieger]

dup packages can occur in nested environments. 

I don't understand your setup and the routing table of the firewall. you don't need the route map. i would simplify the setup first. remove the routing on the opnsense. create an interface in your two uplink vlans and see that you can get the edge node via ping.

10.10.26.0/24 over 10.10.25.101 is not allowed

The Edge node needs a point to point connection to your firewall. The OPNsense needs an ip in the 10.10.25.0/24 and 10.10.26.0/24 network. Then enter 2 neighbors per vlan in frr.

a network plan would help at this point. is your edge vm outside or inside your nested lab? both are possible and depending on that the setup is slightly different.

[TryllZ]

The setup is as follows.

Firewall, vCenter, and NSX-T are all running as regular VMs on baremetal ESXi. 

4 Nested ESXi

2 VM's running on Nested ESXi

2 Edge Nodes running on Nested ESXi

The firewall has 2 individual interfaces for each VLAN25 (10.10.25.1) and VLAN26 (10.10.26.1).

Only 2 neighbours are added in NSX-T BGP settings, 10.10.25.1 and 10.10.26.1.

Currently I'm suspecting a routing loop due use of 1 firewall. The tutorial I was following used 2 dufferent routers, still didn't understand why 2nd uplinks are not reachable, both uplinks are exactly the same in configuration.

I'll setup a 2nd firewall and test this again.

Thanks again  for all the help.

[DanielKrieger]

your setup is pretty similar to mine. this works with a firewall. a firewall also does not trigger a routing loop. You can get problems with async routing and TCP strict when using multiple VLAN interfaces for peering.

What are your dvPGs, how is your edge profile configured, how are your uplink segments and transport zone?

[TryllZ]

Great to know I'm doing something not different from others, hoping the below makes sense.

In NSX-T the T0 HA is Active/Active, in dvSwitch the portgroups Edge_UL1 and Edge_UL2 are Active/Standby of each other.

DVSwitch Portgroups

Edge Node Profile

Uplink Profile

Transport Zone

On the Firewall Side

Interfaces

Neighbours

10.10.25.101 of Edge Node 1 is connected to FW interface 9_NSX_Edge_UL1 and Edge Node 2 10.10.26.101 to 10_NSX_Edge_UL2

BGP Neighbour Configuration Uplink 1 (Left), Uplink 2 (Right)

Prefix list

[DanielKrieger]

What I noticed is that you seem to have applied your VLAN transport zone to your ESX hosts as well - this is not necessary. The uplink VLAN segments only need to be known to the edge nodes. How is your teaming policy written? Are you using named teaming policy? What are all the prefixes and routemaps doing? Personally, I would keep the setup simple for now and deploy something like that later. But I don't see anything that triggers your problem. Can you check on the switch if it can see the mac address of your edge node? I still think this is a layer 2 problem. Also, check the OPNSense to see if it has a mac address of your edge nodes from the 26 network. Just make a ping and then look in the Arp Table.

The problem is not so easy to debug remotely. In my opinion it's not the NSX or the OPNSense settings, much more something is crooked in your nested environment. Can you put a testvm into the dvPG and give it the VLAN ID 26 and test if it can communicate? I also had problems at the beginning with my nested environment that MAC addresses were not learned cleanly. Also make sure that Promiscuous Mode, Mac Address changes and forged transmits are active on the port group of the OPNSense.

[TryllZ]

Thanks for taking the time, really appreciate it.

---

What I noticed is that you seem to have applied your VLAN transport zone to your ESX hosts as well - this is not necessary.

---

Thanks for adding that.

---

How is your teaming policy written? Are you using named teaming policy?

---

Not sure what you mean by that.

What are all the prefixes and routemaps doing?

---

Learned to configure BGP in OPNSense from here as the documentation is not that great for OPNSense and BGP.

---

Can you check on the switch if it can see the mac address of your edge node?

I still think this is a layer 2 problem. Also, check the OPNSense to see if it has a mac address of your edge nodes from the 26 network. Just make a ping and then look in the Arp Table.

 

Can you put a testvm into the dvPG and give it the VLAN ID 26 and test if it can communicate?

I'll check these, just that the switch is just a smart unmanaged switch.

---

The problem is not so easy to debug remotely.

Definitely agreed.

---

Also make sure that Promiscuous Mode, Mac Address changes and forged transmits are active on the port group of the OPNSense.

---

The only confusion I have with this is where all is this applicable, everwhere ?, I have this configured on the vSwitch on Baremetal ESXi, is this to be configured on ALL dvSwitch Portgroups ?

[DanielKrieger]

---

The only confusion I have with this is where all is this applicable, everwhere ?, I have this configured on the vSwitch on Baremetal ESXi, is this to be configured on ALL dvSwitch Portgroups ?

---

It depends, I would check. I use the distributed switch on all my baremetal ESX servers. I have 2 ESX servers for management, running the vCenter, my firewall cluster, my NSX Manager, ALB Manager, Unraid and my backup server with Veeam. For my lab I have 3 Intel NUCs running ESX virtual servers which then have the NSX workload. These are only started when needed. If I want to test NSX, the ESX hosts for NSX come up, if I need my Tanzu or ALB Lab, I start other virtual ESX servers. because of the flexibility i use distributed switches everywhere and because i am lazy to type.

[TryllZ]

Interesting setup, I use 1 server and 1 switch to keep cost down.

I added VM to Edge_UL2 (VLAN26) and made ping test as well, interestingly, the VM can reach the firewall interface.

Edge has nothing in ARP table for Edge 2nd Uplinks

However, I reset firewall table states and suddenly both Edge node's second interfaces (10.10.26.101, 10.10.26.102) were able to reach firewall interface, and even after this there was no entry in the ARP table for both of these interfaces.

And when the edge nodes' 2nd uplink was able to reach the firewall, the 1st interface (VLAN 25) which was pinging before was not pinging.

Not sure I understand why this is the case.

Could it be due to NSX-T and Edge versions, or the HA for Edge setup in T0 Gateway (point being only 1 interface is active at a time) ?

I'm using NSX-T 4.0.1, Edge versions are also 4.0 something. The reason I'm bringing this is because I read on several sites that the way uplinks and their relevant VLANs were configured were changed between version 2.x and 3.x, unsire if 3.2 and 4.0.1 have similar ways. The Design guide is still 3.2

[TryllZ]

And after all reboot, its back to the same issue of not pinging VLAN 26 interfaces.

[DanielKrieger]

Even if the T0 is active/passive, the second interface must be pingable. What you describe sounds like a Mac learning problem. Have you checked the security settings of all port groups?

Was your test VM able to reach the Edge VM?
Did you have the TestVM on the same ESX virtual host as the Edge Node?
What does the ARP table on the switch look like? Are the Mac addresses of the edge there?

The edge nodes should always have the appropriate NSX version to the manager and co. The way the uplinks are configured has not really changed from 3.2 to 4.X.

 

[TryllZ]

What you describe sounds like a Mac learning problem

Likely a firewall problem then.

Have you checked the security settings of all port groups?

All settings are default, only Edge Uplink portgroup settings were changed to Allow, will change all portgroups settinga to Allow.

Was your test VM able to reach the Edge VM?

No

Did you have the TestVM on the same ESX virtual host as the Edge Node?

No, will test this and above.

I will add though that I did test a traceroute in Edge and traffic leaves through its Management interface which is in 10.10.15.0 network.

What does the ARP table on the switch look like? Are the Mac addresses of the edge there?

Its a very basic switch.

The edge nodes should always have the appropriate NSX version to the manager and co.

Where can this be checked, matching versions of NSX-T with Edge. To my knowledge Release Nodes are only for NSX-T and no matching Edge version information is there.

[TryllZ]

---

The edge nodes should always have the appropriate NSX version to the manager and co.

---

Found this list, I'm using the right versions, highlighted in Red.

VMware NSX-T Component Build Matrix | virten.net

Name Main Build Manager Controller Edge Kernel Module L2 VPN Client Standalone Edge

NSX 4.1.0.2	21761691	21761695	21761699	21761693	21761699
NSX 4.1.0	21332672	21332677	21332680	21332675	19300606
NSX 4.0.1.1	20598726	20598732	20598735	20598730	19300606
NSX 4.0.0.1	20159689	20159694	20159697	20159692	11197779

[TryllZ]

Looks like a firewall related issue, I did a traceroute

edge1> traceroute 10.10.26.102
traceroute to 10.10.26.102 (10.10.26.102), 30 hops max, 60 byte packets
 1  10.10.15.1  5.577 ms  5.385 ms  4.633 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  *^C

The firewall is sending ARP Broadcast request trying to find the IP address.

Sender IP 10.10.26.1 is the firewall interface.

I'm going to check about this on the OPNSense forum, see what they have to say.

Appreciate all the help  

[DanielKrieger]

the question is, why would the edge node go through the firewall at all? You do the traceflow from your edge and the edge should have an IP in the 26 network. So there is no reason to go through the firewall. It also looks to me that you are not in the VRF context of the SR T0 and then it is clear that the Edge is trying to communicate with the 10.109.26.102 via the MGMT interface.

[TryllZ]

Will run a Traceflow again.

Both Edge nodes' both interfaces (10.10.25.101, 10.10.25.102, 10.10.26.101, 10.10.26.102) are configured for BGP Peering with the both firewall's interfaces (10.10.25.1, 10.10.26.1) which is only not happening on the 26 interface.

True, I id not check any of this from the VRF in T0, will check this.

[TryllZ]

From within VRF I can ping all interfaces from 1 edge node to another, still can't ping firewall interface.

edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

Traceflow

[DanielKrieger]

Hi, sorry I'm just getting back to you, but I had to rewire my entire lab first. Anyway - I tried to reproduce your problem, but I didn't manage to do so. The whole thing works for me. You could also give your firewall virtual interfaces in the uplink networks and tag them via the portgroup. Then the traffic would be untagged and any problems with double tagging would be solved. But then you have to change your peering a bit and need 2 nvds on the edge VMs. But this works without problems in nested labs.

[TryllZ]

No worries,

Appreciate you taking the time to reproduce the problem.

Currently I'm going through the Edge logs, and will make it simpler and see where the issue is.

Thanks again  

[TryllZ]

Thanks  

I set the Edge Uplink portgroups to trunking.

And firewall ARP table now has the interface attached.

Now both interfaces are in Established state, and BGP peering on all Edge Interfaces successfully.

edge1> vrf 2
edge1(tier0_sr[2])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.101  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:12:58     UP  46      20      12     4
10.10.26.1                          65555       Estab 00:12:58     UP  46      20      12     14

Thu Aug 24 2023 UTC 17:54:55.772

edge2> vrf 1
edge2(tier0_sr[1])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.102  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:15:18     UP  48      23      12     12
10.10.26.1                          65555       Estab 00:15:18     UP  51      23      12     6

Thu Aug 24 2023 UTC 17:57:02.232

The issue is OPNSense VLAN interfaces cannot be created without tags, or cannot be set as 0.

[DanielKrieger]

Were your uplink interfaces not on trunk? Normally it is sufficient to specify only the VLANs you need on the trunk. But I am glad that it works now.

To make the OPNSense untagged the traffic, you have to assign a new interface to the VM and assign the interface directly under assignments. I personally would use a virtual vyos (opensource) router to do the BGP peering. NSX peers to the vyos and the OPNSense also peers to the vyos. The advantage would be you don't have to change your firewall every time you want to test something in BGP and the vyos is much more flexible in BGP configuration.

My Vyos has 2 interfaces, one to the firewall and 1 interface with vlan subinterface for BGP peering with my NSX environment.

[TryllZ]

Were your uplink interfaces not on trunk? Normally it is sufficient to specify only the VLANs you need on the trunk.

The uplink interfaces were Trunks for 2 VLANs only on each Uplink Portgroup, same as yours, 25, and 24 (Uplink 1), and 26, 24 (Uplink 2).

Earlier

Now

I had removed VLANs specified on the Trunks and allowed all (0-4094) on both the Uplink Portgroups in Distributed Switch.

[Leaha00]

Hey, did you ever get to the bottom of this, I have been trying to setup an NSX lab at home with an OPNsense router and I am having the same issue, I can ping the physical network, including the physical router, but the traceroute stops dead at the OPNsense VLAN GW thats connected to NSX?

I have Physical Router --> OPNSense --> NSX

Thanks

[TryllZ]

Hi,

I'm afraid no, the only way I got this to work was by allowing all VLANs through the Distributed Switch NSX Uplink portgroups (0-4094), which as far as I know is a valid VMware design.

[DanielKrieger]

 

I have the same Setup at a Customer and in my Homelab (fully Nested) and it runs very well. 

My Setup is Nested Edge on virtual ESXi hosts running on pysical ESX Hosts -> vyos Routter -> virtual PFSense Cluster (not Nested, runs on my physical ESX Servers) PPPoE dial IN or if PPPoE Fails for some reasons -> LTE Router - Internet

I use BGP ECMP between Edge - Vyos - PFSense.

You have to setup your routing right and your Outbound NAT on the PFSense/OPNSense. Even if the Networks get lerned from BGP, the Outbound NAT will not work Out of the Box, because your NSX Segment is not a Connected Network at the Firewall.

 

[DanielKrieger]

You have no default route in the routing table of your T0 SR. Therefore the traffic in the traceflow is dropped. You have to set default originate on your firewall so that the default route is passed on to your T0.