VMware NSX

 View Only
Expand all | Collapse all

VM Not reaching Internet via Edge ?!

TryllZ

TryllZAug 19, 2023 10:58 AM

TryllZ

TryllZAug 20, 2023 10:24 AM

  • 1.  VM Not reaching Internet via Edge ?!

    Posted Aug 18, 2023 09:32 PM

    Hi All,

    My NSX network setup is as follows, there are no firewall restrictions anywhere.

    TryllZ_0-1692393854208.png

    Both VMs can ping each other which tells me T1 Gateway is working.

    Both Edge nodes can ping firewall interface and internet as well.

    TryllZ_0-1692394407588.png

    TryllZ_2-1692394057283.png

    TryllZ_1-1692394043349.png

    Its the VMs that cannot reach the internet, the reply is coming from T0 interface.

    TryllZ_3-1692394127207.png

    Traceroute results with the following.

    TryllZ_1-1692394464844.png

    Any thoughts where the issue might be ?



  • 2.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 01:13 AM

    How do you setup your route redistribution on your t0 and t1 route advertisement?



  • 3.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 10:58 AM

    On T0 Gateway

    TryllZ_0-1692442512114.png

    On T1 Gateway

    TryllZ_1-1692442552649.png

     



  • 4.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 12:24 PM

    have you created return routes on the firewall for your segments or are you using a dynamic routing protocol? it looks like your firewall can't send the traffic back. It knows the edge nodes, because they are in a network which is connected to your firewall.



  • 5.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 12:28 PM

    Thanks,

    I'm using dynamic routing with BGP, all routes are advertising fine in the router, and edge nodes as well.



  • 6.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 01:07 PM

    Is your TEP network functional? Can you do a traceflow under Plan-Troubleshoot? Can you ping the TEP IP addresses from your ESX server? 

    ping ++netstack=vxlan <dst IP> -s 1600 -d 

    Is your T0 activ-activ? If activ/actvi URPF Mode on none?

    can you look your rounting table on the edge vm and look if the segments in the routing table of your sr t0



  • 7.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 01:21 PM

    I'll do a traceflow once on the system.

    I had checked, ESXi server can ping TEP addresses of all ESXi hosts in the TEP, will still recheck.

    No, the T0 is not Active/Active when checked in edge CLI, its Active and Never Established (if I recall correctly). The firewall cannot ping the 2nd uplink interface IP addresses, 10.10.26.101, and 10.10.26.102. However, in the GUI the T0 HA is Active Active.

    I had checked routing table in Edge, it had all the networks, including segments, I have all networks allowed in prefix list.

    Will share the results in some time.

    Thanks  



  • 8.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 02:33 PM

    Results..

    ESXi Pinging TEP addresses, pinging ESXi TEP IP is 10.10.25.51, 10.10.25.52

    Traceflow from Edge Node interface

    TryllZ_0-1692454467305.png

    Traceflow from VM to internet

    TryllZ_1-1692454760475.png

    BGP Summary from Edge Node (State is Active but Uptime/Downtime is Never)

    Edge Node Logical Router, and Routing Table

    URPF mode is Strict.



  • 9.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 19, 2023 02:45 PM

    Where it says Dropped for No Route found, is this for incoming traffic or outgoing traffic, from Edge Node ?



  • 10.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 09:37 AM

    Hi  

    I think I understand why its not working.

    I set static default route, and the VM's can reach internet. which tells me there is no default route set on the router for BGP to be advertised.


  • 11.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 09:39 AM

    Yes that explains it. Our posts have just overlapped. You can specify in the BGP that the default route is passed along.



  • 12.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 09:41 AM

    Great, thanks, will do so and test again..

    On a similar note, why is the 2nd Edge Uplink on both Nodes in Active mode and not Established even though the HA is Active/Active in T0, any thoughts on that ?



  • 13.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 09:52 AM

    Bildschirmfoto 2023-08-20 um 11.47.10.png

     This are my FW Settings (PFSense Cluster) for my Neighbor.

    The problem with the 2nd edge uplink could be manifold.

    1. i would check if you can ping your firewall over the 2nd ip address of your edge node and vice versa.
    2. check the bgp configuration, sometimes it's a simple number error of the IP or the update interface.
    3. what does the NSX GUI show?
    4. is the BFD profile correct, are the timers right?


    PS: Kudos would be nice if I helped, because I still need them for my VMware Rewards profile



  • 14.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 09:58 AM

    1. i would check if you can ping your firewall over the 2nd ip address of your edge node and vice versa.

    I have checked it already, the firewall cnanot ping the 2nd interface on both edge nodes.

    2. check the bgp configuration, sometimes it's a simple number error of the IP or the update interface.

    All configurations are the same on both uplinks, the only thing being the 2 uplinks are connected to 2 interfaces on the same firewall.

    3. what does the NSX GUI show?

    For the 1st uplink it shows Success, I can see BGP exchange happening in both NSX and Firewall.

    For the 2nd Uplink it shows Down.

    4. is the BFD profile correct, are the timers right?

    This is default and has been untouched.



  • 15.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 10:03 AM

    Okay, then we have a problem in your setup. Which firewall are you using? Single or cluster?
    Is your lab nested?
    How are your VLANs configured?
    The edge IP must be pingable, even if no BGP neighborhood is established. As long as your layer 2 is not clean, no BGP will work.



  • 16.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 10:11 AM

    The Firewall is OPNSense, single, for now, I might gowith HA or setup 2 firewalls, not sure yet.

    Yes this is a nested lab.

    Sorry unsure how to answer "How is my VLAN configured", its with sub-interfaces on the firewall.

    I'll add my network diagram in a while, that should make the picture clearer.

    NSX VLANs are as follows, Host TEP (VLAN 23), Edge TEP's (VLAN 24), and Edge Uplinks (Uplink 1 VLAN 25, Uplink 2 VLAN26). Edgeup Uplink portgroups in the Distributed Switch are are carrying VLANs 25, 24 (Uplink 1), and 26, 24 (Uplink 2).



  • 17.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 10:15 AM

    What are your security settings on your uplink dvPG?
    You need to allow promiscuous mode, mac address changes and forged transmits for it to work cleanly.

    Bildschirmfoto 2023-08-20 um 12.14.53.png



  • 18.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 10:20 AM

    I recall setting that to Allowed on the Baremetal, will need to double-check on the Edge Uplinks..



  • 19.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 10:24 AM

    Thanks a lot  appreciate all the help..



  • 20.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 02:45 PM

    I changed Security settings on both Uplink portgroups to Accept all the 3.

    Added Prefix list and route maps on the router.

    Strangely the router is showing the below in its routing table.

    TryllZ_0-1692542290879.png

    For the network 10.10.26.0 the next hop is 10.10.25.101 and 10.10.25.102, 65000 is the AS number for NSX BGP.

    The firewall cannot ping the 2nd Uplinks on both Edge Nodes (10.10.26.101, 10.10.26.102), nor can the Edge Nodes ping the firewall interface 10.10.26.1.

    I tested ping from within the Edge Node, of now is how the interface 10.10.26.101 responds compared to 10.10.26.102 with DUP ping responses.

    On the 2nd Edge Node its the reverse, 10.10.26.101 responds with DUP while 10.10.26.101 responds normally.

    Pretty sure I've messed up somewhere, could it be due to 1 router with multiple interfaces.

     

     

     



  • 21.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 10:00 PM

    dup packages can occur in nested environments. 

    I don't understand your setup and the routing table of the firewall. you don't need the route map. i would simplify the setup first. remove the routing on the opnsense. create an interface in your two uplink vlans and see that you can get the edge node via ping.

    10.10.26.0/24 over 10.10.25.101 is not allowed

    The Edge node needs a point to point connection to your firewall. The OPNsense needs an ip in the 10.10.25.0/24 and 10.10.26.0/24 network. Then enter 2 neighbors per vlan in frr.

    a network plan would help at this point. is your edge vm outside or inside your nested lab? both are possible and depending on that the setup is slightly different.



  • 22.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 20, 2023 11:19 PM

    The setup is as follows.

    Firewall, vCenter, and NSX-T are all running as regular VMs on baremetal ESXi. 

    4 Nested ESXi

    2 VM's running on Nested ESXi

    2 Edge Nodes running on Nested ESXi

    The firewall has 2 individual interfaces for each VLAN25 (10.10.25.1) and VLAN26 (10.10.26.1).

    Only 2 neighbours are added in NSX-T BGP settings, 10.10.25.1 and 10.10.26.1.

    Currently I'm suspecting a routing loop due use of 1 firewall. The tutorial I was following used 2 dufferent routers, still didn't understand why 2nd uplinks are not reachable, both uplinks are exactly the same in configuration.

    I'll setup a 2nd firewall and test this again.

    Thanks again  for all the help.



  • 23.  RE: VM Not reaching Internet via Edge ?!

    Posted Aug 21, 2023 08:01 AM