VMware NSX

Hi All,

I have setup lab of NSX-T 2.4 where Im trying to communicate East-West communication but after multiple attempt Im unable to establish the communication amount 2 diffrent subnet VM's

My lab environment

vmware workstation 15.5.0

NSX-T 2.4

ESXi 6.7

vCenter 6.7

Created 3 network on Workstaton 15.5.0

vmnet-8 ( For management Network) 192.168.0.x

vmnet-0 (vmnetwork) 192.168.1.x)

vmnet-1 (vmnetwork) 192.168.2.x)

I have created T-1 Gateway and 2 segments

Advertise Route (All Static Routes & All Connected Segments & Service Ports) on T-1 Gateway.

Seg-1 subnet 192.168.1.1

Seg-2 subnet 192.168.2.1

Created 2 VMs on vCenter and allocated IP

vm1 192.168.1.2

vm2 192.168.2.2

Added the segment on the VM network as per the IP address.

Now when I try to ping vm2 ip from vm1 VM, I'm unable to ping but only able to ping the default gateways of both the segments.

Please let me know if I missed somewhere.

-Sachin

Did you specify an Edge Cluster when creating the T1 router? If so, even without using services, traffic is going to try and go through the SR.

Well, As per my understanding there is no necessity to deploy edge nodes or cluster on T1 gateway under NSX-T . As by default "DR" Distributed router get installed on each transport node which in my case are ESXi Host which take care of floating distributed services across nodes.

To answer your question : No I have not deployed Edge nodes and cluster under T1 Gateway.

Adding points:

As stated I have 2 segments created using 2 different subnets.

I  have deployed 2 VMs connected with  2 segments having different subnet. Whenever, both the VMs ; hosted on same ESXi host, the east west communication works well and there is no down status identified on Node, Transport zone or TEP on ESXi host. However, I'm unable to find any active TEP under the monitoring section of Node in NSX-T UI.

Now, If i move one VM to other ESXi host, then whole thing breaks down where the communication between both the VMs lost. TEP status shows as down on both the ESXi host, Node and Transport zone status shows as down status.

-Sachin

So the problem is in your TEP network. The tunnels you see in the monitoring section are only present if there are VMs connected to overlay networks. If there is nothing it makes no sense to establish these tunnels, which continuously test connection between TEPs to ensure all is working. That's why when you move the VM you get the errors, because then monitoring begins.

What network have you used for TEP? When you prepared you host TNs what transport network did you use and how did you configure this? Host preparation creates a vmk10 interface used for TEP communication and this is not working between the hosts in your lab.

Your problem is probably the MTU as a common complaint with later versions of Workstation is that they broke the higher MTU abilities. If you cannot ping between TEPs with this command, then this is the issue.

vmkping -S vxlan <TEP> -d -s 1572 -c 10

Yes, Im unable to ping the TEP IP's.  Is there any solution to resolve this..

If you can't ping between TEPs then nothing is going to work. I am not aware of a solution of increasing this MTU when using a modern version of Workstation. This is yet another reason that nested, complex labs involving NSX don't usually work too well.

hmm, so does that mean VMware workstation has compatibility issue with NSX-T version.

Any further comments.

-Sachin

Any further comments.

Don't use Workstation/Fusion for a nested NSX-T lab. That is all.

But can you ping with a smaller packet size, something less than 1472? MTU might not be the only issue here.

This is why I asked some other questions:

What network have you used for TEP? When you prepared you host TNs what transport network did you use and how did you configure this? Host preparation creates a vmk10 interface used for TEP communication and this is not working between the hosts in your lab.

Regarding MTU it is possible to enable jumbo frames on VMware Workstation:

Enable Jumbo Frames on Windows Host

Regarding MTU it is possible to enable jumbo frames on VMware Workstation:

Enable Jumbo Frames on Windows Host

From some of the other reports I've seen, this functionality appears to be broken. There was a post a while back by Mike Roy acknowledging this somewhere.

Tried enabling Jumbo frame on workstation network but still the same.

But can you ping with a smaller packet size, something less than 1472? MTU might not be the only issue here.

Send us some screenshots of your ESXi host network configuration, TN profile, etc.

Hi Mauricioamorim,

Yes, Im able to ping the TEP with lower  1470-75 as well as with jumbo frame 9000 but still unable to communicate between TEP-TEP amount 2 ESXi host.

Please find some of the screenshots of Traceflow and Node down status of NSX-T .

-Sachin

Yes, Im able to ping the TEP with lower  1470-75 as well as with jumbo frame 9000 but still unable to communicate between TEP-TEP amount 2 ESXi host.

So you're testing by pinging the TEP on host-A...from host A itself? That proves nothing and of course you can do that. You're still not proving anything with your traceflow screenshots. If you can't ping from host-A to host-B between the TEPs, even at a smaller MTU size, then you have a fundamental networking connectivity problem that transcends NSX-T.

@daphnissov Im able to do vmkping of TEP between both the ESXi host without any issue , no matter if the mtu is smaller or jumbo frame. So nothing fundamentally wrong anywhere.

As stated, the issue is the same, where im not able to perform East-West connectivity of vms which are on diffrent subnet hosting on individual ESXi host.

Let me know if have have any solution for this.

-Sachin

Ok, now I'm confused because previously you said you cannot ping between TEPs. Now suddenly you can? How about tell us what the TEPs are between your hosts and show the console output of the command I provided you earlier listing the destination TEP of the other host.

@daphnissov Im not playing here quiz quiz show.

I really have the issue then only I have raised the issue in community.

If you have the solution thats well and good. otherwise its ok no issue at all.

-Sachin

bhards4​

You keep asking for a "solution" yet the problem is not clear. That's why we're trying to determine what doesn't work. When you say "no, I cannot ping between TEPs" and then you later say "yes, I can ping between TEPs" you are the one playing "quiz quiz show" with us. Without systematically testing what can cause your failure (such as no TEP-TEP communication), we cannot offer a solution for you.

ok,

let me brief you again for your convinience

Im using VMware workstation 15.5 as the underlay infrasructure.

1) I have 2 ESXi host with IP 192.168.0.111 and 192.168.0.112

2) 2 segments created web (192.68.1.1/24 and 192.168.2.1/24)

3) TEP pool is in 192.168.0.131.-140

TEP IP assigned on ESXi are 192.168.0.131 and 192.168.0.132

Now as you have asked whether you were able to ping TEP on both the ESXi or not, then the ans is yes

Host 1

[root@esxi2021in:~] vmkping -S vxlan 192.168.0.132 -d -s 1465 -c 10

PING 192.168.0.132 (192.168.0.132): 1465 data bytes

1473 bytes from 192.168.0.132: icmp_seq=0 ttl=64 time=0.032 ms

1473 bytes from 192.168.0.132: icmp_seq=1 ttl=64 time=0.362 ms

1473 bytes from 192.168.0.132: icmp_seq=2 ttl=64 time=0.054 ms

1473 bytes from 192.168.0.132: icmp_seq=3 ttl=64 time=0.267 ms

1473 bytes from 192.168.0.132: icmp_seq=4 ttl=64 time=0.156 ms

1473 bytes from 192.168.0.132: icmp_seq=5 ttl=64 time=0.033 ms

1473 bytes from 192.168.0.132: icmp_seq=6 ttl=64 time=0.230 ms

1473 bytes from 192.168.0.132: icmp_seq=7 ttl=64 time=0.052 ms

1473 bytes from 192.168.0.132: icmp_seq=8 ttl=64 time=0.065 ms

1473 bytes from 192.168.0.132: icmp_seq=9 ttl=64 time=0.085 ms

Host 2

[root@esxi2021in:~] vmkping -S vxlan 192.168.0.131 -d -s 1465 -c 10

PING 192.168.0.132 (192.168.0.132): 1465 data bytes

1473 bytes from 192.168.0.131: icmp_seq=0 ttl=64 time=0.230 ms

1473 bytes from 192.168.0.131: icmp_seq=1 ttl=64 time=0.478 ms

1473 bytes from 192.168.0.131: icmp_seq=2 ttl=64 time=0.491 ms

1473 bytes from 192.168.0.131: icmp_seq=3 ttl=64 time=0.328 ms

1473 bytes from 192.168.0.131: icmp_seq=4 ttl=64 time=0.099 ms

1473 bytes from 192.168.0.131: icmp_seq=5 ttl=64 time=0.462 ms

1473 bytes from 192.168.0.131: icmp_seq=6 ttl=64 time=0.157 ms

1473 bytes from 192.168.0.131: icmp_seq=7 ttl=64 time=0.445 ms

1473 bytes from 192.168.0.131: icmp_seq=8 ttl=64 time=0.133 ms

1473 bytes from 192.168.0.131: icmp_seq=9 ttl=64 time=0.255 ms

- Sachin

Thank you, this is now helpful information. Please repeat the command as I have provided in this post and show the output.

Hi,

Please find the response below

What network have you used for TEP? - Its on the same subnet where the ESXi and the NSX appliance is "192.168.0.x"

When you prepared you host TNs what transport network did you use: I have prepare both the ESXi host using Transport node profile.

Host preparation creates a vmk10 interface : vmk10 interface is showing up on both the ESXi host but im unable to ping the TEP using vmkping on both the ESXi host.

-Sachin

Hey Sachin,

Can you validate the setup as https://blogs.vmware.com/services-education-insights/files/2019/02/Nest-Lab-Tech-Doc-2.6.pdf

This document describes how to build a NSX-T nested lab on vSphere, not one of the desktop products like Workstation.