VMware NSX

 View Only
  • 1.  How does ESXi detect VM IP Address?

    Posted Aug 13, 2016 12:18 AM

    It's been mentioned that all ESXi hosts will inform the NSX Controllers of the VM IP and MAC addresses that are connected to each of the VXLAN VNIs that are local to the host? That being said, how does a host detect the IP and MAC of the VM? I don't believe it's getting that info from vCenter.



  • 2.  RE: How does ESXi detect VM IP Address?

    Broadcom Employee
    Posted Aug 13, 2016 01:59 AM

    ESXi learns the MAC addresses of the VMs by intercepting arp/dhcp broadcasts at one of the earlier slots in the IO chain.  The User World Agent then sends the information to the controller(s).

    NSX can learn associated IP addresses with VMs by VMware tools, or using ARP snooping and DHCP snooping.

    NSX Learning IP addresses

    VMware NSX for vSphere 6.2 Documentation Center



  • 3.  RE: How does ESXi detect VM IP Address?

    Posted Aug 13, 2016 02:20 AM

    So if we have a VM with a static IP (so not using DHCP) and is silent on the network (meaning it has been ARP'd for or hasn't sent an ARP itself), the Controllers wouldn't know where that VM lives? What happens when we have 2 VMs on the same VXLAN segment but different hosts, and one of the VMs is completely silent (no ARP activity), and VM1 wants to talk to VM2? Since the Controllers wouldn't have the info on VM2 (because it is silent), will the host for VM1 have to send a broadcast first, and then after a ARP response from VM2 would the Controllers learn about VM2's MAC and IP?



  • 4.  RE: How does ESXi detect VM IP Address?

    Posted Aug 13, 2016 09:09 AM

    These articles explain  very similar scenario, i.e. how does VM1 on H1 learns the MAC Address of VM2 on H2:

    https://telecomoccasionally.wordpress.com/2014/10/27/nsx-v-under-the-hood-vxlan-arp-suppression/

    http://www.routetocloud.com/2014/12/nsx-v-ip-discovery/

    Step1: If Silent VM2 has not sent any Arp Query requests for its default gateway, or any DHCP requests, then the SwSec dvFilter cannot learn and send the IP Address to add the Controllers ARP Table. For this reason the first arp request of H1 to Controller gets ff:ff:ff:ff:ff:ff (I don't know) response. But during this request Controllers ARP table is populated with VM1 Arp  Entry through a VM IP Update Packet.

    (If the replication mode is Multicast, this step is already bypassed since thhere are no controllers, and the ARP Request  is sent  through a Multicast IP destination address that all VTEPs with members on that VXLAN has registered to)

    Step2:In that case H1 has no other choice to consider the ARP as an ordinary  BUM packet, Replicating this Broadcast as Unicast Packets to all other VTEPs that has members on the VXLAN. (Hybrid Mode offloads to Pysicala as  Multicast similar to Multicast Mode).  Since H2 VTEP receives this Packet and it knows that the MAC Address belongs to VM2 locally, the packet is forwarded to VM2. VM2 sees the Arp Query to itself as well as  IP and MAC Address of VM1  already in the packet, and sends a Unicast Arp Reply to VM1.  H2 looks for the MAC Address of VM1 and learns from the Controller that VM1 is behind VTEP1 on H1.  During the Arp Reply,  SwSec of H2 sends VM IP Update packet  to the Controller. At this phase the ARP Table of the contoller has both VM1 and VM2 Entries. So if VM3 on H3 needs to learn MAC address or VM1 or VM2 may learn it from Controller directly, reducing the broadcast, thus the CPU Load on Hosts. By this way the Scalability of the VXLAN increases, very Efficient mechanism for Small to Large scale environments.

    Don't know when it may be useful to disable ARP Suppression, but  is  enabled by checking "Enable IP Discovery" during the Logical Switch VXLAN creation. If not enabled there will be more ARP broadcasts in addition to other broadcast traffic:

    One Important point may be the DLR Arp Resolution Process.  Could not find how it works on the documentation, but Since DLR is connected to vdrPort which has no SwSec Filter, so Controllers can't learn about LIF Interfaces of DLR Instances and may return ff:ff:ff:ff:ff:ff as in Step1. But since on every host there is a DLR instance running this ARP Response should be from this Local LIF interface, and Controllers may not have a Role about Resolving the MAC Address of the DLR LIF by VMs.

    But if a packet has reached to the DLR on H1 from VM4 (again on H1, but on another VNI such as VXLAN-5002)  which is destined to VM1  on the same host (which is on VXLAN-5001  VNI)  then DLR may resolve ARP locally. But if the same VM4 needs to send a Packet to VM2 on VXLAN-5001 on H2, then it may ask the Controller to Suppress the Broadcast, but not sure if it uses this mechanism or sends as BUM traffic without asking the Controller first. Not having a SwSec dvFilter prevents registering the Mac of DLR LIF  to Controller through VM IP Update, but it may or may not prevent learning the VM2 Mac address directly from the Controller without asking every host for this MAC address that is already on the Arp cache of the Controller.