ESXi

 View Only
  • 1.  ESXi network configuration?!

    Posted Aug 03, 2015 09:57 AM

    Hi Folks

    I've a question for you guys, maybe it''s too silly for you but I am quite unsure about the following. If i have a ESXi Server with an onboard quadport NIC and I have a Cisco switch with spanning-tree activated what happens when I:

    Attach all 4 ports to the switch, create a standard vSwitch on the ESXi and attach all 4 NICs as active with LB policy "based on orginating Port ID".

    Based on my understanding it will distribute all my VMs across these NICs attached and all VMs are able to communicate (send AND receive) via those 4 vNICS/NICS. It will failover a VM which is pinned to a VNIC to another VNIC when i unplug the specific wire.

    BUT

    My collegues told me NO! You will be only able to send data over all NICs but receive only over one vNIC due to the source/destination MAC address of the VMs respectively the NIC of the host.

    Is that true? What exactelly is the process of this policy "orginating port ID". Because there is an additional one called "based on MAC....".

    Well usually we configure systems like this:

    A server has 2x quad port NICS. There is a vswitch with 2 attached NICs (first on card 1, first on 2nd one) and we configure "based on originating port ID". Further we do NOT configure a stack neither a Etherchanne on the switches which are connected to the NICs (via cross).

    We only want to have simpe LB and failover in case of a path outage. Will this work properly (send and receive)???? My colleagues told me that will not work as long as the switches are not stacked?? Why should it not? From my understanding the VM will switch over to another vNIC which is connected to another NIC which again is connected to an other standalone switch.

    When do wenn need to configure "Based on MAC" or "IP  HASH"? Well i know that when you configure "IP Hast" then you need to setup an Etherchannel between the Host and the Switch and set the policy to IP Hash otherwise you dont have additional bandwith.

    Why would we need to set adapters to standby and not active? Is that used to deny issues with loops in a not spanning-tree environment?

    How do you configure the network (amount of NICs, vSwitches, portgroup assignment, vmotion etc. and the switches)?

    Thank you so much



  • 2.  RE: ESXi network configuration?!
    Best Answer

    Posted Aug 03, 2015 11:52 AM

    what happens when I:

    Attach all 4 ports to the switch, create a standard vSwitch on the ESXi and attach all 4 NICs as active with LB policy "based on orginating Port ID".

    Based on my understanding it will distribute all my VMs across these NICs attached and all VMs are able to communicate (send AND receive) via those 4 vNICS/NICS. It will failover a VM which is pinned to a VNIC to another VNIC when i unplug the specific wire.

    BUT

    My collegues told me NO! You will be only able to send data over all NICs but receive only over one vNIC due to the source/destination MAC address of the VMs respectively the NIC of the host.

    Is that true? What exactelly is the process of this policy "orginating port ID". Because there is an additional one called "based on MAC....".

    It's pretty straightforward, in the default "based on orginating Port ID" configuration each vNIC of a VM only has a single active physical uplink at any given point in time. This physical NIC is used by the host to send as well as receive traffic for this particular vNIC (because your physical switch will learn the VM vNIC MAC on the currently active port it will only forward traffic through this link).

    The distribution is static and only changes when you add/remove/plug/unplug physical uplinks or the vNIC (i.e. power off/power on, disconnect/connect the vNIC). The behavior is the same for standard as well as distributed vSwitches with this policy. You can see the current mapping in the (r)esxtop network view:

    The Route based on source MAC hash option is very similar, but instead of using the internal virtual port ID, it establishes a static mapping based on the source MAC address of an Ethernet frame being sent from a vNIC. This approach also maintains a nice static MAC table on your physical switch to prevent MAC flapping. Traffic of a single VM vNIC will only use multiple ports if the VM uses multiple source MAC addresses. Usually that should never occur unless you do some fancy networking stuff inside the VM like multiple virtual interfaces, and it also means you need to allow forged transmits and MAC changes in the vSwitch port group security options. This article sums it up nicely:

    Hostile Coding: VMware: MAC hash based LB

    A server has 2x quad port NICS. There is a vswitch with 2 attached NICs (first on card 1, first on 2nd one) and we configure "based on originating port ID". Further we do NOT configure a stack neither a Etherchanne on the switches which are connected to the NICs (via cross).

    We only want to have simpe LB and failover in case of a path outage. Will this work properly (send and receive)???? My colleagues told me that will not work as long as the switches are not stacked?? Why should it not? From my understanding the VM will switch over to another vNIC which is connected to another NIC which again is connected to an other standalone switch.

    Your colleague is wrong, you do not need any special configuration for the port ID based load balancing mechanism. This will work just fine with failover, due to the static simple mapping explained above. It will be just like plugging a system from one switch into the other (in the same broadcast domain).

    When do wenn need to configure "Based on MAC" or "IP  HASH"? Well i know that when you configure "IP Hast" then you need to setup an Etherchannel between the Host and the Switch and set the policy to IP Hash otherwise you dont have additional bandwith.

    The MAC based forwarding is also static, but based on the source MAC and has nothing to do with etherchannel/LACP. See my explanation and the article above.

    Why would we need to set adapters to standby and not active? Is that used to deny issues with loops in a not spanning-tree environment?

    vSwitches won't form loops unless you do some really ugly things in a VM with multiple vNICs and configure some birding within that VM OS. The active/standby settings are basically just primary/secondary prioritization in case you want traffic of a particular port group to go through one specific uplink, unless a failover occurs.

    How do you configure the network (amount of NICs, vSwitches, portgroup assignment, vmotion etc. and the switches)?

    It depends. On a lot of factors.

    To sum it up:

    - segregate your network with VLANs as if in a completely physical setup

    - put vMotion on a private non-routed VLAN with a dedicated physical uplink (or active/standby team settings that make sure vMotion is not shared with other traffic except in a failover case)

    - use as many physical uplinks as you need for bandwidth

    - for IP-storage (NFS, iSCSI) or FT, use dedicated physical uplinks as well



  • 3.  RE: ESXi network configuration?!

    Posted Aug 03, 2015 01:09 PM

    Dear MKguy

    Your explanation is just awesome I really do value the effort you spend writing this for me! Thanks.

    Well, could you give me that esx top network statement you mentioned? I would like to use that statement for some customer environments.

    Further just to be 100% sure:

    Based on your outline it is correct that when i have a vSwitch with 2 vNIC and 2 VMs running on a portgroup mapped to both of these vNICS / NICs they are able to send and receive indepent from each other and parallel by using both vNICs assigned to the vSwitch? And there is not only one NIC active and receive and sends data for both VMs?

    Environment example


    vSwitch 0

    Portgroup VMs

    vNIC 0 and 4

    VM1

    NIC: 1x

    Portgroup VMs

    VM2

    NIC: 1x

    Portgroups VMs

    Thank you



  • 4.  RE: ESXi network configuration?!

    Posted Aug 03, 2015 01:31 PM

    You're welcome.

    Well, could you give me that esx top network statement you mentioned? I would like to use that statement for some customer environments.

    In esxtop, just press "n" to enter the network view.

    Based on your outline it is correct that when i have a vSwitch with 2 vNIC and 2 VMs running on a portgroup mapped to both of these vNICS / NICs they are able to send and receive indepent from each other and parallel by using both vNICs assigned to the vSwitch? And there is not only one NIC active and receive and sends data for both VMs?

    Discalimer: I use(d) the term "vNIC" exclusively to refer to VM's virtual network interfaces or virtual vmkernel ports of a host. I refer to the physical uplink NICs of a vSwitch as "(physical) uplink", "vmnic" (VMware term used in the GUIs etc) or "physical NIC". This distinction is important.

    If I understand your example correctly with 2 vNICs (say vNIC0, vNIC1; the number of VMs does not matter) and 2 active physical uplinks (say vmnic0, vmnic1) with the port-ID based teaming policy:

    There is no guarantee that vNIC0 and vNIC1 will each be mapped to a dedicated uplink. Depending on the vNIC port-ID of which you have no control, they may both use vmnic0 at the same time, or vNIC0 may be mapped to vmnic0 while vNIC1 uses vmnic1 etc.

    In any case, any single vNIC (and subsequently all its transmitted frames, no matter the source MAC) will only use a single physical uplink at any given point in time. Consequently it will only receive frames from the physical network switches by that physical NIC. So no MAC-flapping issues that could arise if you would try to balance traffic of one vNIC to multiple physical uplinks without something like etherchannel/LACP.

    It goes without saying that the traffic distribution should be more balanced the more vNICs/VMs are attached to a vSwitch, but at the end of the day the mapping of virtual to physical NICs entirely depends on an arbitrary value you have no real control of, the port-ID.

    I hope this clears it up.