ESXi Network design questions and best practice

[virtualizingtheworld]

Hi,

Am currently revising how best to assign network traffic for Management/vMotion/vSAN/VM Traffic when having 6x NIC scenario with same 10GB/s uplink speed

By default the ESXi Management gets connected to a standard vSwitch, is it best practice to leave it like this with a load balancing of Route based on originating virtual port and both network cards as Active?

For vMotion I have been using the vSAN dvSwitch (1 Active and 1 Standby uplink with Route based on originating virtual port) and specifying the other uplink as the primary with NIOC preferring the vSAN traffic in case of NIC failure. Potentially if the ESXi Management uses a dvSwitch I could place vMotion on their uplinks instead while using the same concept of setting the other uplink as primary and NIOC.

As with regards to VM traffic, is it best practice to set LACP active on Physical Switch side and a LAG on vCenter or to not set LACP on physical Switch side and use Route based on physical NIC load? Does this change which Load balance to use as default is Route based on originating virtual port?

Thanks

[VirtPhysical]

Regarding LAG Ports, unless you have a real specific reason to use it, I wouldnt bother. Just adds extra complexity that you may not see much benefit from.

In terms of vMotion networking etc, there is some really good documentation on this. See: Networking Best Practices for vSphere vMotion

You dont need to set the uplinks to Active/Standby though as each vmkernel adapter only uses a single uplink. You can leave these set to Active/Active however i would move vMotion off the vSAN switch and onto your management switch

Or see https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-67-p…  page 38 onwards

[Scott Vessey]

Moderator: Moved to vSphere vNetwork

[MikeStoica]

I'd leave the vSAN traffic alone, nothing else. Create a dvSwitch only for vSAN traffic and you can create another one for Management where you can have the vMotion traffic also.

[virtualizingtheworld]

The vSAN traffic would use only one uplink, the other uplink is there for failover so doing nothing really hence why I assigned vMotion to it.

The other option as you mention is to create a new dvSwitch to be used for Management and vMotion using the same above principle.

Do you see any disadvantages of having the Management vmkernel migrated to a dvSwitch?

Is there any documented process for migrating the Management from a Standard to Distributed Switch?

I am thinking the following, would this cause any downtime?

Create Distributed Switch

Assign Hosts

Assign both Physical Adapters

Create Port Group

Assign Management vmk to Port Group

Migrate any VM's Networking to the newly created Port Group

Thanks

[virtualizingtheworld]

ok so based on the responses the ideal setup should be:

1x dvs (2x Uplinks) for both Management and vMotion in separate VLAN's with 1 Active and 1 Standby uplink (Alternate active uplink)

1x dvs (2x Uplinks) for vSAN. Does this dvs need to be 1x per cluster since each cluster vSAN traffic will be in a different VLAN? (Planning to use LACP/LAG so it diversifies uplink usage)

1x dvs (2x Uplinks) for VM Traffic. This should be just one dvs for all clusters for ease of management (Again planning to use LACP/LAG so it diversifies uplink usage)

The above will result in at least 3x dvs, do you agree that this the best practice/easier for troubleshooting?

If not please suggest alternate (such as by less # of dvs if possible and why preferred)

Thanks

[NathanosBlightc]

It's not needed to create isolated distributed switches for each traffic inside the virtual network, you can separate them (mgmt, vMotion, vSAN & VMs) via changing the failover order for each of dvPortGroup inside single VDS.

However management can be easier while the troubleshooting maybe more complex