Do I really need vMotion VLAN?

[Titans99]

Hi, my ESX 5 cluster will have four hosts with 8 NIC's each.  Currently I have two physical switches with two physical VLAN's each (switches are trunked).  I was going to use the NIC's in the following way:

• 2 nics for Management/vMotion (vSwitch0) connected to VLAN1 across both switches
• 2 nics for VM network (vSwitch1) connected to VLAN1 across both switches
• 2 nics for DMZ network (vSwitch2) connected to VLAN2 across both switches
• 2 nics unused

My question is... would it benefit me to create a 3rd physical VLAN and use the final two NIC's for dedicated vMotion?  One of the reasons for the question is I'm running low on physical switch ports, but I would have enough to accomadate vMotion if there would be benefit.  Thank you!

[MKguy]

Never, ever put vMotion traffic and anything else, especially not VM traffic on the same VLAN (and/or uplinks). Besides being very bursty, vMotion transfers unencrypted memory content of your live VMs over the network.

This configuration seems most appropriate in your case:

- use an isolated, non-routed private VLAN/subnet for vMotion

- put ESXi management traffic on a separate VLAN/subnet

- you can have these 2 VLANs on the same vSwitch

The VM and DMZ-VM networks should be on their own, dedicated VLANs and vSwitch too. Also, if you're ending up on the same physical switches for these anyways, you might just as well combine the 2 port groups onto one vSwitch too (with 2 pNICs, if that provides sufficient bandwidth for you or 4).

[Titans99]

Thanks for the reply.  I will separate the vMotion traffic, but I think I will need to keep Management and VM network on the same VLAN for now because that is where the physical vCenter server resides.  I can't change the IP of the vCenter server because it doubles as an internal DNS server. 

Attached JPG is what I was thinking.... any thoughts? (ignore the fact the NIC's show disconnected)

I guess I could spin up a new vCenter VM (or P2V the one I have) to separate the two and accomplish getting Management isolated.  I'm always a bit leary of using a vCenter VM for some reason - maybe shouldn't be.

[MKguy]

I will separate the vMotion traffic, but I think I will need to keep Management and VM network on the same VLAN for now because that is where the physical vCenter server resides.  I can't change the IP of the vCenter server because it doubles as an internal DNS server.

You can still put your host management interfaces into a separate VLAN/subnet and leave the vCenter in the VM network. There is no requirement that they must be on the same subnet and there's not any significant traffic going on between vCenter and the hosts it manages.

Your networking setup should work but it looks a bit "clumsy" considering you will put everything on the same physical switches anyways. Is there a reason you're not tagging VLANs on the vSwitch?

Like I said in my other post, I'd combine the management and vMotion VLANs on one vSwitch with 2 uplinks. Set the active/standby failover order for the 2 vMotion vmk NICs accordingly and don't bother with settings for the insignificant management traffic. At least unless you do backups/replication/frequent conversions or such things through that management interface.

[Titans99]

I see.  I would just have to open firewall ports between the vCenter and the host management network.  Shouldn't be a problem.

There's no technical reason I am not VLAN tagging at the vSwitch level.  It's just more lack of network understanding on when and why to do it.  Physical VLAN's are easy for me to make sense of.

When you say "combine the management and vMotion VLANs on one vSwitch with 2 uplinks", are you saying the management and vMotion traffic will be on the same VLAN with the same IP subnet?  (sorry for my lack of comprehension)

By the way, I have free'd up more switch ports by configuring some of the iLO's in to use the "shared network port" feature, so I have enough to comfortably accomadate all 32 NIC's (8 per server).

[MKguy]

When you say "combine the management and vMotion VLANs on one vSwitch with 2 uplinks", are you saying the management and vMotion traffic will be on the same VLAN with the same IP subnet?  (sorry for my lack of comprehension)

No, I mean you should configure the physical switch ports this vSwitch is connected to with ordinary 802.1q VLAN tagging for your separate vMotion and management network. Then you assign these 2 VLAN IDs to the respective vmkernel interfaces.

The ethernet frames will arrive with VLAN tags on the vSwitch, which will hand it to the vmkernel ports accordingly or will add the correct VLAN tag when transmitting frames. This preserves full layer 2 isolation between the 2 VLANs.

Until now your physical switch handled the whole VLAN tagging and un-tagging, you merely bridged the connections carrying a single VLAN to your host ("External Switch Tagging" in VMware lingo). This requires different vSwitches if you want to separate traffic.

I suggest you read the following to achieve a better understanding of the ESXi networking side:

http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

http://kb.vmware.com/kb/1003806

http://kb.vmware.com/kb/1004074

[Titans99]

Thanks, I will review the documents and/or maybe hire a network consultant for a couple hours.  I actually do understand what you are suggesting and the concept, but I'm not exactly sure how to configure "ordinary 802.1q VLAN tagging for the separate vMotion and management network" on the physical switch side.  As you can see I usually just create a VLAN on the physical switch containing specific physical ports and never tag on the vSwitch side, but I'm sure it's fairly simple.

I'm also not sure how I would configure the additional interface on the firewall to allow access to the management network VLAN (i.e. vCenter to Hosts) using the 802.1q technique, but probably outside the scope of this thread.  I assume it would have to be tagged as well somehow.

One final clarification if you don't mind.  In your scenario, would the managment and vMotion IP's be on different IP subnets?

Thank you kindly for your time.

[MKguy]

One final clarification if you don't mind.  In your scenario, would the managment and vMotion IP's be on different IP subnets?

Yes. Generally, separate VLANs should always be associated with distinctly separate IP-subnets. And remember that the vMotion subnet does not need to be routed at all, it can be any private IP address space as long as it doesn't overlap with other vmkernel ports for management for example.

Glad if I was of help.

[Titans99]

Ok, I think I have it.  I took a few minutes to draw it out so maybe it will help others later.  See attached - on the right track?

[MKguy]

Yea, looks ok to me.

[Titans99]

Thank you for your patience!