Tricky networking question

[HendersonD]

Instead of tape backup we use an Exagrid disk based backup solution and Commvault software to manage the backups. Commvault is installed on two servers running under ESXi 4.1. One server manages the process and is called the Commserv and the other server handles the actual data flow and is called the Media Agent. We have two nics in the Media Agent, one attached to a Cisco switch and visible to the public network and one attached directly to the Exagrid. Backup data flows into the Media Agent via the public network and flows out of the Media Agent to the Exagrid via a private network. The diagram below shows the current setup.

The networking diagram in ESXi is also shown below showing the one nic all by itself that attaches the ESX host directly to the Exagrid box

The suggestion is to get rid of this private network (IPs of 192.168.30.x) and pull the one nic into the either the storage network or public network (VM Network). I can do that but my thought is I still want the traffic going from the Media Agent to the Exagrid to be in a different vlan than either my VMs live in or my storage uses. If I pull the nic into one of these two vSwitches I have two questions:

1. Should I add this nice to my VM Network or Storage Network?
2. Can I just use vlan tagging and create another vlan for the traffic going from my Media Agent to the Exagrid?

Either way, I think I will end up with different physical hookups as shown in the diagram below where the traffic is going from the Media Agent to the Exagrid via the switch.

[logiboy123]

I'd be more tempted to take vmnic2 and put it in vSwitch0, thus giving your service console network redundancy.

To be honest, unless you are using non standard load balancing techniques then you will not see any improvement in throughput by adding vmnic2 to either vSwitch1 or vSwitch2. I suggest investingation of all the different load balancing techniques to determine if you have the right equipment to make your solution really efficient and smart. See this whitepaper as a starter:  http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

Apparently with a smart enough switch you can use IP Hash to create effective throughput across multiple NICs.

In regards to your storage network, you may see some improvement if you review the Multipath IO Vendor Agnostic walkthrough;

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html

After moving vmnic2 to one of your other vSwitches (I'm hoping you will move it to vSwitch0) then yes, simply create another VM Network port group on vSwitch2, assign it a differnt IP range and VLAN and you are good to go.

[HendersonD]

Right now I am saturating both nics during a backup, the one nic with incoming traffic to the Media Agent and the one nic with outgoing traffice to my Exagrid. My backup windows are fine though so while having more bandwidth would be great, it is not a show stopper right now. I am hoping about a year from now to do a network upgrade and jump to 10GB ethernet.

I know that I have no nic redundance in my Service Console right now. With only 6 nics I had to make some choices.

The other thing I thought about was to do exactly as you suggest, put this nic into vSwitch0 and get some redundancy for my Service Console. I could then take vSwitch1 and through vlan tagging, and the accompanying work on the physical switch to create trunk ports, setup this vSwitch to handle traffic from two different vlans, vlan126 (storage) and vlan128(just the Exagrid). I could then setup NIC Teaming in vSwitch1 configuration such that:

vmnic0 was active for vlan126 and standby for vlan128

vmnic1 was active for vlan128 and standby for vlan126

As long as a nic does not fail in vSwitch1 then traffic should come in to the Media Agent via vmnic0 and go out vmnic1 to the Exagrid. The main thing is to use 2 nics for the media agent. If I only use 1 nic then incoming backup traffic from the network and outgoing backup traffic to the Exagrid need to share the same 1GB connection. With 2 nics, traffic comes in one and out the other.

[mjpagan]

I might think about adding the nics from vswitch2 and vswitch3 to your vswitch0, configuring etherchanneling on the 3750 and IP HASH on vswitch0.  You would also configure the etherchannel (Cisco calls it a channelgroup I think) as a trunk port that way you can pass all of your VLANs to your VMware port groups.

This has the benefit of getting you service console redundancy, more throughput for your backups (on four nics) and the ability to separate your networks with VLANs.

I would definietly look at upgrading to to 10G when you can because the 5TB Exagrid is the "lowest" model that will support a 10G cards.

Mike P

VCP3/VCP4 MCSE

[HendersonD]

Mike,

Thanks for the reply, you have me thinking about two different things now. The exagrid box does have two nics that can be used for backup purposes. The tricky part is Commvault's Media agent. I had the Commvalt engineer in last week and we discussed two different ways to utilize more than one nic for incoming and outgoing traffic to the media agent, thus increasing throughput. Since I am meeting my backup windows right now, I decided to forgo the work necessary to increase backup throughput beyond my current 1GB incoming and 1GB outgoing. Of course all of this is in light on jumping to 10GB ethernet in a year.

The networking diagram I provided was for my ESXi server running in my disaster recovery site. I also have 4 ESXi servers running in my production site. Taking your idea to the extreme, could all nics (6 in my case in production) be hooked to one vSwitch with this vSwitch handling Console, VM, and Storage traffic? Vlan tagging for the vSwitch along with etherchannel and trunk ports on the physical switch side to separate traffic and provide higher throughputs.

Dave

[mjpagan]

With an Exagrid you when you create your shares they are available on all IPs so that when you configure your backup software's backup to disk location you can manually load balance the shares by configuring them with different IPs.  For example if you had two IPs on your Exagrid 1.1.1.1 and 1.1.1.2 you could configure your B2D folders \\1.1.1.1\folder1 and \\1.1.1.2\folder2, that way simultaneous backup jobs would use different NICs.  This works in the Backup Exec world but I don't know if it translates to CommVault, so your mileage may vary.

I think your plan is OK except for adding storage to the same vswitch. Typically storage networks have jumbo frames configured and since that needs to be end to end (from your host to your storage) the inclusion into one vswtich would stop you from configuring jumbo frames.  I would take an separate pNIC and use that for your storage network.

Sorry, it took so long to get reply.  I was on vacation and only recently started looking at email again

Mike P

VCP3/VCP4 MCSE

[HendersonD]

Mike,

If you look at my networking diagram, my first thought was that I really need redundancy on my service console. This can be accomplished by taking vmnic2, which is dedicated to the Exagrid, and pull it into vSwitch0. Unless I purchase another nic card, that leaves me with how to send data to the Exagrid.

If I take vSwitch1, which is now just NFS traffic, I could also use it for the Exagrid by using vlan tagging. NFS traffic on vlan 126 and Exagrid on vlan 128. All of my storage traffic already has Jumbo Frames enabled. In other words, I have Jumbo Frames enabled on my Netapp filer, the Cisco 3750 switch, vSwitch1, and the VMKernel port. I think that the nics in the Exagrid can do Jumbo Frames.

The question becomes the "end to end" thing for Jumbo Frames. All traffic going to the Exagrid during backup goes through the Commvault Media agent which is one of my VMs. I want to make sure that I have two nics involved here, one nic for incoming traffic to the Media Agent from the switch and one nic from the media agent to the Exagrid (via the switch). This gives me 1GB of bandwidth coming in and 1GB going out.

Some of this traffic starts on my Netapp filer (NDMP dumps) but I think for backup purposes it goes out the management port (vlan 125) which I do not have Jumbo Frames enabled. I also have some stand alone servers (vlan 125 again) so trafic coming into the Media Agent would not be via Jumbo Frames but the traffic going from the Media Agent to the Exagrid would be Jumbo Frames. I am guessing this would not work.

Dave

[HendersonD]

I decided to put another nic card in my ESXi host which gave me another connection for my service console as well as two connections from the Commvault Media Agent to the Exagrid. Commvault is setup to take advantage of both of these connections.