VMware Ethernet with Cisco EtherChannel and Trunking Question

[DITGUY2012]

Hello,

Trying to get our etherchannels working correctly with our network admin. Here's our config with a couple of questions.

1. Since this is creating the trunk between switch and host directly, should I use spanning-tree portfast edge trunk ?

2. When do we use no ip address and when not to for this setup? I see examples online showing with and without it.

3. In some articles it describes being in access mode while in others it says dot1q trunk mode. See here for access mode. Admin thinks he should use dot1q.

4. Etherchannel is fully support and preferred spreading links across cards, right? Not all links on same card?

5. Does the order matter for changing load balancing algorithm in vmware first or doing cisco work first? I did vmware first. (IP-hash)

This is on a 6500 series switch with 4 cards/blades.

!
interface Port-channel200
 switchport
 switchport access vlan 81
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 69,81,172,896
 switchport mode trunk
 switchport nonegotiate
!
interface GigabitEthernet2/35
 switchport
 switchport access vlan 81
 switchport trunk allowed vlan 69,81,172,896
 switchport mode trunk
 switchport nonegotiate
 spanning-tree portfast edge
 channel-group 200 mode on
!
interface GigabitEthernet8/10
 switchport
 switchport access vlan 81
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 69,81,172,896
 switchport mode trunk
 switchport nonegotiate
 spanning-tree portfast edge
 channel-group 200 mode on

[MKguy]

You and your network admin should follow the instructions described in these KB articles and not any others if you're unsure:

VMware KB: Host requirements for link aggregation for ESX and ESXi

VMware KB: Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/H…

For LACP deployments (only with 5.1 and dvSwitches):

VMware KB: Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/H…

You would need dot1q trunk mode if you're tagging multiple VLANs across the links (which is common for ESX(i) hosts with multiple networks on the same NICs).

[HeathReynolds]

1. Yes, edge trunk. This is a best practice for cSphere HA.

2. You are fine with the Switchport command

3. yep, dot1q to carry multiple VLANs.

4. Yes, works fine across cards (of similar types).

5. no, the algorithm is independent on each end.

[DITGUY2012]

Thanks you two. It's odd. When I tried setting some of the interfaces into trunk mode it said I couldn't if the encapsulation was in auto mode. Those few interfaces only let me set it in trunk mode when I specified dot1q encapsulation explicitly first. For all the other interfaces trunk mode setting worked fine. It's as if it defaults to dot1q so doesn't require it explicitly but if you ever changed off of it, going forward you'd have to specify it manually first.

MKguy: We tried following that article but there were a few issues/questions.

1. The use of "no ip address" was not seen in many vmware discussions. Trying to understand that one better, although HeathReynolds says we don't need it. Case in point. Also it says to use access mode, yet everywhere else says trunk mode. It also doesn't mention edge portfast setting yet other vmware articles do.

[Josh26]

DITGUY2012 wrote:

1. The use of "no ip address" was not seen in many vmware discussions. Trying to understand that one better, although HeathReynolds says we don't need it. Case in point. Also it says to use access mode, yet everywhere else says trunk mode. It also doesn't mention edge portfast setting yet other vmware articles do.

You really can't talk about the "right" mode because it's dependant on your situation. If you only had a single VLAN running down a link you might elect to use access mode. This is particularly common where a NIC is dedicated to iSCSI. It's worth considering that the crash dump logger can't use a tagged VLAN so consider that.

Generally if you happen to want VMs running on multiple VLANs accessed over one NIC, then it's a trunk mode port.

Just think of it as connecting a switch to a switch. What's the "right" way? Trunk mode in general. What if your other switch is a "dumb" switch sitting on someone's desktop? Then that uplink would probably be an access port.

ESXi doesn't negotiate spanning tree so it should be configured as an edge on your switch. If you are running classic STP this may just look like "spanning-tree portfast" and should probably be accompanied by bpduguard.

The requirement for "no ip address" would depend heavily on whether your switch defaults to wanting to route on that port, in which case you may also want the "switchport" command. Because this is hardware dependant there can't be a "right" answer.

[DITGUY2012]

Wow Josh. If only you had written the documentation. That's the trouble with it. People like me look for definitive instructions because we're not 24x7 network guys. Great documentation would layout the reasons to go this way or that like you just did. That being said, here's my summary based on this discussion.

1. We have multiple VLANs running down 3 links on the etherchannel. Thus we should use trunk mode since it's not a single one.

2. We have another etherchannel with just 1 vlan (vmotion), but across two links. that should be access mode. Or is it still trunk because it's multiple links?

3. I'm not sure what crash dump logger is nor how it applies to my situation.

4. I've seen the bpduguard setting before what is the exact syntax to use for us? 6500 series.

5. I'm not sure if our switch would want to route on this port. How do I tell? Right now I have switchport setting in there.

Thanks!    

[Josh26]

DITGUY2012 wrote:

Wow Josh. If only you had written the documentation. That's the trouble with it. People like me look for definitive instructions because we're not 24x7 network guys. Great documentation would layout the reasons to go this way or that like you just did. That being said, here's my summary based on this discussion.

1. We have multiple VLANs running down 3 links on the etherchannel. Thus we should use trunk mode since it's not a single one.

2. We have another etherchannel with just 1 vlan (vmotion), but across two links. that should be access mode. Or is it still trunk because it's multiple links?

3. I'm not sure what crash dump logger is nor how it applies to my situation.

4. I've seen the bpduguard setting before what is the exact syntax to use for us? 6500 series.

5. I'm not sure if our switch would want to route on this port. How do I tell? Right now I have switchport setting in there.

Thanks!   

No problems :smileyhappy: For reference, the other documentation issue is one of language. On HP networking hardware, "trunk" is their word for etherchannel, and has nothing to do with the "access vs trunk" port debate, which they refer to as "tagged vs untagged" :smileyhappy:

1. Yes

2. You could certainly make this an access port.

3. If your server ever starts to have a problem with crashing, VMware may want a copy of the crash log in order to debug it. You can see an article on setting it up here: Setting up the ESXi 5.0 Dump Collector | VMware vSphere Blog - VMware Blogs.

4. Sorry I'm not familiar with that model

5. I believe if the switchport setting is there and connectivity seems to work you can consider yourself safe

[DITGUY2012]

Some overdue responses:

@Josh - thanks for the help. I found out the bpdufilter setting. We have local disks on each host. Do we still need dump collector? we don't use auto deploy (I think).

@MKGuy - still don't know if we should use no ip address. examples in the docs and online are mixed when showing config examples for setting etherchannels up with vmware and citrix.

@all - We put a VM on the port-group in one of the distributed switches and gave it an IP in the subnet for the VLAN associated with that PG. Still can't ping it though and it can't ping the gateway for that VLAN. Any thoughts on troubleshooting? Here are the config of the ports on each host in that etherchannel:

interface GigabitEthernet8/9

switchport

switchport access vlan 81

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 81,896

switchport mode trunk

switchport nonegotiate

speed 1000

duplex full

spanning-tree portfast edge

spanning-tree bpdufilter enable

channel-group 200 mode on

end

!

interface Port-channel200

switchport

switchport access vlan 81

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 81,896

switchport mode trunk

switchport nonegotiate

spanning-tree portfast edge trunk

!