PXE boot process does not work on LACP etherchannel.

[M0rph77]

Hello guys,

Question; I'm trying to implement a scripted install of ESXi4.1 via PXE.

The host has a basic 6 x 1GB nic design on the back, segmented in 3 static LACP etherchannels with all are dot 1Q vlan trunks cross stacked on 2 3750G catalyst switches. It's just a standard config based on the cisco/vmware best practices doc.

Now, the issue I'm having is in the pxe boot process. The pxe agent gets an ip address from the dhcp server, but the tftp operations halts ending in a pxe timeout error. When I break the port channel, the process does work.

The settings;

Cisco 3750;

interface port-channel x

switchport trunk encapsulation dot1q

switchport trunk native vlan 999

switchport nonegotiate

spanning-tree portfast trunk

!

interface gix/x/x

..

channel-group x mode on

...

!

int vlan 999

ip address xxx

ip helper-address xxx

The pxe process uses the native vlan to find the dhcp server during bootstrap.

Again; with the etherchannel configured, pxe does not work, without it does.

Obviously this issue is widely know, but I can't seem to find much information about it.

Is this a known issue?

How do you have your esx scripted install through pxe implemented?

Thanks,

[titaniumlegs]

Bare Ethernet cards don't have any kind of link aggregation capability.  That's normally done by something in the operating system.  Link aggr with just firmware on the cards would require the cards to talk to each other, like through the PCI bus.  Not likely.

Are DHCP and TFTP coming from the same server?

Two things I can think of you can try.

1.  Leave LACP enabled on the switch, but unplug the other NIC.  Not a good solution long term, but may tell you if the problem is LACP on the port, or if the switch might be trying to reply on the other link of the channel, to the card not doing PXE.

2.  Disable PXE on the NIC currently doing PXE and enable on the other NIC.

Another thing that wasn't spelled out, is when you disabled LACP, was the port still on VLAN 999?

Not holding high hope these will resolve anything, but they may shed some light.

[M0rph77]

So what you are saying is for LACP to work on a link it needs to be configured end to end.. No magic tricks to get this working with 1 nic in the channel (in the pxe pre boot situation)..

Thinking about this; regardig pxe install as a requirement I have 2 choices;

-1 (ugly); add "the removal of the etherchannel" in the installation procedure pre pxe install.

-2; don't use etherchannel for the manament and vmotion nics, use virtualport id loadbalancing and use overriding active / standby nic policies for management network portgroup and the inverse for vmotion (to prevent vmotion saturating the link and thus cut off management traffic with ha isolation response kicking in)

Any other design scenarios possible?

[titaniumlegs]

The problem is, or one problem anyway, that both NICs are connected and have a link, but the drivers and SW to make them a NIC team are not running yet.  The switch sees both ports up and may attempt to send traffic on both, depending on load balancing algorithms.  (I could be wrong on the internals - I've never dissected LACP behaviour to this level before.)  This is why I asked about DHCP and TFTP from different servers - if the switch is balancing on IP hash, then one server's traffic may resolve to one port and the other server to the other (which isn't doing PXE).

You're right that #1 is ugly.  It also won't work, or will be very difficult to automate for people who do stateless / autodeploy.

In #2, you might not have to override failover order.  MAC or port hash depends on the MAC or port (no kidding, right?!).  With two NICs, essentially the even MACs or ports go on one, and the odd on the other, until the number of ports changes either administratively or due to failure.  So, if you can consistently set it up such that the MAC or port (whichever you choose) are one even and one odd, then you're set.  One way to do this might be to edit esx.conf to set specific MAC addresses and reboot.

A trick for virtual port is to check the virtual port ID (esxcfg-info -n or the esxtop network screen show it ), and if it's not even or odd (whichever you want), delete and re-add until you get what you want.  Not pretty, but scriptable.

In vSphere 5 and autodeploy, you can specify MAC addresses as part of a host profile.

Peter

[M0rph77]

vSphere 5 autodeploy, doesnt that also ínvolves pxe?

Regarding the seperation of vmotion and management traffic, I found a couple of good views in the blogosphere;

http://www.yellow-bricks.com/2011/03/22/esxi-management-network-resiliency/

http://pelicanohintsandtips.wordpress.com/2011/06/16/esxi4-1-vmotion-using-incorrect-vmnic/

I think the best design for our environment (2 nics for management and vmotion) is to use seperate vlans for management and vmotion traffic.. and using the following setup

• 2 Portgroups (Management Network and vMotion VMkernel)
  
• 1 vSwitch, 2 vmnics, vmnic0 and vmnic1 connected to different physical switches (no static lacp port channel)
  
• Management network and vmotion in different vlans
  
• Management Network active on vmnic0 and standby on vmnic1
  
• vMotion VMkernel active on vmnic1 and standby on vmnic0
  
• Failback set to Yes
  

I now can use pxe based scripted deployments, and have made the management network resiliant..