10Gb Ethernet Adapters and VM Connectivity Issues

[gabe0819]

I have three HP Proliant DL380 G6 machines, running ESX4.0.0 U1 (all patches, critical and non as of 01/05/10). Inside of each, I have a HP NC522SFP Dual Port 10GbE Server Adapater that I'm trying to use for VM/LAN Traffic. I also have two onboard Broadcom NetXtreme II BCM5709 1000Base-T NICs.

What I'm trying to do is create a vSwitch for VMs that will use the 10Gb NICs as its uplinks to our core swithc and create another vSwitch for SC and VMotion on the onboard 1Gb NICs. The issue I'm running into is that VMs on the '10Gb vSwitch' lose connectivity after a day or two.

I have tried both distributed switches and local vSwitches with the same results. The VM fails on all three of the servers. Here are my local vSwitch settings as well as the EtherChannel setup on the Cisco 6509 (with VSS) side:

vSwitch1 (Port Group is the same, no VLAN tagging)

Number of Ports - 56

Promiscuous Mode - Reject

MAC Address Change - Accept

Forged Transmits - Accept

Traffic Shaping - Disabled

Load Balancing - IP Hash

Network Failure - Link Status Only

Notify Switches - Yes

Failback - Yes

Active Adapters - vmnic4, vmnic5 (the 10Gb NICs)

Standby/Unused Adpaters - None

Switch side port channel

interface Port-channel61

description ESX Server 11 10Gb Port Channel (includes 10Gb 1/1/2 & 2/1/2)

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

spanning-tree portfast edge trunk

interface TenGigabitEthernet1/1/2

description ESX Server 11 10Gb Port 1 (part of Port Channel 61)

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

spanning-tree portfast edge trunk

channel-group 61 mode on

interface TenGigabitEthernet2/1/2

description ESX Server 11 10Gb Port 2 (part of Port Channel 61)

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

spanning-tree portfast edge trunk

channel-group 61 mode on

A couple other things...VMs work fine on the copper/onboard NICs. The NIC card is in the supported hardware list. It follows the VMs on different ESX boxes. If you need more information, please let me know. Thank you in advanced!

[chilow]

Are there any NIC related errors in the vmkernel logs of the ESX servers?

-C

[gabe0819]

Rebooting the ESX server fixes it temporarily. I've rebooted the server, the VM is connecting. I will now be able to pinpoint the issue timewise a bit better. Once I see it go down again, I will scour the logs. Stay tuned.

[kac2]

if you do these commands on your 6509:

show interface Port-channel61

show interface TenGigabitEthernet1/1/2

show interface TenGigabitEthernet2/1/2

do you see any input errors?

is it set at the correct duplex?

[ML1979]

We have the very similar setup, and we went into similar issue. Problem seems to be resolved.

A continuous ping showing "time out" for our VM connect using 10Gb vswitch.

Here is how we troubleshoot it :

Are you using SFP or CX4 10Gb module ? Have you check the effective length of your 10Gb module can handle ?

Our using CX4 on 3Com 4500G, effective length for our module is 3Metres max. Replaced with a shorter cable, problem fixed.

Next, faulty CX4 cable. One of our SAN data link from one of our Controller showing very poor performance compare with the "GOOD" one.

After some test, I found the new BELKIN CX4 cable is bad. Bought few 3Com CX4 cable for replacement, problem resolved.

I hope these help.

[vGuy]

Hello,

we have a similar issue on 10gb NC522SFP with ESX 4.0 u2..the NIC shows as up but becomes inactive in the sense it cannot see any traffic..and also the cdp data is not shown. We see the following errors in vmkernel:

Aug 19 06:38:25 esx1 vmkernel: 58:13:18:15.155 cpu8:4239)WARNING: LinNet: netdev_watchdog: NETDEV WATCHDOG: vmnic2: transmit timed out

Aug 19 06:38:29 esx1 vmkernel: 58:13:18:19.157 cpu2:4232)<4>nx_nic[vmnic2]: Unable to create the tx context, code 17 Error: Timed Out
Aug 19 06:38:29 esx1 vmkernel: 58:13:18:19.157 cpu2:4232)<3>nx_nic[vmnic2]: Warning :tx context not initialized.Reset Not Complete
Aug 19 06:38:29 esx1 vmkernel: 58:13:18:19.157 cpu2:4232)<3>nx_nic[vmnic2]: transmit timeout, NEED to reset.
Aug 19 06:38:29 esx1 vmkernel: 58:13:18:19.157 cpu14:4241)<3>nx_nic: vmnic2 NIC Link is up

is there a bug with this adapter? appriciate any help...

[virtualdavid]

hello,

we also have been running the same config for 1 year now and the environment has never been stable. Port channels are falling from time to time on any server, due to the card issues.

we have upgraded the NICs NC522SFP to the latest firware and latest nx_driver but the problem still occurs.

we have opened a ticket with VMware support but there is no timeframe to resolve this issue.

we are thinking about purchasing other NIC cards and we are trying to find 10GB cards that have no issues. the suport didn't want to advise us on the 10GB cards they don't have issues with, unfortunately...

[vGuy]

Update: I have not seen this issue reappear again after upgrading the firmware on our DL 380G6 servers..