Broadcom NetXtreme II 5709 quad-port NIC issues

[FrostyatCBM]

Environment:

-- ESXi v4.1 running on:

---- 3 x Dell R710 servers (48GB RAM) with shared SAS storage (no iSCSI) each with:

------ 4 x Broadcom PowerEdge R710 BCM5709 Gigabit Ethernet ports on motherboard (working fine)

------2 x Broadcom NetXtreme II BCM5709 1000Base-T on 2-port card (working fine)

------4 x Broadcom NetXtreme II BCM5709 1000Base-T on 4-port card (problems!)

Have installed ESXi 4.1 fresh to each server.  When configuring the Management Network I noticed that:

-- server ESX1 correctly shows 10 NICs

-- server ESX2 incorrectly shows 8 NICs (2 on the 4-port card are "missing")

-- server ESX3 incorrectly shows 6 NICs (all 4 on the 4-port card are "missing")

I've downloaded from VMware the latest Broadcom 5709 drivers for ESXi v4.1 and installed them via the vSphere CLI (successfully) and rebooted.  No difference.

I've swapped the cards around in the server and the 'faults' follow the cards.

So 2 out of 3 brand new (I hope!) cards which were purchased in the one order are apparently faulty.  Bad batch maybe?

Should I go with trying to replace these cards with the same model of Broadcom NetXtreme II 5709 4-port cards ... or should I try another vendor (e.g. an Intel 4-port card)?

Suggestions welcome.

[James Bowling]

It is possible that you got a bad batch.  Have you tried placing the cards into a physical Windows or Linux box to see if all the ports show up?  My personal opinion is to use Intel cards.  I have always had good luck with them.

[FrostyatCBM]

I haven't tried with a Windows server; I suppose I could install Windows 2003 or 2008 on one of the R710's but its a lot of effort to go to just for troubleshooting.

I think I might talk to my Dell account manager and ask him to swap the cards for Intel quad-port instead.

[James Bowling]

It would be.  I was thinking you might have a physical sitting around that already had an install on it.  Although, I think the option of having them replaced is best.

[idle-jam]

i would do the same as you to get the card swap. if you have a vmware support i would log a call and have that as an additional point to force your account manager to change the card for you.

[FrostyatCBM]

The efficiency with which the Dell support team sourced and delivered the replacement Broadcom cards was excellent.  They arrived this morning at 8:00am and I have installed them into our R710 servers.  All the ports are now being enumerated on all the servers (i.e. I can now see 4 motherboard NICs, 4 Broadcom 4-port card NICs, and 2 Broadcom 2-port card NIC, for a total of 10 NICs).

So 2 of my previous 3 "brand new" 4-port cards were clearly faulty, and I frankly have some doubts about whether they were truly new or not.  One thing I do know for sure now ... the cards that I've got (based on the serial numbers on the chipsets) are much older stock than what I had before.

The not-so-good news is that the ports are not being enumerated identically in all the servers:

ESX1
-- Broadcom mboard NICs: vmnic0, vmnic1, vmnic2, vmnic3
-- Broadcom 4-port NICs: vmnic4, vmnic5, vmnic8, vmnic9
-- Broadcom 2-port NICs: vmnic6, vmnic7

ESX2
-- Broadcom mboard NICs: vmnic0, vmnic1, vmnic2, vmnic3
-- Broadcom 4-port NICs: vmnic6, vmnic7, vmnic8, vmnic9
-- Broadcom 2-port NICs: vmnic4, vmnic5

ESX3
-- Broadcom mboard NICs: vmnic0, vmnic1, vmnic2, vmnic3
-- Broadcom 4-port NICs: vmnic4, vmnic5, vmnic6, vmnic7
-- Broadcom 2-port NICs: vmnic8, vmnic9

In other words, 3 different configuration results, despite everything being supposedly identical:
-- 3 identical Dell R710 servers
-- 3 identical physical configurations (Broadcom and all other PCI cards are in identical PCI slots)
-- 3 identical software installs of ESXi 4.1

I am at my wits end here.  I cannot put this gear into production because I am not confident that the pNICs will be mapped to the same vmNICs upon a reboot (I intend to test this next week to see if they move around at all ... hopefully my fears are unrealised in that regard).

It is not clear to me at this point whether this is a Broadcom card issue, and ESXi 4.1 configuration issue, or a Dell R710 server issue.  I will take this up with professional support and try to get a clear read on that.

Any suggestions?  e.g. is there an ESXi 4.1 setting which controls the sequence of NIC enumeration?  I thought it would be based on something like the sequence of the PCI buses, but apparently not?

[Mauro Bonder]

have a patch to apply with corretions in broadcom driver BNX2X. Check more informations in: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1029398

Or download in http://www.vmware.com/patch/s3portal.portal?_nfpb=true&_windowLabel=SearchPatch&SearchPatch_actionOverride=%2Fportlets%2Fpatchupdate%2FfindPatch&_pageLabel=s3portal_pages_downloadPatch_page

[FrostyatCBM]

Thanks MauroBonder ... unfortunately I am already running that new driver version ... the problem seems to be at a deeper level, where the pNICs are being first detected by the server and/or by ESXi 4.1

[FrostyatCBM]

Card swap for Intel 4-port NICs is expected to happen in a couple of days.  In the meantime I have done some more testing:

1.  shutdown all 3 servers, removed the Broadcom 2-port cards, rebooted ... after this all the 4-port card vmNICs were "in sequence" but retained their vmNIC number ... i.e. if the server had assigned them vmNIC4/5/8/9 previously, then they remained like this ... not an unexpected result.

2.  reinstalled ESXi 4.1 on all 3 servers (with the 2-port NICs still removed) ... after this all the 4-port card vmNICs were both "in sequence" and "sequential" ... i.e. all 3 servers now reported them as being vmNIC4/5/6/7 which is also expected.

I am pretty confident though that if I re-introduce the 2-port NICs and reinstall ESXi 4.1 then the sequencing of the vmNIC numbers will be out of whack again.  So I think I will wait for the Intel cards to arrive and will then do some more experimenting, to see whether they do or do not behave themselves.

Here's a question:  when installing NICs to servers, is it necessaryto introduce them one at a time, so that the vmNIC numbers are added sequentially?

What do you do if you buy a brand new server that has (for example) 12 NICs (4 on the motherboard, plus 2 x 4-port extra cards)?  Are you supposed to do a build with just one of the 4-port cards installed, then later add the extra 4-port card?

[James Bowling]

I would assume that this is just a matter of preference. But I have not actually had this issue. When I install cards they normally fall in-line correctly.

James Bowling

[a_p_]

Actually the NICs should be picked up in the order of the PCI ID's as described in e.g. http://kb.vmware.com/kb/1002315

Maybe it is helps to run a "Reset configuration data" in the server's BIOS after plugging in the NICs.

The really weird thing is the numbering in ESX1, where even the ports on one NIC are not in an ascending order!?

André

[FrostyatCBM]

OK, I think I have this fixed now ... or at least, I think I understand what happened.  I re-read the document:

http://kb.vmware.com/kb/1002315

and realised that I had not really properly comprehended the importance of "During the installation of an ESX host ..."  when referring to how NICs are enumerated.  Here's what I think happened to me:

(1) installed 3 cards in my 3 hosts (n.b. 2 of the cards were faulty)

(2) installed ESXi v4.1 (because of the faulty cards, the enumeration of the NICs went haywire)

(3) replaced the faulty cards and rebooted

(4) still didn't get the right sequence of vmNICs ... reason? ... I didn't reinstall ESXi at this point and was relying on the previous install where the NIC enumeration didn't work right ... hence ESXi retained its 'dodgy' vmNIC numbers because the total number and type of NICs in the server remained the same (!)

(5) removed the 2-port cards, leaving in place 8 NICs total

(6) still didn't get the right sequence of vmNICs ... reason? ... I didn't reinstall ESXi at this point and was still relying on the previous install where the NIC enumeration didn't work right ... so ESXi retained its 'dodgy' vmNIC numbers (I presume that ESXi maps these, based on something like MAC address --> vmNIC ID)

(7) reinstalled ESXi v4.1 (now with just 4 motherboard ports perserver and an additional 4-port card per server)

(8) now I had a properly sequenced set of vmNICs ... 0-3 for the motherboard ... 4-7 for the 4-port card ... because all the cards were functioning properly and ESXi enumerated them correctly during the install

(9) put the extra 2-port cards back in and rebooted

(10) all vmNICs are still enumerating correctly and in the right sequence ... my 2-port cards are showing as vmNIC8 and vmNIC9 in all 3 servers.

I would bet that if I did a complete reinstall of ESXi 4.1 at this point, all the servers will correctly enumerate the NICs in exactly the same sequence on each server.

So the "take away" from this, for me, is "If you install ESXi on a server with a faulty NIC, don't just replace the NIC ...reinstall ESXi from scratch".  Maybe I should have known to do that in the beginning, but I guess "you don't know what you don't know".

Now I know ... :smileywink: