We have a 4GB Brocade switch in our IBM Bladecentre with 4 trunked ISLs going to an 8GB Brocade switch which acts as our core SAN switch. Recently I've been seeing large numbers of Time BB_Credit Zero on all 4 of the ISL ports (on both switches) in the range of 15-50% of total frames. After doing some poking around, I found that the number of buffer credits assigned to the ports, as well as the link distance, is different on the two switches:
User Port Lx Max/Resv Buffer Needed Link Remaining
Port Type Mode Buffers Usage Buffers Distance Buffers
---- ---- ---- ------- ------ ------- --------- ----------
16 E - 8 17 17 <2km
17 E - 8 17 17 <2km
18 E - 8 17 17 <2km
19 E - 8 17 17 <2km 76
0 E - 8 26 26 2km
1 E - 8 26 26 2km
2 E - 8 26 26 2km
3 E - 8 26 26 2km
I don't know a great deal about buffer credits, but this seems suspicious to me. Could this mismatch be causing large BB_Credit Zero's? Is it advisable to up the buffer credits to be 26 on both switches?
That may depend on the FOS and ASIC type. Which is the FOS release used? And what are the switch models?
The Bladecenter switch is a Brocade 4020 Embedded Switch running FOS v6.2.2d, ASIC type Goldeneye. The core switch is a Brocade 6510 Switch running FOS v7.0.0c, ASIC type Condor3.
The first obvious question is what is the distance between the BC and the core switch? If they are within the DC the default BB credit assignment is correct. Increasing the BB credits will only attempt to mask the true problem. Most likely you have a high latency / slow drain device that is causing this issue. I suggest you review the Brocade Fabric Resiliency paper: http://www.brocadechina.com/download/Fabric_Resiliency_Best_Practices_en.pdf.
Sebastian also has a good description of this problem here: http://community.brocade.com/docs/DOC-2312
Yes, these switches are both in the same data center, probably about 20m of fibre between them. Thanks for the links, I will read through them.
Is it normal for two switches to see different link distances like this? It seems strange to me.
The amount of BB credits reserved for a port depends on the ASIC, switch type, FOS code. so we can see different number of BB credits in both ends. Since BB credits affect only to one direction (I tell you mine, you tell me yours), switchs with different values of BB credits should get along.