Brocade Fibre Channel Networking Community

Expand all | Collapse all

Poor performance and latency issues

  • 1.  Poor performance and latency issues

    Posted 05-08-2019 12:34 AM

    Hello,

     

    I have eight Brocade 6510 SAN switches, each four are stacked. Stachs are isolated. I ran the SAN health audit, and got two reports,  one for each stack. Now, I have concern about the following and I need you please to clarify and tell me what should I be doing about this.

    1. What's the measurement unit used for counting? so when the number has a letter next to it, like "m", or "k", is this equavilent to "Million" and "K" for "Thousand", or "Giga and Mega" maybe?
    2. some ports shows high values for signal lost count, does htis mean that the physical connection has issues? (SFP, cables).
    3. Concerning the Class3 discarded frames", what does this exactely mean?
    4. The interface of switches shows latency warning s for the ISL connections, the SFPs used are 16G all. So, what should I do regarding this please??

    Thank you,


    #BrocadeFibreChannelNetworkingCommunity


  • 2.  Latency issues between 6510 models

    Posted 05-08-2019 01:42 AM

    Hello,

     

    I'm working on optimizing our SAN fabric, I've found tons of latency warnings on the ISL ports, and I've also realized that the BB credit on ISL ports is FC: 8 only, I think it should be higher thatn this. How could I increase it, and waht would you recommend for me please?

     

    I'm not an advanced expert with these thjings, is there a good training course that I can take?


    #BrocadeFibreChannelNetworkingCommunity


  • 3.  Re: Latency issues between 6510 models

    Posted 05-08-2019 10:23 AM

    @Mohamed Roushdy 

     

    I merged both Thread in the single one, since handle the same issue/question.

     

    here in the community are a lot of CRC Thread and Buffer Credit.

     

    the Buffer credit value is 8 by default.

     

    about the issue with the latency,

     

    with the command sfpshow -operands ( see command reference manuals )

     

    check if you have such SFP begin with follow Serial Number

     

    HAF618xxxx


    #BrocadeFibreChannelNetworkingCommunity


  • 4.  Re: Latency issues between 6510 models

    Posted 05-09-2019 01:21 AM
    Hi,

    I've checked all eight switches I have, and I have the following S/N only:
    HAA214 (majority)
    HDA214 (couple)
    UAF1094 (only one)


    #BrocadeFibreChannelNetworkingCommunity


  • 5.  Re: Latency issues between 6510 models

    Posted 05-09-2019 01:45 AM

    Concerning the BB default value, do you think I should increase it for ISL connections?

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 6.  Re: Poor performance and latency issues

    Posted 05-09-2019 08:23 AM
    1. Concerning the Class3 discarded frames", what does this exactely mean?
      1. There might be a Slow Drain device somewhere in your fabric.
    2. The interface of switches shows latency warning s for the ISL connections, the SFPs used are 16G all. So, what should I do regarding this please??
      1. This is possibly related to the above question.

    What is the topology of the fabrc? (Hub-Spoke/Star? Hybrid? Mesh?)

     

    Is the issue seen in both fabrics?

     

    Do any of the SFPs autonegotiate to a lower speed?

     

    sfpshow -all | grep -e Port -e Power (Case sensitive)

     

    This will show the RX and TX Power for all SFPs. Look for a low RX Power as compared to the TX Power. If you see this, you might have a cable issue. (Low meaning double digit uWatts)

     

    statsclear

    Wait for an hour or so, then

    porterrshow

     

    This will show the stats from the time of the last stats clear. If clean, the issue may have been seen during fabric forming/power on of servers, etc. Continue to monitor. If not clean, it should be a little bit easier to pinpoint the issue.

     

    There are a couple of free classes available here that can help you out. I highly recommend reviewing them.

     

    TSPERF-320

    TSPLOG-320

    TSDEV-320

    FPI-220

    MAPS

     

    This isn't a be-all list. But it is a good start.


    #BrocadeFibreChannelNetworkingCommunity


  • 7.  Re: Poor performance and latency issues

    Posted 05-10-2019 05:39 AM
      |   view attached

    Thanks for the valuable information. I will go through it and let you know the result. I joined this comlex environment couple of months ago, and my experience with Brocade is just with single/standalone swithces, not with complex fabrix, and I'm learning about this architecture from Brocade tutorials now, so, thanks for being patient with me till I understand everything to be able to better optimize our equipment :).

     

    Answering your questions here:

     

    What is the topology of the fabrc? (Hub-Spoke/Star? Hybrid? Mesh?) Not sure, attached a diagram for one of the fabrics, and both are identical. 

     

    - Is the issue seen in both fabrics? yes on both fabrics, only in the local ISL per site, I mean, not on the ISL of the inter-site connection.

     

    -  Do any of the SFPs autonegotiate to a lower speed? Yes, All are set to AutoNegotiate, and I've found that one of the ISLs detects the speed as 8G, despite the fact that the SFP is 16G. I will set the speed to manual.

     

    The weekend is here now, so, I will run the comands you've mentioned by next week adn let you know the resukts, meanwhile, resoind my feedback for now if you'd like to.

     

    Thanks,


    #BrocadeFibreChannelNetworkingCommunity


  • 8.  Inter-site connectivity

    Posted 05-09-2019 01:40 AM
      |   view attached

    Hello,

     

    We have a strachetd datacenter, the distance betwen both is less than a KM, connected with dark fiber cables. I have two Brocade 6510 switch stacks, four on each stack. Each stack is spreaded accross the two sites, two here, and two there. I attached a diagram for the current connectivity. There's only a single ISL cable between both sites, should't it be a pair of cables between sites for redundancy and load balancing? as we are using MetroClustered SA storages, and servers from both sites access storage on the other site.

     

    Good day,


    #BrocadeFibreChannelNetworkingCommunity


  • 9.  Re: Inter-site connectivity

    Posted 05-09-2019 08:27 AM

    The short answer is "yes".

     

    Pose the question to those in charge as something like this: "What happens if [port connecting the two buildings] goes down? This could be switch port failure, an SFP failure, a cabling failure, etc...

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 10.  Re: Poor performance and latency issues

    Posted 05-10-2019 05:42 AM
    Ah sorry I didn't see that you've recommended courses for me to preview, I've already started this :), however, thanks for the recommendation, I will check what you've mentioned.

    Good day,
    #BrocadeFibreChannelNetworkingCommunity


  • 11.  Re: Poor performance and latency issues

    Posted 05-14-2019 12:15 PM

    Yes, All are set to AutoNegotiate, and I've found that one of the ISLs detects the speed as 8G, despite the fact that the SFP is 16G. I will set the speed to manual.

     

    I would recommend not setting the speed to manual as it might cause other issues. Instead, it does look as though we have something specific to look at. IE: That port/cable set.


    #BrocadeFibreChannelNetworkingCommunity


  • 12.  Re: Poor performance and latency issues

    Posted 05-15-2019 12:04 PM
    I will review the cables, and I won't set the speed to manual as recommended.

    Thanks,
    #BrocadeFibreChannelNetworkingCommunity


  • 13.  Re: Poor performance and latency issues

    Posted 05-15-2019 12:22 PM

    With a port to focus on, there are quite a few commands that could shed some light (pun) on the issue.

     

    errdump (with a text capture)

    porterrshow

    portlogshow

    portlogshowport

    portshow

    portstatsshow

     

    Other commands that might be useful:

    trunkshow

    fabricshow

    islshow

     

    These are a good starting place.


    #BrocadeFibreChannelNetworkingCommunity


  • 14.  Re: Poor performance and latency issues

    Posted 05-11-2019 03:24 AM
    Here's an output for the Rx Tx power.

    SFPRxPowerdBm SFPRxPoweruW SFPTxPowerdBm SFPTxPoweruW
    -3.2 483.8 -3.3 467.9
    -3.2 483.8 -3.3 467.9
    -3.2 483.8 -3.3 467.9
    -3.2 483.8 -3.3 467.9
    -3.2 483.8 -3.3 467.9
    -3.2 483.8 -3.3 467.9
    -3.2 483.8 -3.3 467.9
    -4.4 362.5 -2.7 537.8
    -4.4 362.5 -2.7 537.8
    -3.4 454.8 -3 506.7
    -3.4 454.8 -3 506.7
    -3.8 414 -2.4 582.1
    -3.8 414 -2.4 582.1
    -3.8 414 -2.4 582.1
    -3.8 414 -2.4 582.1
    -3.8 414 -2.4 582.1
    -3.8 414 -2.4 582.1
    -3.8 414 -2.4 582.1
    -3.8 414 -2.4 582.1
    -2.4 578.3 -2.4 574.1
    -2.4 578.3 -2.4 574.1
    -2.7 541.7 -2.6 543.4
    -2.7 541.7 -2.6 543.4
    -2.7 541.7 -2.6 543.4
    -1.9 651.2 -2.5 559.1
    -1.9 651.2 -2.5 559.1
    -1.9 651.2 -2.5 559.1
    -3.9 403.5 -2.4 577.6
    -3.5 451.8 -2.4 574.8
    -2.4 575.6 -3.4 453.1
    -2.6 550.3 -2.6 547.2
    -3.7 429.8 -3.5 447.1
    -3.2 478.1 -2.5 563.1
    -1.5 709 0.2 1057.6
    -2.2 606.4 -3.2 483
    -2.1 610.4 -3.3 471.7
    -2.1 622.9 -2.4 578.9
    -2.5 563.3 -2.4 573.4
    -2.6 550.8 -2.6 555
    -3.1 486.8 -2.5 562.8
    -11.5 70.8 -2.5 567.9
    -11.5 70.8 -2.5 567.9
    -13.4 45.5 -2.4 574.2
    -13.4 45.5 -2.4 574.2
    -11.1 78.3 -2.3 592.7
    -11.1 78.3 -2.3 592.7
    -11.3 73.7 -3 503.5
    -11.3 73.7 -3 503.5
    -10.2 95.3 -2.6 550
    -10.2 95.3 -2.6 550
    -9.3 118.2 -2.6 551.1
    -9.3 118.2 -2.6 551.1
    -3 497.7 -2.3 591.7
    -3 497.7 -2.3 591.7
    -2.8 522.8 -2.4 577.2
    -2.8 522.8 -2.4 577.2
    -3.6 432.1 -2.3 587.7
    -3 497.9 -2.3 594.5
    -3.1 494 -2.7 542.8
    -3.6 435.8 -2.3 585.7
    -2.9 514.6 -2.2 607.7
    -2.9 514.6 -2.2 607.7
    -2 631.3 -2.4 573.3
    -2 631.3 -2.4 573.3
    -2.2 607.6 -2.2 603.7
    -2.2 607.6 -2.2 603.7
    -2 625 -2.3 592.3
    -2 625 -2.3 592.3
    -2.6 543.6 -2.2 607.1
    -2.6 543.6 -2.2 607.1
    -2.6 544.8 -2.3 588.4
    -2.6 544.8 -2.3 588.4
    -2.8 528.2 -2.2 600.9
    -2.8 528.2 -2.2 600.9
    -2.6 553.1 -2.2 600.6
    -2.6 553.1 -2.2 600.6
    -3.4 458.4 -2.2 605.8

    #BrocadeFibreChannelNetworkingCommunity


  • 15.  Re: Poor performance and latency issues

    Posted 05-14-2019 08:10 AM

    This grouping is rather low. Is this RX or TX? (I would assume RX...)

     

    -11.5 70.8 -2.5 567.9
    -11.5 70.8 -2.5 567.9
    -13.4 45.5 -2.4 574.2
    -13.4 45.5 -2.4 574.2
    -11.1 78.3 -2.3 592.7
    -11.1 78.3 -2.3 592.7
    -11.3 73.7 -3 503.5
    -11.3 73.7 -3 503.5
    -10.2 95.3 -2.6 550
    -10.2 95.3 -2.6 550
    -9.3 118.2 -2.6 551.1
    -9.3 118.2 -2.6 551.1

     

    I would recommend reviewing all the cabling.


    #BrocadeFibreChannelNetworkingCommunity


  • 16.  Re: Poor performance and latency issues

    Posted 05-14-2019 08:12 AM

    If you want to see the ports associated with this, run:

     

    sfpshow -all | grep -e Port -e Power

     

    The grep portion is case sensitive.


    #BrocadeFibreChannelNetworkingCommunity