Brocade Fibre Channel Networking Community

Expand all | Collapse all

BE_PORT_LATENCY on DCX

Jump to Best Answer
  • 1.  BE_PORT_LATENCY on DCX

    Posted 05-11-2018 03:12 AM

    I'm reciving alert with back end ports latency.

     

    BE Port 6/85, Condition=ALL_BE_PORTS(BE_LATENCY_IMPACT==IO_PERF_IMPACT), Current Value:[BE_LATENCY_IMPACT,IO_PERF_IMPACT, 88.6% of 1 secs], RuleName=defALL_BE_PORTS_LATENCY_IMPACT, Dashboard Category=Fabric Performance Impact.

     

    I have recived a bulk port alerts at once.

     

    How can I find the root of issue?

    How can I understood a BE port address 6/85?


    #BrocadeFibreChannelNetworkingCommunity
    #FPI


  • 2.  Re: BE_PORT_LATENCY on DCX

    Posted 05-11-2018 04:47 AM

    Hello,

     

    as i see you based on message its comming from MAPS monitoring. Based on message "ALL_BE_PORTS" means that the latency bottleneck has been detected on backen port of DCX.

     

    What the backend port is si nice explained in MAPS admin guide:

     

    A back-end port connects a core switching blade to a port or application blade (and vice versa). The primary task of back-end ports is to
    route packets passing through a switch's ASICs. Switch (and consequently fabric) performance degrades when there are errors in backend
    ports. MAPS error notification allows you to take corrective action earlier. In terms of monitoring system functionality, back-end
    ports can be connected to ports within a fixed-port switch or to other blades within a Backbone chassis, and so their functionality is
    different from front-end ports, which connect to devices outside of the switch.

    Do you have credit recovery enabled ?


    #BrocadeFibreChannelNetworkingCommunity


  • 3.  Re: BE_PORT_LATENCY on DCX

    Posted 05-11-2018 05:43 AM
    Yes, we have enabled credit recovery on the ISL trunks. And we have enabled QOS on the ISL.

    #BrocadeFibreChannelNetworkingCommunity


  • 4.  Re: BE_PORT_LATENCY on DCX
    Best Answer

    Posted 05-11-2018 05:02 AM
    Hi Anton :)

    You can find the internal port numbers using the bladeportmap command.

    Backend latencies could be caused by all the same reasons that are applicable to the normal ISL latencies. You may think that backend links are ISLs between the ASICs, like there is a small SAN fabric internal to the chassis.

    Is it 8G or 16G DCX? Do you have the bottleneckmon setup? It would fix many internal issues on the flight. Have you seen any other messages around the alert time?
    #BrocadeFibreChannelNetworkingCommunity


  • 5.  Re: BE_PORT_LATENCY on DCX

    Posted 05-11-2018 05:55 AM

    I don't understand the output of bladeportmap.

    CTL

    BPt

    UPt uplink port

    SPt #frontend port-1:backend

    MPt minis port #

    CPt chip port#

    IFi

    GPt

    M# (Minis)

    C# (Chip)

    Sw# -1:BackEnd, 0:FrontEnd

    G#

    ChID

    PBPt:ISL(Backend port)

    OID

     

     bladeportmap 6
    miniS=0 chips/Minis=1 bports/chip=48
    miniS=1 chips/Minis=1 bports/chip=48
    minis_count=2 chip_count=2 port_count=96
    
    CTL BPt  UPt  SPt  MPt  CPt  IFi  GPt  M#   C#   Sw#  G#   ChID       PBPt:ISL        OID
    ================================================================================================
    ENB 0    793  25   0    0    1    0    0    0    2    6    CONDOR3    :No_Module      0x43628000
    ENB 1    797  29   1    1    2    1    0    0    2    6    CONDOR3    :No_Module      0x43628001
    ENB 2    281  9    2    2    3    2    0    0    2    6    CONDOR3    :No_Module      0x43628002
    ENB 3    285  13   3    3    4    3    0    0    2    6    CONDOR3    :No_Module      0x43628003
    ENB 4    789  21   4    4    5    4    0    0    2    6    CONDOR3    :No_Module      0x43628004
    ENB 5    785  17   5    5    6    5    0    0    2    6    CONDOR3    :No_Module      0x43628005
    ENB 6    277  5    6    6    7    6    0    0    2    6    CONDOR3    :No_Module      0x43628006
    ENB 7    273  1    7    7    8    7    0    0    2    6    CONDOR3    :No_Module      0x43628007
    ENB 8    -1   -1   8    8    13   0    0    0    -1   1804 CONDOR3    7/2:            0x43628008
    ENB 9    -1   -1   9    9    13   1    0    0    -1   1804 CONDOR3    7/4:            0x43628009
    ENB 10   -1   -1   10   10   15   2    0    0    -1   1806 CONDOR3    7/51:           0x4362800a
    ENB 11   -1   -1   11   11   15   3    0    0    -1   1806 CONDOR3    7/52:           0x4362800b
    ENB 12   -1   -1   12   12   13   4    0    0    -1   1804 CONDOR3    7/1:            0x4362800c
    ENB 13   -1   -1   13   13   13   5    0    0    -1   1804 CONDOR3    7/3:            0x4362800d
    ENB 14   -1   -1   14   14   15   6    0    0    -1   1806 CONDOR3    7/49:           0x4362800e
    ENB 15   -1   -1   15   15   15   7    0    0    -1   1806 CONDOR3    7/50:           0x4362800f
    ENB 16   -1   -1   16   16   21   0    0    0    -1   1812 CONDOR3    8/2:            0x43628010
    ENB 17   -1   -1   17   17   18   1    0    0    -1   1809 CONDOR3    8/49:           0x43628011
    ENB 18   -1   -1   18   18   18   2    0    0    -1   1809 CONDOR3    8/52:           0x43628012
    ENB 19   -1   -1   19   19   21   3    0    0    -1   1812 CONDOR3    8/3:            0x43628013
    ENB 20   -1   -1   20   20   21   4    0    0    -1   1812 CONDOR3    8/1:            0x43628014
    ENB 21   -1   -1   21   21   21   5    0    0    -1   1812 CONDOR3    8/4:            0x43628015
    ENB 22   -1   -1   22   22   18   6    0    0    -1   1809 CONDOR3    8/51:           0x43628016
    ENB 23   -1   -1   23   23   18   7    0    0    -1   1809 CONDOR3    8/50:           0x43628017
    ENB 24   276  4    24   24   25   0    0    0    2    9    CONDOR3    :No_Module      0x43628018
    ENB 25   272  0    25   25   26   1    0    0    2    9    CONDOR3    :No_Module      0x43628019
    ENB 26   788  20   26   26   27   2    0    0    2    9    CONDOR3    :No_Module      0x4362801a
    ENB 27   784  16   27   27   28   3    0    0    2    9    CONDOR3    :No_Module      0x4362801b
    ENB 28   280  8    28   28   29   4    0    0    2    9    CONDOR3    :No_Module      0x4362801c
    ENB 29   284  12   29   29   30   5    0    0    2    9    CONDOR3    :No_Module      0x4362801d
    ENB 30   792  24   30   30   31   6    0    0    2    9    CONDOR3    :No_Module      0x4362801e
    ENB 31   796  28   31   31   32   7    0    0    2    9    CONDOR3    :No_Module      0x4362801f
    ENB 32   -1   -1   32   32   33   0    0    0    -1   514  CONDOR3    1/2:            0x43628020
    ENB 33   -1   -1   33   33   35   1    0    0    -1   562  CONDOR3    1/50:           0x43628021
    ENB 34   -1   -1   34   34   35   2    0    0    -1   562  CONDOR3    1/49:           0x43628022
    ENB 35   -1   -1   35   35   33   3    0    0    -1   514  CONDOR3    1/4:            0x43628023
    ENB 36   -1   -1   36   36   33   4    0    0    -1   514  CONDOR3    1/3:            0x43628024
    ENB 37   -1   -1   37   37   35   5    0    0    -1   562  CONDOR3    1/51:           0x43628025
    ENB 38   -1   -1   38   38   35   6    0    0    -1   562  CONDOR3    1/52:           0x43628026
    ENB 39   -1   -1   39   39   33   7    0    0    -1   514  CONDOR3    1/1:            0x43628027
    ENB 40   -1   -1   40   40   41   0    0    0    -1   769  CONDOR3    2/1:            0x43628028
    ENB 41   -1   -1   41   41   41   1    0    0    -1   769  CONDOR3    2/2:            0x43628029
    ENB 42   -1   -1   42   42   41   2    0    0    -1   769  CONDOR3    2/3:            0x4362802a
    ENB 43   -1   -1   43   43   41   3    0    0    -1   769  CONDOR3    2/4:            0x4362802b
    ENB 44   -1   -1   44   44   48   4    0    0    -1   818  CONDOR3    2/50:           0x4362802c
    ENB 45   -1   -1   45   45   48   5    0    0    -1   818  CONDOR3    2/51:           0x4362802d
    ENB 46   -1   -1   46   46   48   6    0    0    -1   818  CONDOR3    2/52:           0x4362802e
    ENB 47   -1   -1   47   47   48   7    0    0    -1   818  CONDOR3    2/49:           0x4362802f
    ENB 48   795  27   0    0    1    0    1    1    2    18   CONDOR3    :No_Module      0x43628030
    ENB 49   799  31   1    1    2    1    1    1    2    18   CONDOR3    :No_Module      0x43628031
    ENB 50   283  11   2    2    3    2    1    1    2    18   CONDOR3    :No_Module      0x43628032
    ENB 51   287  15   3    3    4    3    1    1    2    18   CONDOR3    :No_Module      0x43628033
    ENB 52   791  23   4    4    5    4    1    1    2    18   CONDOR3    :No_Module      0x43628034
    ENB 53   787  19   5    5    6    5    1    1    2    18   CONDOR3    :No_Module      0x43628035
    ENB 54   279  7    6    6    7    6    1    1    2    18   CONDOR3    :No_Module      0x43628036
    ENB 55   275  3    7    7    8    7    1    1    2    18   CONDOR3    :No_Module      0x43628037
    ENB 56   -1   -1   8    8    12   0    1    1    -1   1851 CONDOR3    7/61:           0x43628038
    ENB 57   -1   -1   9    9    10   1    1    1    -1   1849 CONDOR3    7/8:            0x43628039
    ENB 58   -1   -1   10   10   10   2    1    1    -1   1849 CONDOR3    7/10:           0x4362803a
    ENB 59   -1   -1   11   11   12   3    1    1    -1   1851 CONDOR3    7/56:           0x4362803b
    ENB 60   -1   -1   12   12   10   4    1    1    -1   1849 CONDOR3    7/12:           0x4362803c
    ENB 61   -1   -1   13   13   12   5    1    1    -1   1851 CONDOR3    7/62:           0x4362803d
    ENB 62   -1   -1   14   14   10   6    1    1    -1   1849 CONDOR3    7/15:           0x4362803e
    ENB 63   -1   -1   15   15   12   7    1    1    -1   1851 CONDOR3    7/58:           0x4362803f
    ENB 64   -1   -1   16   16   22   0    1    1    -1   1861 CONDOR3    8/10:           0x43628800
    ENB 65   -1   -1   17   17   18   1    1    1    -1   1857 CONDOR3    8/56:           0x43628801
    ENB 66   -1   -1   18   18   18   2    1    1    -1   1857 CONDOR3    8/61:           0x43628802
    ENB 67   -1   -1   19   19   22   3    1    1    -1   1861 CONDOR3    8/15:           0x43628803
    ENB 68   -1   -1   20   20   22   4    1    1    -1   1861 CONDOR3    8/12:           0x43628804
    ENB 69   -1   -1   21   21   22   5    1    1    -1   1861 CONDOR3    8/8:            0x43628805
    ENB 70   -1   -1   22   22   18   6    1    1    -1   1857 CONDOR3    8/62:           0x43628806
    ENB 71   -1   -1   23   23   18   7    1    1    -1   1857 CONDOR3    8/58:           0x43628807
    ENB 72   278  6    24   24   25   0    1    1    2    21   CONDOR3    :No_Module      0x43628808
    ENB 73   274  2    25   25   26   1    1    1    2    21   CONDOR3    :No_Module      0x43628809
    ENB 74   790  22   26   26   27   2    1    1    2    21   CONDOR3    :No_Module      0x4362880a
    ENB 75   786  18   27   27   28   3    1    1    2    21   CONDOR3    :No_Module      0x4362880b
    ENB 76   282  10   28   28   29   4    1    1    2    21   CONDOR3    :No_Module      0x4362880c
    ENB 77   286  14   29   29   30   5    1    1    2    21   CONDOR3    :No_Module      0x4362880d
    ENB 78   794  26   30   30   31   6    1    1    2    21   CONDOR3    :No_Module      0x4362880e
    ENB 79   798  30   31   31   32   7    1    1    2    21   CONDOR3    :No_Module      0x4362880f
    ENB 80   -1   -1   32   32   33   0    1    1    -1   522  CONDOR3    1/10:           0x43628810
    ENB 81   -1   -1   33   33   34   1    1    1    -1   570  CONDOR3    1/58:           0x43628811
    ENB 82   -1   -1   34   34   34   2    1    1    -1   570  CONDOR3    1/61:           0x43628812
    ENB 83   -1   -1   35   35   33   3    1    1    -1   522  CONDOR3    1/12:           0x43628813
    ENB 84   -1   -1   36   36   33   4    1    1    -1   522  CONDOR3    1/15:           0x43628814
    ENB 85   -1   -1   37   37   34   5    1    1    -1   570  CONDOR3    1/62:           0x43628815
    ENB 86   -1   -1   38   38   34   6    1    1    -1   570  CONDOR3    1/56:           0x43628816
    ENB 87   -1   -1   39   39   33   7    1    1    -1   522  CONDOR3    1/8:            0x43628817
    ENB 88   -1   -1   40   40   44   0    1    1    -1   780  CONDOR3    2/12:           0x43628818
    ENB 89   -1   -1   41   41   44   1    1    1    -1   780  CONDOR3    2/10:           0x43628819
    ENB 90   -1   -1   42   42   44   2    1    1    -1   780  CONDOR3    2/15:           0x4362881a
    ENB 91   -1   -1   43   43   44   3    1    1    -1   780  CONDOR3    2/8:            0x4362881b
    ENB 92   -1   -1   44   44   48   4    1    1    -1   826  CONDOR3    2/58:           0x4362881c
    ENB 93   -1   -1   45   45   48   5    1    1    -1   826  CONDOR3    2/62:           0x4362881d
    ENB 94   -1   -1   46   46   48   6    1    1    -1   826  CONDOR3    2/61:           0x4362881e
    ENB 95   -1   -1   47   47   48   7    1    1    -1   826  CONDOR3    2/56:           0x4362881f

    It is the 16G DCX.

    We have FPI enabled, he was disable bottleneck.

    I want to find FE ports, wich create traffic to BE port (Mapping FE to BE).


    #BrocadeFibreChannelNetworkingCommunity


  • 6.  Re: BE_PORT_LATENCY on DCX
    Best Answer

    Posted 05-11-2018 06:17 AM

    Hello,

     

    based on your output its DCX 8510 - 4. Only in this case you may have core blade in slot 6.

    From your output latency is on 6/85, which means slot 6 and backend port 85. As you may see its mapped to port 1/62 on slot 1.

     

    ENB 85   -1   -1   37   37   34   5    1    1    -1   570  CONDOR3    1/62:           0x43628815

    If you will check portmap for slot 1 you will see to which ASIC is this CORE BLADE backend port connected. 

    But there my knowledge about mapping ends...there is command sloterrshow where you may check errors but i still didnt decode that.

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 7.  Re: BE_PORT_LATENCY on DCX
    Best Answer

    Posted 05-14-2018 02:20 AM

    Hi!

    FOS 7.4.1c

    I did'n make statsclear.

    The Alerts is rising time to time.

    I found that the BE port going to slot 1.

    root> bladeportmap 6

    BPt - Back End port number

    PBPt:ISL - Outgoing Back End slot number/port number

    I have seen BB credit pick on the FE port 1/9 at this time. http://c2n.me/3U468o8

    This is ISL to blade switch.

    On the blade switch i have seen BB credit pick on the FE port 1/9.

    This is the NPIV port (Power server) with one virtual port with pick in port send Mb/s and have one LUN


    #BrocadeFibreChannelNetworkingCommunity


  • 8.  Re: BE_PORT_LATENCY on DCX

    Posted 05-14-2018 03:13 AM

    Hello, 

     

    do you see any bottleneck on this NPIV port? Check portstatsshow about this.

    It looks like that you have slowdrain device in your fabric which is consuming your BB credits.

     

    Sometimes its hard do find out that, my advice is to clear stats on all switches and look on c3 TX/RX timeouts after some time.

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 9.  Re: BE_PORT_LATENCY on DCX

    Posted 05-14-2018 01:26 AM

    Hi,

     

    I would watch out for FE FPI Messages at these times no find the contributing FE Port(s).

     

    TL


    #BrocadeFibreChannelNetworkingCommunity


  • 10.  Re: BE_PORT_LATENCY on DCX

    Posted 05-14-2018 02:02 AM
    Hi,
    There is not FPI Messages at this time on the FE ports.
    #BrocadeFibreChannelNetworkingCommunity