Brocade switch ISL configuration for DWDM link,

[Anonymous]

Hi Experts,

                   We have two IBM-B40 brocade SAN switches in production and DR site which are ~ 10km apart connected with DWDM link. The ISL port are long wave SFP+, configured as e-port, LE mode which are connected to DWDM switch. But the ISl ports shows offline. Please suggest what are the other things to consider.

Thanks

J

[Anonymous]

-active or passive xWDM component may require different settings.

-also progection switching if your xWDM (provider) has configured that

-SFP, again active or passive xWDM gear if active SFP only needs to reach the WDM gear while with passive you need to light the entire length yourself.

LE suggests you lit the fibres yourself

But difficult to say with no topology logs etc.

[felipon]

Dion's right, please post the output of switchshsow, portshow and portcfgshow <slot/port> in order to try to figure out the reason of the ISL being offline.

rgds

[Anonymous]

Hi Dion /felipn, Thanks for your response,
here is the output of commands listed.

SWITCH_1

1) switchshow

:FID128:admin> switchshow
switchName:     LINK1
switchType:     66.1
switchState:    Online
switchMode:     Native
switchRole:     Principal
switchDomain:   73
switchId:       fffc49
switchWwn:     
zoning:         OFF
switchBeacon:   OFF
FC Router:      OFF
Allow XISL Use: OFF
LS Attributes:  [FID: 128, Base Switch: No, Default Switch: Yes, Address Mode 0]
 
Index Port Address Media Speed State     Proto
==============================================
   0   0   490000   id     2G   No_Light    FC  LD
   1   1   490100   id     N8   No_Light    FC
   2   2   490200   id     N8   No_Light    FC
   3   3   490300   id     N8   No_Light    FC
   4   4   490400   id     N8   No_Light    FC
   5   5   490500   id     N8   No_Light    FC
   6   6   490600   id     N8   No_Light    FC
   7   7   490700   id     N8   No_Light    FC
   8   8   490800   id     N4   Online      FC  F-Port  50:05:07:63:08:38:c1:d0
   9   9   490900   id     N8   No_Light    FC
  10  10   490a00   id     N8   No_Light    FC
  11  11   490b00   id     N8   No_Light    FC
  12  12   490c00   id     N8   No_Light    FC
  13  13   490d00   id     N8   No_Light    FC
  14  14   490e00   id     N8   No_Light    FC
  15  15   490f00   id     N8   No_Light    FC
  16  16   491000   id     N8   No_Light    FC
  17  17   491100   id     N8   No_Light    FC
  18  18   491200   id     N8   No_Light    FC
  19  19   491300   id     N8   No_Light    FC
  20  20   491400   id     N8   No_Light    FC
  21  21   491500   id     N8   No_Light    FC
  22  22   491600   id     N8   No_Light    FC
  23  23   491700   id     N8   No_Light    FC
  24  24   491800   id     N8   No_Light    FC
  25  25   491900   id     N8   No_Light    FC
  26  26   491a00   id     N8   No_Light    FC
  27  27   491b00   id     N8   No_Light    FC
  28  28   491c00   id     N8   No_Light    FC
  29  29   491d00   id     N8   No_Light    FC
  30  30   491e00   id     N8   No_Light    FC
  31  31   491f00   id     N8   No_Light    FC
  32  32   492000   id     N8   No_Light    FC
  33  33   492100   id     N8   No_Light    FC
  34  34   492200   id     N8   No_Light    FC
  35  35   492300   id     N8   No_Light    FC
  36  36   492400   id     N8   No_Light    FC
  37  37   492500   id     N8   No_Light    FC
  38  38   492600   id     N8   No_Light    FC
  39  39   492700   id     N8   No_Light    FC

FID128:admin> portshow 0
portIndex:   0
portName: port0
portHealth: OFFLINE
 
Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x1001        PRESENT U_PORT RRDY_MODE
LocalSwcFlags: 0x0
portType:  17.0
POD Port: Port is licensed
portState: 2    Offline
Protocol: FC
portPhys:  4    No_Light        portScn:   2    Offline
port generation number:    8740
state transition count:    8724
 
portId:    490000
portIfId:    4302000f
portWwn:   20:00:00:05:33:4b:46:1c
portWwn of device(s) connected:
 
Distance:  auto (desired = 20 Km)
portSpeed:  2Gbps
 
LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 4          Frjt:         0     
Unknown:           0          Loss_of_sync: 3794       Fbsy:         0     
Lli:               2239920    Loss_of_sig:  7509
Proc_rqrd:         263        Protocol_err: 0
Timed_out:         0          Invalid_word: 3559033018
Rx_flushed:        0          Invalid_crc:  53458
Tx_unavail:        0          Delim_err:    216
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        3750
Suspended:         0          Lr_out:       1231
Parity_err:        0          Ols_in:       3
2_parity_err:      0          Ols_out:      3750
CMI_bus_err:       0

3) FID128:admin> portcfgshow 0/0
Area Number:              0
Speed Level:              2G
Fill Word(On Active)      1(Arbff-Arbff)
Fill Word(Current)        1(Arbff-Arbff)
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             LD
VC Link Init              ON
Desired Distance          20 Km
Reserved Buffers          26
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
Locked E_Port             ON
ISL R_RDY Mode            ON
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV enable            OFF
NPIV capability           ON
QOS E_Port                OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
Credit Recovery           ON
F_Port Buffers            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
CSCTL mode:               OFF

==================================================================

Switch2

1) FID128:admin> switchshow
switchName:     DCESANPR1
switchType:     66.1
switchState:    Online
switchMode:     Native
switchRole:     Principal
switchDomain:   74
switchId:       fffc4a
switchWwn:    
zoning:         OFF
switchBeacon:   OFF
FC Router:      OFF
Allow XISL Use: OFF
LS Attributes:  [FID: 128, Base Switch: No, Default Switch: Yes, Address Mode 0]
 
Index Port Address Media Speed State     Proto
==============================================
   0   0   4a0000   id     2G   No_Light    FC  LD
   1   1   4a0100   id     N8   No_Light    FC
   2   2   4a0200   id     N8   No_Light    FC
   3   3   4a0300   id     N8   No_Light    FC
   4   4   4a0400   id     2G   No_Light    FC
   5   5   4a0500   id     N8   No_Light    FC
   6   6   4a0600   id     N8   No_Light    FC
   7   7   4a0700   id     N8   No_Light    FC
   8   8   4a0800   id     N4   Online      FC  F-Port  50:05:07:63:08:38:c1:cf
   9   9   4a0900   id     N8   No_Light    FC
  10  10   4a0a00   id     N8   No_Light    FC
  11  11   4a0b00   id     N8   No_Light    FC
  12  12   4a0c00   id     N8   No_Light    FC
  13  13   4a0d00   id     N8   No_Light    FC
  14  14   4a0e00   id     N8   No_Light    FC
  15  15   4a0f00   id     N8   No_Light    FC
  16  16   4a1000   id     N4   Online      FC  F-Port  50:05:07:63:08:18:c1:cf
  17  17   4a1100   id     N8   No_Light    FC
  18  18   4a1200   id     N8   No_Light    FC
  19  19   4a1300   id     N8   No_Light    FC
  20  20   4a1400   id     N8   No_Light    FC
  21  21   4a1500   id     N8   No_Light    FC
  22  22   4a1600   id     N8   No_Light    FC
  23  23   4a1700   id     N8   No_Light    FC
  24  24   4a1800   id     N8   No_Light    FC
  25  25   4a1900   id     N8   No_Light    FC
  26  26   4a1a00   id     N8   No_Light    FC
  27  27   4a1b00   id     N8   No_Light    FC
  28  28   4a1c00   id     N8   No_Light    FC
  29  29   4a1d00   id     N8   No_Light    FC
  30  30   4a1e00   id     N8   No_Light    FC
  31  31   4a1f00   id     N8   No_Light    FC
  32  32   4a2000   id     N8   No_Light    FC
  33  33   4a2100   id     N8   No_Light    FC
  34  34   4a2200   id     N8   No_Light    FC
  35  35   4a2300   id     N8   No_Light    FC
  36  36   4a2400   id     N8   No_Light    FC
  37  37   4a2500   id     N8   No_Light    FC
  38  38   4a2600   id     N8   No_Light    FC
  39  39   4a2700   id     N8   No_Light    FC

2) FID128:admin> portshow 0
portIndex:   0
portName: port0
portHealth: OFFLINE
 
Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x1001        PRESENT U_PORT RRDY_MODE
LocalSwcFlags: 0x0
portType:  17.0
POD Port: Port is licensed
portState: 2    Offline
Protocol: FC
portPhys:  4    No_Light        portScn:   0
port generation number:    0
state transition count:    0
 
portId:    4a0000
portIfId:    4302000f
portWwn:   20:00:00:05:33:f3:c4:9a
portWwn of device(s) connected:
 
Distance:  auto (desired = 20 Km)
portSpeed:  2Gbps
 
LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 0          Frjt:         0
Unknown:           0          Loss_of_sync: 247184     Fbsy:         0
Lli:               1334886    Loss_of_sig:  232834
Proc_rqrd:         5          Protocol_err: 0
Timed_out:         0          Invalid_word: 0
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        0
Suspended:         0          Lr_out:       0
Parity_err:        0          Ols_in:       0
2_parity_err:      0          Ols_out:      0
CMI_bus_err:       0
 
3) FID128:admin> portcfgshow 0/0
Area Number:              0
Speed Level:              2G
Fill Word(On Active)      1(Arbff-Arbff)
Fill Word(Current)        1(Arbff-Arbff)
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             LD
VC Link Init              ON
Desired Distance          20 Km
Reserved Buffers          26
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
Locked E_Port             ON
ISL R_RDY Mode            ON
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV enable            OFF
NPIV capability           ON
QOS E_Port                OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
Credit Recovery           ON
F_Port Buffers            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
CSCTL mode:               OFF

[Anonymous]

Two things that stand out is the selected fillword and VC link init.

ARBff is used at 8Gb and higher, vc link init = ON means you run ARBff when the ISL is in VC_RDY mode.

You are running at 2Gb in R_RDY mode.

I doubt that the xWDM gear understands that ARBff primitive and believe IDLE filwords should be used.

Although VC link Init is ON it should be ignored because you're running in R-RDY mode.

It may however be advisable to set it to OFF to reduce potential confusion in the future.

[felipon]

Hi,

with both ends of the ISL in "no light" status I reckon that the problem is between each port and the xWDM array. Once you get the port IN_sync, you will probably have to apply Dion's recommendations.

what do you see from the xWDM array perspective?
What SFP are you using, Single-mode or mono-mode?

what SfPs do you have on the DWDM arrays?

[Anonymous]

Hi,

what kind of DWDM do you have?

Regards

Uli

[Anonymous]

You configured the port on R-RDY mode with Credit-Recovery enabled. thats not a valid option. Gvine the fact the port is fixed 2G there is no requirement of setting VC link init since that will turn ARBff on if your switch runs FOS 6.3.1 or higher.

So in short:

When thelink should remain in fixed 2G mode and there is active DWDM with TDM MUXes in use:

1. Turn off Credit Recovery

2. Turn off VC_Link init

3. Use fixed (LS and not LD) mode so you can control you buffers better.

If the link uses passive DWDM and it can use 8G speeds use

1. Turn on VC-Link init to use ARBff fillwords

2. Turn on Credit-Recovery

3. Turn on QoS mode if you want to use this feature

4. Turn off ISL R_RDY mode

This should work if you run recent FOS code.

Cheers,

Erwin

[palmerr]

I assume that you solved this issue ? I had a similar issue and can offer some advice if needed