Link_failure

Anonymous · ‎03-16-2014

when i do a portshow on my fc port I see this, does this indicate a link or hardware failure?

dcx2:FID128:root> portshow 2/23

portIndex: 87

portName:

portHealth: HEALTHY

Authentication: None

portDisableReason: None

portCFlags: 0x1

portFlags: 0x24b03 PRESENT ACTIVE F_PORT G_PORT U_PORT NPIV LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI

LocalSwcFlags: 0x0

portType: 17.0

portState: 1 Online

Protocol: FC

portPhys: 6 In_Sync portScn: 32 F_Port

port generation number: 2152

state transition count: 13

portId: 155700

portIfId: 4322001e

portWwn: 20:57:00:05:33:57:44:00

portWwn of device(s) connected:

20:00:00:25:b5:11:0b:1f

20:00:00:25:b5:11:0b:9f

20:00:00:25:b5:11:0b:7f

20:00:00:25:b5:11:0b:bf

20:00:00:25:b5:11:0b:fe

20:00:00:25:b5:11:0b:ef

20:1c:54:7f:ee:b5:c6:80

Distance: normal

portSpeed: N8Gbps

LE domain: 0

FC Fastwrite: OFF

Interrupts: 0 Link_failure: 3 Frjt: 0

Unknown: 377 Loss_of_sync: 7 Fbsy: 0

Lli: 93 Loss_of_sig: 10

Proc_rqrd: 68440 Protocol_err: 0

Timed_out: 0 Invalid_word: 1728881

Rx_flushed: 0 Invalid_crc: 0

Tx_unavail: 0 Delim_err: 0

Free_buffer: 0 Address_err: 0

Overrun: 0 Lr_in: 4

Suspended: 0 Lr_out: 3

Parity_err: 0 Ols_in: 3

2_parity_err: 0 Ols_out: 4

CMI_bus_err: 0

Anonymous · ‎03-16-2014

and this

dcx2:FID128:root> portshow 8/23
portIndex: 215
portName:
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x24b03 PRESENT ACTIVE F_PORT G_PORT U_PORT NPIV LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI
LocalSwcFlags: 0x0
portType: 17.0
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 2150
state transition count: 15

portId: 15d700
portIfId: 4382001e
portWwn: 20:d7:00:05:33:57:44:00
portWwn of device(s) connected:
20:00:00:25:b5:11:0b:8f
20:00:00:25:b5:11:0b:5f
20:00:00:25:b5:11:0b:cf
20:00:00:25:b5:11:0b:0f
20:00:00:25:b5:11:0b:6f
20:00:00:25:b5:11:0b:4f
20:1b:54:7f:ee:b5:c6:80
Distance: normal
portSpeed: N8Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts: 0 Link_failure: 4 Frjt: 0
Unknown: 645 Loss_of_sync: 27 Fbsy: 0
Lli: 98 Loss_of_sig: 30
Proc_rqrd: 413670 Protocol_err: 0
Timed_out: 0 Invalid_word: 797647
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 6
Suspended: 0 Lr_out: 4
Parity_err: 0 Ols_in: 4
2_parity_err: 0 Ols_out: 5
CMI_bus_err: 0

Anonymous · ‎03-16-2014

diagshow:

Diagnostics Status: Sun Mar 16 16:20:39 2014

Slot: 2 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
2/0 56 OK DN 8G Auto -- -- --
2/1 57 OK DN 8G Auto -- -- --
2/2 59 OK DN 8G Auto -- -- --
2/3 58 OK DN 8G Auto -- -- --
2/4 60 OK DN 8G Auto -- -- --
2/5 61 OK DN 8G Auto -- -- --
2/6 63 OK DN 8G Auto -- -- --
2/7 62 OK DN 8G Auto -- -- --
2/8 16 OK DN 8G Auto -- -- --
2/9 17 OK DN 8G Auto -- -- --
2/10 19 OK DN 8G Auto -- -- --
2/11 18 OK DN 8G Auto -- -- --
2/12 20 OK DN 8G Auto -- -- --
2/13 21 OK DN 8G Auto -- -- --
2/14 23 OK DN 8G Auto -- -- --
2/15 22 OK DN 8G Auto -- -- --
2/16 24 OK DN 8G Auto -- -- --
2/17 25 OK DN 8G Auto -- -- --
2/18 27 OK DN 8G Auto -- -- --
2/19 26 OK DN 8G Auto -- -- --
2/20 28 OK DN 8G Auto -- -- --
2/21 29 OK DN 8G Auto -- -- --
2/22 31 OK DN 8G Auto -- -- --
2/23 30 OK UP 8G Auto 1823026950 248043035 1728881
2/24 64 OK DN 8G Auto -- -- --
2/25 65 OK DN 8G Auto -- -- --
2/26 67 OK DN 8G Auto -- -- --
2/27 66 OK DN 8G Auto -- -- --
2/28 68 OK DN 8G Auto -- -- --
2/29 69 OK UP 8G Auto 1886491129 834282991 668420
2/30 71 OK DN 8G Auto -- -- --
2/31 70 OK DN 8G Auto -- -- --
2/32 72 OK UP 8G Auto 1793705475 4259765430 139571
2/33 73 OK UP 4G Auto 292298249 3643542693 3
2/34 75 OK UP 4G Auto 1681905406 3866540566 305496
2/35 74 OK DN 8G Auto -- -- --
2/36 76 OK DN 8G Auto -- -- --
2/37 77 OK DN 8G Auto -- -- --
2/38 79 OK DN 8G Auto -- -- --
2/39 78 OK DN 8G Auto -- -- --
2/40 32 OK DN 8G Auto -- -- --
2/41 33 OK DN 8G Auto -- -- --
2/42 35 OK UP 4G Auto 2710271523 1392938361 14
2/43 34 OK DN 8G Auto -- -- --
2/44 36 OK DN 8G Auto -- -- --
2/45 37 OK DN 8G Auto -- -- --
2/46 39 OK DN 8G Auto -- -- --
2/47 38 OK DN 8G Auto -- -- --

Total Diag Frames Tx: 800
Total Diag Frames Rx: 800

Slot: 8 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
8/0 56 OK UP 4G Auto 3068476965 1677092200 2852
8/1 57 OK DN 8G Auto -- -- --
8/2 59 OK DN 8G Auto -- -- --
8/3 58 OK DN 8G Auto -- -- --
8/4 60 OK DN 8G Auto -- -- --
8/5 61 OK DN 8G Auto -- -- --
8/6 63 OK UP 8G Auto 29638509 14013233 582596
8/7 62 OK DN 8G Auto -- -- --
8/8 16 OK DN 8G Auto -- -- --
8/9 17 OK DN 8G Auto -- -- --
8/10 19 OK UP 4G Auto 607 108 0
8/11 18 OK UP 4G Auto 2080410589 2312253700 13
8/12 20 OK UP 8G Auto 2846749872 1544554319 2416
8/13 21 OK DN 8G Auto -- -- --
8/14 23 OK UP 8G Auto 1639825540 2667874216 1458548
8/15 22 OK UP 4G Auto 593154142 1177327792 2307570
8/16 24 OK UP 4G Auto 3898306237 2747105761 0
8/17 25 OK DN 8G Auto -- -- --
8/18 27 OK DN 8G Auto -- -- --
8/19 26 OK UP 4G Auto 836951007 4264096104 16
8/20 28 OK DN 8G Auto -- -- --
8/21 29 OK DN 8G Auto -- -- --
8/22 31 OK DN 8G Auto -- -- --
8/23 30 OK UP 8G Auto 935533110 2232716263 797647
8/24 64 OK UP 4G Auto 3966358398 1928475535 0
8/25 65 OK DN 8G Auto -- -- --
8/26 67 OK DN 8G Auto -- -- --
8/27 66 OK UP 8G Auto 1314763214 23126552 90
8/28 68 OK DN 8G Auto -- -- --
8/29 69 OK UP 4G Auto 899419283 3638516596 0
8/30 71 OK UP 4G Auto 2862713440 2204495516 0
8/31 70 OK DN 8G Auto -- -- --
8/32 72 OK UP 4G Auto 3552400717 4258290284 0
8/33 73 OK UP 4G Auto 772370041 3964023239 0
8/34 75 OK UP 8G Auto 4267602519 4035745389 129525
8/35 74 OK UP 4G Auto 4019481025 47215083 0
8/36 76 OK DN 8G Auto -- -- --
8/37 77 OK DN 8G Auto -- -- --
8/38 79 OK UP 4G Auto 1889949406 3081918001 295940
8/39 78 OK UP 4G Auto 486201681 2055011623 176370
8/40 32 OK DN 8G Auto -- -- --
8/41 33 OK UP 4G Auto 631288759 3408067344 94464
8/42 35 OK DN 8G Auto -- -- --
8/43 34 OK DN 8G Auto -- -- --
8/44 36 OK UP 4G Auto 1696956891 335721461 3
8/45 37 OK UP 4G Auto 1530495808 3596473303 4180112
8/46 39 OK UP 8G Auto 170599381 135594176 639303
8/47 38 OK DN 8G Auto -- -- --

Total Diag Frames Tx: 800
Total Diag Frames Rx: 800

Slot: 3 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
3/0 32 OK DN 8Gbps -- -- --
3/1 33 OK DN 8Gbps -- -- --
3/2 34 OK DN 8Gbps -- -- --
3/3 35 OK DN 8Gbps -- -- --
3/4 36 OK DN 8Gbps -- -- --
3/5 37 OK DN 8Gbps -- -- --
3/6 38 OK DN 8Gbps -- -- --
3/7 39 OK DN 8Gbps -- -- --
3/8 72 OK DN 8Gbps -- -- --
3/9 73 OK DN 8Gbps -- -- --
3/10 74 OK DN 8Gbps -- -- --
3/11 75 OK DN 8Gbps -- -- --
3/12 76 OK DN 8Gbps -- -- --
3/13 77 OK DN 8Gbps -- -- --
3/14 78 OK DN 8Gbps -- -- --
3/15 79 OK DN 8Gbps -- -- --

Total Diag Frames Tx: 320
Total Diag Frames Rx: 320

Slot: 6 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
6/0 32 OK DN 8Gbps -- -- --
6/1 33 OK DN 8Gbps -- -- --
6/2 34 OK DN 8Gbps -- -- --
6/3 35 OK DN 8Gbps -- -- --
6/4 36 OK DN 8Gbps -- -- --
6/5 37 OK DN 8Gbps -- -- --
6/6 38 OK DN 8Gbps -- -- --
6/7 39 OK DN 8Gbps -- -- --
6/8 72 OK DN 8Gbps -- -- --
6/9 73 OK DN 8Gbps -- -- --
6/10 74 OK DN 8Gbps -- -- --
6/11 75 OK DN 8Gbps -- -- --
6/12 76 OK DN 8Gbps -- -- --
6/13 77 OK DN 8Gbps -- -- --
6/14 78 OK DN 8Gbps -- -- --
6/15 79 OK DN 8Gbps -- -- --

Central Memory: OK
Total Diag Frames Tx: 320
Total Diag Frames Rx: 320
dcx2:FID128:root>

what are the LLI Errs?

alexey.stepanov · ‎03-20-2011

lli is low level interrupts, whatever it means :)

so it looks like you have a blade chassis attached and something is going wrong. you'd better look at the errors with portstatsshow. i'd also login into the access gateway switch and look what's going in there. from experience, the copper ports connected to the blades are the first suspects as the source of all errors.

Anonymous · ‎03-23-2014

I do not have an access gateway but ere are the portstatsshow on the 4 FC ports

dcx2:FID128:root> portstatsshow 2/23

stat_wtx 3240493428 4-byte words transmitted

stat_wrx 4120706335 4-byte words received

stat_ftx 3462882141 Frames transmitted

stat_frx 1371763577 Frames received

stat_c2_frx 0 Class 2 frames received

stat_c3_frx 1371763577 Class 3 frames received

stat_lc_rx 0 Link control frames received

stat_mc_rx 0 Multicast frames received

stat_mc_to 0 Multicast timeouts

stat_mc_tx 0 Multicast frames transmitted

tim_rdy_pri 0 Time R_RDY high priority

tim_txcrd_z 15460 Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc 0- 3: 0 0 0 0

tim_txcrd_z_vc 4- 7: 0 15460 0 0

tim_txcrd_z_vc 8-11: 0 0 0 0

tim_txcrd_z_vc 12-15: 0 0 0 0

er_enc_in 0 Encoding errors inside of frames

er_crc 0 Frames with CRC errors

er_trunc 0 Frames shorter than minimum

er_toolong 0 Frames longer than maximum

er_bad_eof 0 Frames with bad end-of-frame

er_enc_out 1728881 Encoding error outside of frames

er_bad_os 1902230 Invalid ordered set

er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout

er_tx_c3_timeout 0 Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach 0 Class 3 frames discarded due to destination unreachable

er_other_discard 36 Other discards

er_type1_miss 0 frames with FTB type 1 miss

er_type2_miss 0 frames with FTB type 2 miss

er_type6_miss 0 frames with FTB type 6 miss

er_zone_miss 0 frames with hard zoning miss

er_lun_zone_miss 0 frames with LUN zoning miss

er_crc_good_eof 0 Crc error with good eof

er_inv_arb 0 Invalid ARB

open 0 loop_open

transfer 0 loop_transfer

opened 0 FL_Port opened

starve_stop 0 tenancies stopped due to starvation

fl_tenancy 0 number of times FL has the tenancy

nl_tenancy 0 number of times NL has the tenancy

zero_tenancy 0 zero tenancy

dcx2:FID128:root> portstatsshow 8/23

stat_wtx 2663765255 4-byte words transmitted

stat_wrx 3677335610 4-byte words received

stat_ftx 2027301314 Frames transmitted

stat_frx 3549592438 Frames received

stat_c2_frx 0 Class 2 frames received

stat_c3_frx 3549249091 Class 3 frames received

stat_lc_rx 160458 Link control frames received

stat_mc_rx 0 Multicast frames received

stat_mc_to 0 Multicast timeouts

stat_mc_tx 0 Multicast frames transmitted

tim_rdy_pri 44 Time R_RDY high priority

tim_txcrd_z 8673523 Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc 0- 3: 8451111 0 99773 108343

tim_txcrd_z_vc 4- 7: 0 14296 0 0

tim_txcrd_z_vc 8-11: 0 0 0 0

tim_txcrd_z_vc 12-15: 0 0 0 0

er_enc_in 0 Encoding errors inside of frames

er_crc 0 Frames with CRC errors

er_trunc 0 Frames shorter than minimum

er_toolong 0 Frames longer than maximum

er_bad_eof 0 Frames with bad end-of-frame

er_enc_out 797647 Encoding error outside of frames

er_bad_os 855209 Invalid ordered set

er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout

er_tx_c3_timeout 0 Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach 0 Class 3 frames discarded due to destination unreachable

er_other_discard 4 Other discards

er_type1_miss 0 frames with FTB type 1 miss

er_type2_miss 0 frames with FTB type 2 miss

er_type6_miss 0 frames with FTB type 6 miss

er_zone_miss 0 frames with hard zoning miss

er_lun_zone_miss 0 frames with LUN zoning miss

er_crc_good_eof 0 Crc error with good eof

er_inv_arb 0 Invalid ARB

open 0 loop_open

transfer 0 loop_transfer

opened 0 FL_Port opened

starve_stop 0 tenancies stopped due to starvation

fl_tenancy 0 number of times FL has the tenancy

nl_tenancy 0 number of times NL has the tenancy

zero_tenancy 0 zero tenancy

dcx2:FID128:root>

----------------------------------------------------------------

dcx1:FID128:admin> portstatsshow 2/23

stat_wtx 2472858760 4-byte words transmitted

stat_wrx 1254815634 4-byte words received

stat_ftx 2209445671 Frames transmitted

stat_frx 3483937250 Frames received

stat_c2_frx 0 Class 2 frames received

stat_c3_frx 3483915404 Class 3 frames received

stat_lc_rx 10923 Link control frames received

stat_mc_rx 0 Multicast frames received

stat_mc_to 0 Multicast timeouts

stat_mc_tx 0 Multicast frames transmitted

tim_rdy_pri 341 Time R_RDY high priority

tim_txcrd_z 1088068 Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc 0- 3: 3234 0 931403 7147

tim_txcrd_z_vc 4- 7: 0 146284 0 0

tim_txcrd_z_vc 8-11: 0 0 0 0

tim_txcrd_z_vc 12-15: 0 0 0 0

er_enc_in 0 Encoding errors inside of frames

er_crc 0 Frames with CRC errors

er_trunc 0 Frames shorter than minimum

er_toolong 0 Frames longer than maximum

er_bad_eof 0 Frames with bad end-of-frame

er_enc_out 770377 Encoding error outside of frames

er_bad_os 706460 Invalid ordered set

er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout

er_tx_c3_timeout 0 Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach 0 Class 3 frames discarded due to destination unreachable

er_other_discard 19 Other discards

er_type1_miss 0 frames with FTB type 1 miss

er_type2_miss 0 frames with FTB type 2 miss

er_type6_miss 0 frames with FTB type 6 miss

er_zone_miss 0 frames with hard zoning miss

er_lun_zone_miss 0 frames with LUN zoning miss

er_crc_good_eof 0 Crc error with good eof

er_inv_arb 0 Invalid ARB

open 0 loop_open

transfer 0 loop_transfer

opened 0 FL_Port opened

starve_stop 0 tenancies stopped due to starvation

fl_tenancy 0 number of times FL has the tenancy

nl_tenancy 0 number of times NL has the tenancy

zero_tenancy 0 zero tenancy

dcx1:FID128:admin> portstatsshow 8/23

stat_wtx 1737188603 4-byte words transmitted

stat_wrx 1042960656 4-byte words received

stat_ftx 3317111599 Frames transmitted

stat_frx 1440113498 Frames received

stat_c2_frx 0 Class 2 frames received

stat_c3_frx 1440113498 Class 3 frames received

stat_lc_rx 0 Link control frames received

stat_mc_rx 0 Multicast frames received

stat_mc_to 0 Multicast timeouts

stat_mc_tx 0 Multicast frames transmitted

tim_rdy_pri 0 Time R_RDY high priority

tim_txcrd_z 870 Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc 0- 3: 0 0 0 0

tim_txcrd_z_vc 4- 7: 0 870 0 0

tim_txcrd_z_vc 8-11: 0 0 0 0

tim_txcrd_z_vc 12-15: 0 0 0 0

er_enc_in 0 Encoding errors inside of frames

er_crc 0 Frames with CRC errors

er_trunc 0 Frames shorter than minimum

er_toolong 0 Frames longer than maximum

er_bad_eof 0 Frames with bad end-of-frame

er_enc_out 0 Encoding error outside of frames

er_bad_os 3 Invalid ordered set

er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout

er_tx_c3_timeout 0 Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach 0 Class 3 frames discarded due to destination unreachable

er_other_discard 2 Other discards

er_type1_miss 0 frames with FTB type 1 miss

er_type2_miss 0 frames with FTB type 2 miss

er_type6_miss 0 frames with FTB type 6 miss

er_zone_miss 0 frames with hard zoning miss

er_lun_zone_miss 0 frames with LUN zoning miss

er_crc_good_eof 0 Crc error with good eof

er_inv_arb 0 Invalid ARB

open 0 loop_open

transfer 0 loop_transfer

opened 0 FL_Port opened

starve_stop 0 tenancies stopped due to starvation

fl_tenancy 0 number of times FL has the tenancy

nl_tenancy 0 number of times NL has the tenancy

zero_tenancy 0 zero tenancy

dcx1:FID128:admin>

felipon · ‎06-11-2010

hi there,

er_bad_os and er_enc_out can indicate a HW issue. please, clear the stats (commands statsclear and slotstatsclear) and 24 hours after get a new output of portshow and portstatsclear.

alexey.stepanov · ‎03-20-2011

you see, anything that is being sent through the fibre should be either an ordered set or a frame. in your case, we can see lots of errors of these two kinds: 1) bad ordered set, 2) error outside of frame. so what means "outside of frame"? i might be wrong, but if you miss the "start of frame" - then the rest of the frame is detected as "errors outside of frame". all this basically means that these four ports transmit lots of "garbage". since it is not very likely that all connected sfps and/or cables became bad, i suspect that you have some "system" kind of the failure, when all four ports connected to the same (or same kind) software npiv solution (since you said this is not access gateway) are giving the same errors. i believe you should seek support from the vendor who is supporting your device(s) connected to these four ports. i'd first suspect hba firmware and drivers...

alexey.stepanov · ‎03-20-2011

isn't it cisco ucs?

Anonymous · ‎03-25-2014

yes it is ucs. any idea? any suggestions

thanks

Anonymous · ‎03-26-2014

Run the following and post here:

sfpshow -all | grep Rx

switchshow

porterrshow

portcfgshow

Products

Solutions

Education

Tools

Technology

Technology

Products

Tools

Fibre Channel (SAN)

Link_failure

Re: Link_failure

Re: Link_failure

Re: Link_failure

Re: Link_failure

Re: Link_failure

Re: Link_failure

Re: Link_failure

Re: Link_failure

Re: Link_failure

Join the Broadcom Support Community