For more details, please see ourCookie Policy.


Fibre Channel (SAN)

Reply
External Moderator
Posts: 5,620
Registered: ‎02-23-2004

Re: Rapid increasing er_bad_os at 8 Gbit speed

Andreas, they already gave themselves the answer...

--->>>in our case it is not related to media, cabel or SFP.

"It is related to 8 gbit compatiblity issues."

TechHelp24
External Moderator
Posts: 5,620
Registered: ‎02-23-2004

Re: Rapid increasing er_bad_os at 8 Gbit speed

Andreas, they already gave themselves the answer...

--->>>in our case it is not related to media, cabel or SFP.

"It is related to 8 gbit compatiblity issues."

TechHelp24
Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

simply tell them to change the CHA board, or wait till the Microcode upgrade

Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

No need to replace hardware.  Simply upgrade to a FOS level with the portcfgfillword command.  (I think it was introduced somewhere in 6.2)  Brocade defaults to ARBff, some storage devices still expect six IDLEs between frames, and their state machines fail if those IDLEs aren't received.  portcfgfillword <port number>, 0 and you will be all fixed.

This problem CAN cause data flow issues due to excess interrupts in HBAs.  I've seen it on some QL models.

Note that the fill word was changed for good reasons (ARBff improves signal timing), so the latest FOS versions allow you send six IDLEs to satisfy the state machine, and then send ARBff's.  So, you get the IDLEs for devices that require them, and you get the improved signal characteristics of ARBff.

Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

I do not have any idea about portcfgfillword because I have not used it. But at HITACHI level microcode upgrade may resolve the issue.

Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

Hi

I had the same problem with my DCX and 48000 (FC8-48), and I suggest not to ignored this counters, even though it is not related to data transfer but more to link signal and sync. I’d contact my support provider (HP) and the only way they resolve it was re-configured the fill word port mode from 0 to 3.

portcfgfillword slot/port, 0 iddle

portcfgfillword slot/port, 3 arbff if failed use iddle for devices that expect this signal

Note: that if you run this command It will reset the port (disable-enable) so make sure that you servers have more than one path to the target

This configuration allow devices to used arbff as primitive signal or devices that expect idlle primitive signal (like and auto-negociation).

As for technical reason it is related to Electromagnetic Interferance(EMI) and protocol (see T11 updated documentation)

From T11 org

The following FC-FS-2 proposal is for the purpose of reducing EMI with 8G and higher serial link . It replaces IDLE with the use of ARB(FF) which has a lower transition density. This allows the reduction of EMI without more significant changes that would involve randomizing the data pattern.

I suggest that you contact your support provider

If I may had a comment i used the er_bad_os  to identify bad devices like SFP, Cable, lost db, (fc analyzer), or any

Hop this will help.

Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

Hello,

thanks for your answer. But in our case the affected storage vendor doesn't support ARB(FF) on the switch side. I assume that the FC ASIC on the storage side is not well coded and tested.

Regards,

Andreas

Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

Hi,

I have had the same problem. After making sure I had the correct fill word (according to the manufacturer of the arrays in question), I found the problem disappeared when replacing the cabling that was going via a path panel by a brand-new direct attached cable.

I agree the errors do not necessarily suggest a cabling problem, but that is how we solved it.

Alastair

Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

I am facing similar issues on a 48000, whereby er_bad_os is increasing and the ports are flapping as well. Some other ports are facing the same issues as well. Any idea why this is happening?

Firmware : v6.2.2a

Speed:    N4   Online (4Gbps)

portstatsshow 4/1
stat_wtx                915040180   4-byte words transmitted
stat_wrx                2406319128  4-byte words received
stat_ftx                12732318    Frames transmitted
stat_frx                683736493   Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             683736493   Class 3 frames received
stat_lc_rx              0           Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             318         Time BB credit zero (2.5Us ticks)
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              84          Encoding error outside of frames
er_bad_os               11245       Invalid ordered set
er_c3_timeout           2366        Class 3 frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_zone_discard         0           Class 3 frames discarded due to zone mismatch
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy


portcfgshow 4/1
Area Number:              49
Speed Level:              AUTO(HW)
Fill Word:                0(Idle-Idle)
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             OFF
VC Link Init              OFF
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
ISL R_RDY Mode            OFF
RSCN Suppressed           OFF
Persistent Disable        OFF
NPIV capability           ON
QOS E_Port                OFF
Port Auto Disable:        OFF
Mirror Port               OFF
F_Port Buffers            OFF

sfpshow 4/1

Identifier:  3    SFP

Connector:   7    LC

Transceiver: 150c402001000000 100,200,400_MB/s M5,M6 sw Inter_dist

Encoding:    1    8B10B

Baud Rate:   42   (units 100 megabaud)

Length 9u:   0    (units km)

Length 9u:   0    (units 100 meters)

Length 50u:  15   (units 10 meters)

Length 62.5u:7    (units 10 meters)

Length Cu:   0    (units 1 meter)

Vendor Name: FINISAR CORP.

Vendor OUI:  00:90:65

Vendor PN:   FTLF8524P2BNV

Vendor Rev:  A

Wavelength:  850  (units nm)

Options:     0032 Loss_of_Sig,Tx_Disable

BR Max:      0

BR Min:      0

Serial No:   UA80ZN9

Date Code:   060821

Temperature: 34 Centigrade

Current:     6.228 mAmps

Voltage:     3291.9 mVolts

RX Power:    -5.5  dBm (281.5 uWatts)

TX Power:    -4.3  dBm (372.3 uWatts)

Anonymous
Posts: 0

Re: Rapid increasing er_bad_os at 8 Gbit speed

In your case I would say you have some issues with discards in your fabric.

Check your ports for this counter: er_c3_timeout.

You will see on affected ports that servers will have IO errors and a performance issue.

I thinks it is a serious issue to have dicards in the fabric.

As an update: With FOS 6.3.2 Brocade shows the &quot;discards direction&quot; if it is on the TX or RX side

This is cool :-)

Regards,

Andreas

Join the Broadcom Support Community

Get quick and easy access to valuable resources across the Broadcom Community Network.