3 Ports stucked in G_Port after QOS disabled

[Anonymous]

Hi All,

I am facing an issue with 3 ports whereby the QOS was disabled automatically during the firmware upgrade from v6.1.0c ---> v6.2.2a but on HDLM its online. On the switch, these 3 ports are flapping heavily. Anyone encountered this before?

errdump:

2010/08/28-12:40:09, , 72758, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 123) QOS is disabled

2010/08/28-12:40:10, , 72759, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 107) QOS is disabled

2010/08/28-12:41:19, , 72760, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 90) QOS is disabled

2010/08/28-12:41:19, , 72761, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 123) QOS is disabled

2010/08/28-12:41:20, , 72762, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 107) QOS is disabled

2010/08/28-12:41:56, , 72763, SLOT 5 | FID 128, INFO, <switchname>, All RADIUS servers have failed to respond.

2010/08/28-12:42:29, , 72764, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 90) QOS is disabled

2010/08/28-12:42:29, , 72765, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 123) QOS is disabled

2010/08/28-12:42:30, , 72766, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 107) QOS is disabled

2010/08/28-12:43:39, , 72767, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 90) QOS is disabled

2010/08/28-12:43:39, , 72768, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 123) QOS is disabled

2010/08/28-12:43:40, , 72769, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 107) QOS is disabled

2010/08/28-12:44:26, , 72770, SLOT 5 | FID 128, INFO, <switchname>, All RADIUS servers have failed to respond.

2010/08/28-12:44:49, , 72771, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 90) QOS is disabled

2010/08/28-12:44:50, , 72772, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 123) QOS is disabled

2010/08/28-12:44:51, , 72773, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 107) QOS is disabled

2010/08/28-12:45:59, , 72774, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 90) QOS is disabled

2010/08/28-12:46:00, , 72775, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 123) QOS is disabled

2010/08/28-12:46:01, , 72776, SLOT 5 | FID 128, WARNING, <switchname>, Port (ID: 107) QOS is disabled

2010/08/28-12:46:57, , 72777, SLOT 5 | FID 128, INFO, <switchname>, All RADIUS servers have failed to respond.

portshow 8/10 (90):

<switchname>:marmugam> portshow 8/10

portName:

portHealth: HEALTHY

Authentication: None

portDisableReason: None

portCFlags: 0x1

portFlags: 0x4103        PRESENT ACTIVE G_PORT U_PORT LOGIN LED

portType:  10.0

portState: 1    Online

portPhys:  6    In_Sync

portScn:   1    Online

port generation number:    352

portId:    015a00

portIfId:    4382001e

portWwn:   20:5a:00:05:1e:36:48:a6

portWwn of device(s) connected:

Distance:  normal

portSpeed: N4Gbps

LE domain: 0

FC Fastwrite: OFF

Interrupts:        380        Link_failure: 34         Frjt:         0

Unknown:           0          Loss_of_sync: 1          Fbsy:         0

Lli:               380        Loss_of_sig:  2

Proc_rqrd:         0          Protocol_err: 0

Timed_out:         0          Invalid_word: 946

Rx_flushed:        0          Invalid_crc:  0

Tx_unavail:        0          Delim_err:    0

Free_buffer:       0          Address_err:  0

Overrun:           0          Lr_in:        35

Suspended:         0          Lr_out:       272

Parity_err:        0          Ols_in:       34

2_parity_err:      0          Ols_out:      35

CMI_bus_err:       0

Port part of other ADs: No

<switchname>:marmugam>

fabriclog -s

13:08:49.812 SCN LR_PORT (0);g=0x15c                     A1,P1  A1,P1  123   NA

13:08:50.742 SCN LR_PORT (0);g=0x15e                     A1,P1  A1,P1  107   NA

13:08:56.831 ELP TO Rcv                                  A1,P1  A1,P1  90    NA

13:08:56.831 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  90    0x4769

13:08:56.831 ELP Retry 6 times                           A1,P1  A1,P1  90    0x4769

13:09:00.131 ELP TO Rcv                                  A1,P1  A1,P1  123   NA

13:09:00.131 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  123   0x47dc

13:09:00.131 ELP Retry 5 times                           A1,P1  A1,P1  123   0x47dc

13:09:00.703 SCN Port Offline;g=0x160                    A1,P1  A1,P0  90    NA

13:09:00.703 *Removing all nodes from port               A1,P0  A1,P0  90    NA

13:09:00.713 SCN LR_PORT (0);g=0x160                     A1,P0  A1,P0  90    NA

13:09:00.852 SCN Port Online;g=0x160                     A1,P0  A1,P1  90    NA

13:09:00.917 Send ELP requesting VC RDY + ARB mode       A1,P1  A1,P1  90    NA

13:09:00.918 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  90    0x47f8

13:09:01.011 ELP TO Rcv                                  A1,P1  A1,P1  107   NA

13:09:01.011 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  107   0x47ff

13:09:01.011 ELP Retry 5 times                           A1,P1  A1,P1  107   0x47ff

13:09:03.812 SCN LR_PORT (0);g=0x15c                     A1,P1  A1,P1  123   NA

13:09:04.602 SCN LR_PORT (0);g=0x160                     A1,P1  A1,P1  90    NA

13:09:04.742 SCN LR_PORT (0);g=0x15e                     A1,P1  A1,P1  107   NA

13:09:06.911 ELP TO Rcv                                  A1,P1  A1,P1  90    NA

13:09:06.911 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  90    0x48d7

13:09:06.911 ELP Retry 1 times                           A1,P1  A1,P1  90    0x48d7

13:09:10.603 SCN LR_PORT (0);g=0x160                     A1,P1  A1,P1  90    NA

13:09:14.865 ELP TO Rcv                                  A1,P1  A1,P1  90    NA

13:09:14.865 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  90    0x4a12

13:09:14.865 ELP Retry 2 times                           A1,P1  A1,P1  90    0x4a12

13:09:16.082 ELP TO Rcv                                  A1,P1  A1,P1  123   NA

13:09:16.082 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  123   0x357e

13:09:16.082 ELP Retry 6 times                           A1,P1  A1,P1  123   0x357e

13:09:16.963 ELP TO Rcv                                  A1,P1  A1,P1  107   NA

13:09:16.964 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  107   0x4a5c

13:09:16.964 ELP Retry 6 times                           A1,P1  A1,P1  107   0x4a5c

13:09:18.602 SCN LR_PORT (0);g=0x160                     A1,P1  A1,P1  90    NA to respond.

[Anonymous]

This query should be posted under datacentre .

Anyway has QOS been enabled before?

In portshow, alots of errors are there.

could you pls do a portstatsclear on the port and put the output of portshow, portstatsshow, porterrshow , switchstatusshow.

if on HDLM there is no issue, you can ignore the message.

Which are the devices, connected to these ports? Have you checked the FOS compatibility with HBAs and microcodes of the storages befor edoing an upgrade?

In fabriclog it shows :

A1 means Send DIA: After the Switch is granted a Domain_ID, it shall then transmit a DIA SW_ILS request sequence via all ISL other than the Principal ISL

Pssible Value for P,PN are:

P0 Port offline or initial state
P1 Port online

So ur port no 90 shows sometimes P0 means offline or initial state.Also it is in G port. You can do a portdisable/enable.

But there is nothing shown with 107/123 in fabriclog

[ABBA-SYSTEMS]

--->>> I am facing an issue with 3 ports whereby the QOS was disabled automatically during the firmware upgrade from v6.1.0c ---> v6.2.2a

This is not a issue, no behavior, and not a Bug, but caused due Change in the FOS.

QOS was Enable as default in Fabric OS until rel. v6.2.1

Customer want to Upgrade from 6.1.x to/higher as v6.2.1 brocade reccomended to Disable QOS before start Upgrade, and enable after firmwaredownload is completed.

Since FOS 6.2.2 QOS is Disable by default.

Summary,...

What happened ? With firmwaredownload to v6.2.2a QOS was set as Disable.

If you are satisfied with the responses please mark as 'Answered'.
If not, just let me know I can see to get more info.
I would appreciated if you rate the threads.

Thank you.

[Anonymous]

switchstatusshow:

Switch Health Report                        Report time: 08/28/2010 01:58:56 PM
Switch Name:    amsd2adr0001
IP address:     145.59.174.12
SwitchState:    HEALTHY
Duration:       01:45

Power supplies monitor  HEALTHY
Temperatures monitor    HEALTHY
Fans monitor            HEALTHY
WWN servers monitor     HEALTHY
Standby CP monitor      HEALTHY
Blades monitor          HEALTHY
Flash monitor           HEALTHY
Marginal ports monitor  HEALTHY
Faulty ports monitor    HEALTHY
Missing SFPs monitor    HEALTHY

All ports are healthy

--------------------------------------------------------------------------------------------------------------

portshow 9/11(107)

portName:
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x4103        PRESENT ACTIVE G_PORT U_PORT LOGIN LED
portType:  10.0
portState: 1    Online
portPhys:  6    In_Sync
portScn:   1    Online
port generation number:    620
portId:    016b00
portIfId:    4392001a
portWwn:   20:6b:00:05:1e:36:48:a6
portWwn of device(s) connected:

Distance:  normal
portSpeed: N4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        34         Link_failure: 3          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               34         Loss_of_sig:  0
Proc_rqrd:         0          Protocol_err: 0
Timed_out:         0          Invalid_word: 87
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        3
Suspended:         0          Lr_out:       25
Parity_err:        0          Ols_in:       3
2_parity_err:      0          Ols_out:      3
CMI_bus_err:       0

Port part of other ADs: No

portstatsshow 9/11
stat_wtx                1400        4-byte words transmitted
stat_wrx                0           4-byte words received
stat_ftx                28          Frames transmitted
stat_frx                0           Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             0           Class 3 frames received
stat_lc_rx              0           Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             0           Time BB credit zero (2.5Us ticks)
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              112         Encoding error outside of frames
er_bad_os               0           Invalid ordered set
er_c3_timeout           0           Class 3 frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_zone_discard         0           Class 3 frames discarded due to zone mismatch
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

----------------------------------------------------------------------------

portshow 8/10

portName:

portHealth: HEALTHY

Authentication: None

portDisableReason: None

portCFlags: 0x1

portFlags: 0x4103        PRESENT ACTIVE G_PORT U_PORT LOGIN LED

portType:  10.0

portState: 1    Online

portPhys:  6    In_Sync

portScn:   1    Online

port generation number:    628

portId:    015a00

portIfId:    4382001e

portWwn:   20:5a:00:05:1e:36:48:a6

portWwn of device(s) connected:

Distance:  normal

portSpeed: N4Gbps

LE domain: 0

FC Fastwrite: OFF

Interrupts:        50         Link_failure: 4          Frjt:         0

Unknown:           0          Loss_of_sync: 0          Fbsy:         0

Lli:               50         Loss_of_sig:  0

Proc_rqrd:         0          Protocol_err: 0

Timed_out:         0          Invalid_word: 132

Rx_flushed:        0          Invalid_crc:  0

Tx_unavail:        0          Delim_err:    0

Free_buffer:       0          Address_err:  0

Overrun:           0          Lr_in:        4

Suspended:         0          Lr_out:       38

Parity_err:        0          Ols_in:       4

2_parity_err:      0          Ols_out:      4

CMI_bus_err:       0

Port part of other ADs: No

portstatsshow 8/10

stat_wtx                1700        4-byte words transmitted

stat_wrx                0           4-byte words received

stat_ftx                34          Frames transmitted

stat_frx                0           Frames received

stat_c2_frx             0           Class 2 frames received

stat_c3_frx             0           Class 3 frames received

stat_lc_rx              0           Link control frames received

stat_mc_rx              0           Multicast frames received

stat_mc_to              0           Multicast timeouts

stat_mc_tx              0           Multicast frames transmitted

tim_rdy_pri             0           Time R_RDY high priority

tim_txcrd_z             0           Time BB credit zero (2.5Us ticks)

er_enc_in               0           Encoding errors inside of frames

er_crc                  0           Frames with CRC errors

er_trunc                0           Frames shorter than minimum

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              132         Encoding error outside of frames

er_bad_os               0           Invalid ordered set

er_c3_timeout           0           Class 3 frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        0           Other discards

er_zone_discard         0           Class 3 frames discarded due to zone mismatch

er_crc_good_eof         0           Crc error with good eof

er_inv_arb              0           Invalid ARB

open                    0           loop_open

transfer                0           loop_transfer

opened                  0           FL_Port opened

starve_stop             0           tenancies stopped due to starvation

fl_tenancy              0           number of times FL has the tenancy

nl_tenancy              0           number of times NL has the tenancy

zero_tenancy            0           zero tenancy

---------------------------------------------------------------------------------------

portshow 10/11

portName:

portHealth: HEALTHY

Authentication: None

portDisableReason: None

portCFlags: 0x1

portFlags: 0x4103        PRESENT ACTIVE G_PORT U_PORT LOGIN LED

portType:  10.0

portState: 1    Online

portPhys:  6    In_Sync

portScn:   1    Online

port generation number:    636

portId:    017b00

portIfId:    43a2001a

portWwn:   20:7b:00:05:1e:36:48:a6

portWwn of device(s) connected:

Distance:  normal

portSpeed: N4Gbps

LE domain: 0

FC Fastwrite: OFF

Interrupts:        66         Link_failure: 6          Frjt:         0

Unknown:           0          Loss_of_sync: 0          Fbsy:         0

Lli:               66         Loss_of_sig:  0

Proc_rqrd:         0          Protocol_err: 0

Timed_out:         0          Invalid_word: 161

Rx_flushed:        0          Invalid_crc:  0

Tx_unavail:        0          Delim_err:    0

Free_buffer:       0          Address_err:  0

Overrun:           0          Lr_in:        6

Suspended:         0          Lr_out:       48

Parity_err:        0          Ols_in:       6

2_parity_err:      0          Ols_out:      6

CMI_bus_err:       0

Port part of other ADs: No

portstatsshow 10/11

stat_wtx                2048        4-byte words transmitted

stat_wrx                0           4-byte words received

stat_ftx                41          Frames transmitted

stat_frx                0           Frames received

stat_c2_frx             0           Class 2 frames received

stat_c3_frx             0           Class 3 frames received

stat_lc_rx              0           Link control frames received

stat_mc_rx              0           Multicast frames received

stat_mc_to              0           Multicast timeouts

stat_mc_tx              0           Multicast frames transmitted

tim_rdy_pri             0           Time R_RDY high priority

tim_txcrd_z             0           Time BB credit zero (2.5Us ticks)

er_enc_in               0           Encoding errors inside of frames

er_crc                  0           Frames with CRC errors

er_trunc                0           Frames shorter than minimum

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              161         Encoding error outside of frames

er_bad_os               0           Invalid ordered set

er_c3_timeout           0           Class 3 frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        0           Other discards

er_zone_discard         0           Class 3 frames discarded due to zone mismatch

er_crc_good_eof         0           Crc error with good eof

er_inv_arb              0           Invalid ARB

open                    0           loop_open

transfer                0           loop_transfer

opened                  0           FL_Port opened

starve_stop             0           tenancies stopped due to starvation

fl_tenancy              0           number of times FL has the tenancy

nl_tenancy              0           number of times NL has the tenancy

zero_tenancy            0           zero tenancy

-------------------------------------------------------------------------------------------

porterrshow  | grep -e 90 -e 107 -e 123 -e frame -e tx

        frames  enc  crc  crc   too  too  bad  enc disc link loss loss frjt fbsy

       tx   rx   in  err g_eof shrt long  eof  out   c3 fail sync  sig

90:   54    0    0    0    0    0    0    0  215    0    7    0    0    0    0

107:   56    0    0    0    0    0    0    0  215    0    8    0    0    0    0

123:   55    0    0    0    0    0    0    0  216    0    8    0    0    0    0

enc_out and link fail is increasing

-----------------------------------------------------------------------

Which are the devices, connected to these ports? Have you checked the  FOS compatibility with HBAs and microcodes of the storages befor edoing  an upgrade?

The firmware upgrade was done on Fabric 1 if the issue is with the HBA we would have encountered the same issue after the firmware upgdare on Fabric 2

[Anonymous]

Hi Abba,

Yeah I was aware that the QOS will be disabled in v6.2.2 but these ports are flapping heavily and I don't think we can ignore this and the port is still in G_Port but on the server end its online. This is wierd.

Rgds,

Mahen

[Anonymous]

Have you done the portstatsclear and then see the portstatsshow? if after that enc-out is increasing, then it points to Media, You can change cable and then if it still increasing then change SFP.

Btw you can do a portdisable/enable to see the ports are F port or not.If the HBAs are a part of AIX systems, do a rmdev and cfgmgr on these HBAs.

<The firmware upgrade was done on Fabric 1 if the issue is with the HBA we would have encountered the same issue after the firmware upgdare on Fabric 2>

Not necessary it will show on both the HBAs .Sometimes it shows

Try nodefind, nsshow command

Multipathing SW is there , so u can do portenable/disable

You can Ignore QOS warning, but try to resolve the G-port issue to F-port.

[ABBA-SYSTEMS]

--->>>...but these ports are flapping heavily...

since ( I'm assuming ) you have don't disable QOS before start firmwaredownload to 6.2.2a, the ports hangs.

My Workaround, you may have to Disable the QOS, Stop those Ports "portdisable" , then restart now "portenable" and enable QOS now.

[ABBA-SYSTEMS]

@Anonymous

--->>> You can Ignore QOS warning,....

The Question is:

"3 Ports stucked in G_Port after QOS disabled"

Then is not necessary to open such Threads.

Mahendran

at this Point, you can safely close the Thread.

[Anonymous]

Hi Abba,

I tried but its not helping the counters keeps on increasing and btw why should I close the thread. We can't simply ignore the counters that are increasing.

Regards,

Mahendran

[ABBA-SYSTEMS]

--->>>...and btw why should I close the thread.

apparently you have misunderstood my post

this was just a humored answer

but back to you problem.....

--->>> I tried but its not helping the counters keeps on increasing."

I trought that port- disable/enable would help, but apparently is not.