Brocade Fibre Channel Networking Community

Expand all | Collapse all

Issues with DCX8510 CP blade's secondary partition reported after upgrade from 8.1.0 to 8.1.1

  • 1.  Issues with DCX8510 CP blade's secondary partition reported after upgrade from 8.1.0 to 8.1.1

    Posted 10-17-2017 07:14 AM

     

     

    Just now seeing error on console. any workaround?

    It seems partition on CF card is having issue. 

     

    rsync: read errors mapping "/tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm": Input/output error (5)

    dcx85108-fcs01:FID128:admin> rsync: read errors mapping "/boot/bootrom.bin": Input/output error (5)

    rsync: read errors mapping "/fabos/modules/pluto-zentron-module.ko": Input/output error (5)

    rsync: read errors mapping "/tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm": Input/output error (5)

    ERROR: boot/bootrom.bin failed verification -- update discarded.

    ERROR: fabos/modules/pluto-zentron-module.ko failed verification -- update discarded.

    ERROR: tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm failed verification -- update discarded.

    rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]

     

     

     

    ----------

     

    Here’s what it says..

     

    dcx85108-fcs01:FID128:admin> hashow

    Local CP (Slot 7, CP1): Standby

    Remote CP (Slot 6, CP0): Active

    HA enabled, Heartbeat Up, HA State synchronized

    rsync: read errors mapping "/fabos/modules/pluto-zentron-module.ko": Input/output error (5)

     

    dcx85108-fcs01:FID128:admin>

    dcx85108-fcs01:FID128:admin> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      6  CP0        FOS      v8.1.1                                   ACTIVE

                             v8.1.1

      7  CP1        FOS      v8.1.1                                   STANDBY *

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

     

     

     

     

     

    ----------

     

     

     

    FAILED again.

     

    rsync: read errors mapping "/tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm": Input/output error (5)

    rsync: read errors mapping "/boot/bootrom.bin": Input/output error (5)

    rsync: read errors mapping "/fabos/modules/pluto-zentron-module.ko": Input/output error (5)

    rsync: read errors mapping "/tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm": Input/output error (5)

    ERROR: boot/bootrom.bin failed verification -- update discarded.

    ERROR: fabos/modules/pluto-zentron-module.ko failed verification -- update discarded.

    ERROR: tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm failed verification -- update discarded.

    rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]

     

    dcx85108-fcs01:FID128:admin> rsync: read errors mapping "/boot/bootrom.bin": Input/output error (5)

     

     

     

     

    -

    ----

     

    It seems trying to recover secondary partition but then its failing.

     

    --- Partition /dev/hda1 is inconsistent.

     

    dcx85108-fcs01:FID128:admin> rsync: read errors mapping "/fabos/modules/pluto-zentron-module.ko": Input/output error (5)

    Service instances out of sync

     

    [64]: Mon Oct 16 17:01:55 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [65]: Mon Oct 16 17:06:31 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [66]: Mon Oct 16 17:06:31 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    dcx85108-fcs01:FID128:admin> Repairing the secondary partition now.

    Please wait ...

    rsync: read errors mapping "/boot/bootrom.bin": Input/output error (5)

     

     

     

     

    dcx85108-fcs01:FID128:admin> hashow

    Local CP (Slot 6, CP0): Standby

    Remote CP (Slot 7, CP1): Active

    HA enabled, Heartbeat Up, HA State synchronized

    dcx85108-fcs01:FID128:admin> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      6  CP0        FOS      v8.1.1                                   STANDBY *

                             v8.1.1

      7  CP1        FOS      v8.1.1                                   ACTIVE

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

    dcx85108-fcs01:FID128:admin> exit

    logout

     

    Fabric OS (dcx85108-fcs01)

     

     

    telnet> quit01 console login:

    Connection to ts41h41.gdl.englab.netapp.com closed.

    bash-3.00$ rconsole dcx85108-fcs01-cp1

    dcx85108-fcs01-cp1(=>dcx85108-fcs01-cp1.gdl.englab.netapp.com) - telnet requested to ts41h41.gdl.englab.netapp.com 2044

    Trying 10.228.1.68...

    Connected to ts41h41.gdl.englab.netapp.com.

    Escape character is '^]'.

     

     

     

    Fabric OS (dcx85108-fcs01)

     

     

    dcx85108-fcs01 console login: admin

    Password:

    dcx85108-fcs01:FID128:admin>

    dcx85108-fcs01:FID128:admin> slotshow

     

    Slot   Blade Type     ID    Status

    -----------------------------------

      1     SW BLADE     96     ENABLED

      2     UNKNOWN             VACANT

      3     SW BLADE    153     ENABLED

      4     UNKNOWN             VACANT

      5     CORE BLADE   98     ENABLED

      6     CP BLADE     50     ENABLED

      7     CP BLADE     50     ENABLED

      8     CORE BLADE   98     ENABLED

      9     UNKNOWN             VACANT

    10     SW BLADE     97     ENABLED

    11     UNKNOWN             VACANT

    12     UNKNOWN             VACANT

     

     

    dcx85108-fcs01:FID128:admin>

    dcx85108-fcs01:FID128:admin> reboot

    Warning: This command is being run on a control processor (CP)

    based system and will cause the active CP to reboot.

    This will cause disruption to all traffic in this chassis.

    Are you sure you want to reboot the active CP [y/n]?y

    2017/10/16-17:03:42, [RAS-1007], 17074, SLOT 7 CHASSIS, INFO, Brocade_DCX, System is about to reload.

    Rebooting! Mon Oct 16 17:03:52 EDT 2017

     

    Broadcast message from root (ttyS0) Mon Oct 16 17:03:52 2017...

     

    The system is going down for reboot NOW !!

    INIT: Switching to runlevel: 6

    INIT: Sending processes the TERM signal

    dcx85108-fcs01:FID128:admin> rsync: read errors mapping "/fabos/modules/pluto-zentron-module.ko": Input/output error (5)

    Service instances out of sync

    2017/10/16-17:03:55, [FSSM-1003], 17075, SLOT 7 CHASSIS, WARNING, Brocade_DCX, HA State out of sync.

    eth0: Link is down at 31696881 jiffies

    eth3: Link is down at 31696897 jiffies

    bonding: bond0: released all slaves

    eth1: Link is down at 31696920 jiffies

    eth2: Link is down at 31696928 jiffies

    ##RSCMON exiting due to signal: 19, pending signals: 0x0, 0x0

    ##exiting due to signal: 19, pending signals: 0x0, 0x0

    Stopping diagnostics

    Unmounting all filesystems.

    Please stand by while rebooting the system...

    reboot_event: Calling bus0 flush

    Restarting system.

    .

     

    The system is coming up, please wait...

     

     

    U-Boot 1.1.3 (Jun  1 2017 - 11:38:13)

     

    CPU:   8548_E, Version: 2.1, (0x80390021)

    Core:  E500, Version: 2.2, (0x80210022)

    Clock Configuration:

           CPU:1199 MHz, CCB: 399 MHz,

           DDR: 199 MHz, LBC:  49 MHz

    L1:    D-cache 32 kB enabled

           I-cache 32 kB enabled

    Board: Zentron

    CPU Board Revision 255.198 (0xffc6)

        PCI2: disabled

    I2C:   ready

    DRAM:  initdram: Initializing DDRSDRAM

    initdram: Normal boot in progress

    dimm_spd_sdram 2001: memsize = 800

    initdram: Call ddr_enable_ecc dram size -2147483648

        DDR: 2048 MB

    POST RAM test disabled.

    Adjusted memory 0x7d000000 2097152000

    Now running in RAM - U-Boot at: 74f7d000

    trap_init : 0x0

    system inventory subsystem initialized

    FLASH:  4 MB

    L2 cache 512KB: enabled

    CPLD: Init complete

    ATA interface setup upm 256: ffcff00, ffcfc04, ffffc00, fffffc01

    PCI: gd->brcd_flags = 0, PCI init

    Skip our host bridge

            00  11  8086  b555  0680  1a

            00  12  8086  b555  0680  1a

            00  13  8086  b555  0680  1a

            00  14  1131  1561  0c03  3b

            00  14  1131  1562  0c03  3b

               CPLD rev: 0xd0

      CPU0 CPLD version: 0xba

    In:    serial

    Out:   serial

    Err:   serial

    Net:

    ENET2: PHY is not applicable

    ENET1: PHY is Broadcom BCM5461S 10/100/1000 BaseT PHY (2060c1)

    ENET0: PHY is Broadcom BCM5461S 10/100/1000 BaseT PHY (2060c1)

    ENET3: PHY is Broadcom BCM5461S 10/100/1000 BaseT PHY (2060c1)

     

    board_init_r: Adjusted ram 2097152000 0x7d000000 2048000k

    Checking system RAM - press any key to stop test

     

    Checking memory address: 00100000

     

    System RAM test using Default POST RAM Test succeeded.

     

    set_bootstatus: BS_LOAD_OS, platform_idx = 2

    Hit ESC to stop autoboot:  0

    Map file at LBA sector 0x2b8100

    do_bootm 171: load_addr 0x02000000 addr 0x00400000

    do_bootm 174: ## Booting image at 00400000 load_addr 0x02000000 ...

       Image Name:   Linux-2.6.14.2

       Image Type:   PowerPC Linux Multi-File Image (gzip compressed)

       Data Size:    2675443 Bytes =  2.6 MB

       Load Address: 00000000

       Entry Point:  00000000

       Contents:

       Image 0:  1596767 Bytes =  1.5 MB

       Image 1:  1078663 Bytes =  1 MB

       Uncompressing Multi-File Image ... do_bootm_linux 565: ## Current stack ends at 0x74F5BB38 => set upper limit to 0x00800000

    ## initrd at 0x00585DAC ... 0x0068D332 (len=1078663=0x107587)

       Loading Ramdisk to 1fef8000, end 1ffff587 ... OK

    initrd_start = 1fef8000, initrd_end = 1ffff587

    ## Transferring control to Linux (at address 00000000) ...

    tlbcam_index=11

    mpc85xx_setup: Doing Pcie bridge setup

    cpld_init: Map PCI RAS device

     

    Installing Linux 2.6 Kernel

    Attempting to find a root file system on hda2...

    INIT: version 2.78 booting

    Bypassing firmware validation.

    --- Partition /dev/hda1 is inconsistent.

    --- Its content will be restored to be the same as that of /dev/hda2.

    --- Please check the version and re-load firmware if necessary after the system boots up.

    INIT: Entering runlevel: 3

    loading bonding module

    Enabling FOS failure detection feature

    enable fabos log

    Starting Fabric OS Services...

    FIPS off, RRD...

     

     

    Enter trace_initialize

    fablog module loaded with mtracer!

     

    sys_chip_init: class 0 max 24 Done

    ethsw_ha_enabled = 1

    main-cpu in Neptune...

    HASM enable auto-reboot

     

    Detected 2 CPs in Chassis

    Probe for PCI timeout detect

    .

     

     

    Fabric OS (dcx85108-fcs01)

     

     

    dcx85108-fcs01 console login: admin

    Password:

    *****************************************************************

    Logging into STANDBY CP, not all commands are fully supported !!

    *****************************************************************

    unknown(0):FID128:admin> slotshow

    rbash: slotshow: command not found

    unknown(0):FID128:admin> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      7  CP1        FOS      v8.1.1                                   STANDBY *

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

    unknown(0):FID128:admin> Proxy: ip2ipc started (run=1, rc=0), debug=0

    Proxy: ipc2ip started (rc=0)

    HAM: HAM:ham_open_htbt:Open heartbeat: 3e0006

    HAM: HAM:ham_open_htbt:Open heartbeat: 3f0107

    Oct 16 17:05:36  Heartbeat to 3e0006 Up!

    2017/10/16-17:05:35, [HAM-1004], 17076, SLOT 7 CHASSIS, INFO, Brocade_DCX, Processor rebooted - Reboot.

     

    Exchange HA Sync between CP

     

     

     

    -------------

     

    OK let me see if I can console access for both.

     

    I have tried hafailover but it seems second partition has problem.

     

    dcx85108-fcs01:FID128:admin> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      6  CP0        FOS      v8.1.1                                   STANDBY

                             v8.1.1

      7  CP1        FOS      v8.1.1                                   ACTIVE *

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

    dcx85108-fcs01:FID128:admin> fabricshow

    Switch ID   Worldwide Name          Enet IP Addr    FC IP Addr      Name

    -------------------------------------------------------------------------

      2: fffc02 10:00:c4:f5:7c:8c:20:60 10.228.184.171  0.0.0.0         "brcdg610-lfcs02"

      3: fffc03 50:00:53:35:65:98:4f:7e 0.0.0.0         0.0.0.0         "fcr_xd_3_2"

    114: fffc72 10:00:00:05:1e:04:55:98 10.228.184.108  0.0.0.0        >"brcd5000-fcs02"

    115: fffc73 10:00:00:05:1e:34:7e:9c 10.228.184.109  0.0.0.0         "brcd4100-fcs02"

    116: fffc74 10:00:00:05:1e:44:32:00 10.228.184.110  0.0.0.0         "brcddcx-fcs02"

    117: fffc75 10:00:00:05:1e:54:0b:44 10.228.184.113  0.0.0.0         "brcdmace-fcs01"

    118: fffc76 10:00:00:05:1e:37:ac:30 10.228.184.114  0.0.0.0         "brcd7500-fcs02"

    119: fffc77 10:00:00:05:1e:35:26:30 10.228.184.115  0.0.0.0         "brcd200e-fcs02"

    120: fffc78 10:00:00:27:f8:62:b0:a0 10.228.184.231  0.0.0.0         "brcd6505-fcs03"

    121: fffc79 10:00:00:05:33:56:9c:00 10.228.184.205  0.0.0.0         "dcx85108-fcs01"

    122: fffc7a 10:00:00:05:33:56:59:00 10.228.184.208  0.0.0.0         "dcx85104-fcs01"

    123: fffc7b 10:00:00:27:f8:3d:c7:f5 10.228.184.249  0.0.0.0         "brcd6520-fcs02"

    125: fffc7d 10:00:50:eb:1a:f9:18:4c 10.228.185.167  0.0.0.0         "brcdG620-fcs02"

    127: fffc7f 10:00:c4:f5:7c:2d:35:08 10.228.184.214  172.16.0.61     "brcdx6-4-fcs02"

    160: fffca0 50:00:53:35:65:90:1e:7f 0.0.0.0         0.0.0.0         "fcr_fd_160"

     

    The Fabric has 15 switches

     

     

    dcx85108-fcs01:FID128:admin>

    dcx85108-fcs01:FID128:admin> reboot

    Warning: This command is being run on a control processor (CP)

    based system and will cause the active CP to reboot.

    This will cause disruption to all traffic in this chassis.

    Are you sure you want to reboot the active CP [y/n]?y

    2017/10/16-17:03:42, [RAS-1007], 17074, SLOT 7 CHASSIS, INFO, Brocade_DCX, System is about to reload.

    Rebooting! Mon Oct 16 17:03:52 EDT 2017

     

    Broadcast message from root (ttyS0) Mon Oct 16 17:03:52 2017...

     

    The system is going down for reboot NOW !!

    INIT: Switching to runlevel: 6

    INIT: Sending processes the TERM signal

    dcx85108-fcs01:FID128:admin> rsync: read errors mapping "/fabos/modules/pluto-zentron-module.ko": Input/output error (5)

    Service instances out of sync

    2017/10/16-17:03:55, [FSSM-1003], 17075, SLOT 7 CHASSIS, WARNING, Brocade_DCX, HA State out of sync.

    eth0: Link is down at 31696881 jiffies

    eth3: Link is down at 31696897 jiffies

    bonding: bond0: released all slaves

    eth1: Link is down at 31696920 jiffies

    eth2: Link is down at 31696928 jiffies

    ##RSCMON exiting due to signal: 19, pending signals: 0x0, 0x0

    ##exiting due to signal: 19, pending signals: 0x0, 0x0

    Stopping diagnostics

    Unmounting all filesystems.

    Please stand by while rebooting the system...

    reboot_event: Calling bus0 flush

    Restarting system.

    .

     

    The system is coming up, please wait...

     

     

    U-Boot 1.1.3 (Jun  1 2017 - 11:38:13)

     

    CPU:   8548_E, Version: 2.1, (0x80390021)

    Core:  E500, Version: 2.2, (0x80210022)

    Clock Configuration:

           CPU:1199 MHz, CCB: 399 MHz,

           DDR: 199 MHz, LBC:  49 MHz

    L1:    D-cache 32 kB enabled

           I-cache 32 kB enabled

    Board: Zentron

    CPU Board Revision 255.198 (0xffc6)

        PCI2: disabled

    I2C:   ready

    DRAM:  initdram: Initializing DDRSDRAM

    initdram: Normal boot in progress

    dimm_spd_sdram 2001: memsize = 800

    initdram: Call ddr_enable_ecc dram size -2147483648

        DDR: 2048 MB

    POST RAM test disabled.

    Adjusted memory 0x7d000000 2097152000

    Now running in RAM - U-Boot at: 74f7d000

    trap_init : 0x0

    system inventory subsystem initialized

    FLASH:  4 MB

    L2 cache 512KB: enabled

    CPLD: Init complete

    ATA interface setup upm 256: ffcff00, ffcfc04, ffffc00, fffffc01

    PCI: gd->brcd_flags = 0, PCI init

    Skip our host bridge

            00  11  8086  b555  0680  1a

            00  12  8086  b555  0680  1a

            00  13  8086  b555  0680  1a

            00  14  1131  1561  0c03  3b

            00  14  1131  1562  0c03  3b

               CPLD rev: 0xd0

      CPU0 CPLD version: 0xba

    In:    serial

    Out:   serial

    Err:   serial

    Net:

    ENET2: PHY is not applicable

    ENET1: PHY is Broadcom BCM5461S 10/100/1000 BaseT PHY (2060c1)

    ENET0: PHY is Broadcom BCM5461S 10/100/1000 BaseT PHY (2060c1)

    ENET3: PHY is Broadcom BCM5461S 10/100/1000 BaseT PHY (2060c1)

     

    board_init_r: Adjusted ram 2097152000 0x7d000000 2048000k

    Checking system RAM - press any key to stop test

     

    Checking memory address: 00100000

     

    System RAM test using Default POST RAM Test succeeded.

     

    set_bootstatus: BS_LOAD_OS, platform_idx = 2

    Hit ESC to stop autoboot:  0

    Map file at LBA sector 0x2b8100

    do_bootm 171: load_addr 0x02000000 addr 0x00400000

    do_bootm 174: ## Booting image at 00400000 load_addr 0x02000000 ...

       Image Name:   Linux-2.6.14.2

       Image Type:   PowerPC Linux Multi-File Image (gzip compressed)

       Data Size:    2675443 Bytes =  2.6 MB

       Load Address: 00000000

       Entry Point:  00000000

       Contents:

       Image 0:  1596767 Bytes =  1.5 MB

       Image 1:  1078663 Bytes =  1 MB

       Uncompressing Multi-File Image ... do_bootm_linux 565: ## Current stack ends at 0x74F5BB38 => set upper limit to 0x00800000

    ## initrd at 0x00585DAC ... 0x0068D332 (len=1078663=0x107587)

       Loading Ramdisk to 1fef8000, end 1ffff587 ... OK

    initrd_start = 1fef8000, initrd_end = 1ffff587

    ## Transferring control to Linux (at address 00000000) ...

    tlbcam_index=11

    mpc85xx_setup: Doing Pcie bridge setup

    cpld_init: Map PCI RAS device

     

    Installing Linux 2.6 Kernel

    Attempting to find a root file system on hda2...

    INIT: version 2.78 booting

    Bypassing firmware validation.

    --- Partition /dev/hda1 is inconsistent.

    --- Its content will be restored to be the same as that of /dev/hda2.

    --- Please check the version and re-load firmware if necessary after the system boots up.

    INIT: Entering runlevel: 3

    loading bonding module

    Enabling FOS failure detection feature

    enable fabos log

    Starting Fabric OS Services...

    FIPS off, RRD...

     

     

    Enter trace_initialize

    fablog module loaded with mtracer!

     

    sys_chip_init: class 0 max 24 Done

    ethsw_ha_enabled = 1

    main-cpu in Neptune...

    HASM enable auto-reboot

     

    Detected 2 CPs in Chassis

    Probe for PCI timeout detect

    .

     

     

    Fabric OS (dcx85108-fcs01)

     

     

    dcx85108-fcs01 console login: admin

    Password:

    *****************************************************************

    Logging into STANDBY CP, not all commands are fully supported !!

    *****************************************************************

    unknown(0):FID128:admin> slotshow

    rbash: slotshow: command not found

    unknown(0):FID128:admin> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

     7  CP1        FOS      v8.1.1                                   STANDBY *

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

    unknown(0):FID128:admin> Proxy: ip2ipc started (run=1, rc=0), debug=0

    Proxy: ipc2ip started (rc=0)

    HAM: HAM:ham_open_htbt:Open heartbeat: 3e0006

    HAM: HAM:ham_open_htbt:Open heartbeat: 3f0107

    Oct 16 17:05:36  Heartbeat to 3e0006 Up!

    2017/10/16-17:05:35, [HAM-1004], 17076, SLOT 7 CHASSIS, INFO, Brocade_DCX, Processor rebooted - Reboot.

     

    Exchange HA Sync between CP

    bootenv: Could not get requested variable bootargs.

    2017/10/16-17:05:57, [FV-1001], 17077, SLOT 7 CHASSIS, INFO, dcx85108-fcs01, Flow Vision daemon initialized.

    SNMP Research EMANATE/Lite Agent Version 16.2.0.9

    Copyright 1989-2006 SNMP Research, Inc.

    All service instances in sync

    2017/10/16-17:06:31, [FSSM-1002], 17078, SLOT 7 CHASSIS, INFO, Brocade_DCX, HA State is in sync.

    HAM: HAM:ham_notify:Firmware download failed.

    Firmware commit operation has started to restore the secondary partition.

    2017/10/16-17:06:31, [SULB-1003], 17079, SLOT 7 CHASSIS, INFO, Brocade_DCX, Firmwarecommit has started.

    Validating the filesystem ...

     

    unknown(0):FID128:admin>

    unknown(0):FID128:admin>

    unknown(0):FID128:admin> slotshow

    rbash: slotshow: command not found

    unknown(0):FID128:admin> exit

    logout

     

     

    Fabric OS (dcx85108-fcs01)

     

     

    dcx85108-fcs01 console login: admin

    Password:

    *****************************************************************

    Logging into STANDBY CP, not all commands are fully supported !!

    *****************************************************************

    dcx85108-fcs01:FID128:admin> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      6  CP0        FOS      v8.1.1                                   ACTIVE

                             v8.1.1

      7  CP1        FOS      v8.1.1                                   STANDBY *

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

    dcx85108-fcs01:FID128:admin> firmwaredownloadstatus

    [1]: Fri Jun 16 00:52:22 2017

    Slot 6 (CP0, active): Firmware is being downloaded to standby CP. This step may take up to 30 minutes.

     

    [2]: Fri Jun 16 00:59:17 2017

    Slot 6 (CP0, active): Firmware has been downloaded successfully to Standby CP.

     

    [3]: Fri Jun 16 00:59:21 2017

    Slot 6 (CP0, active): Standby CP is going to reboot with new firmware.

     

    [4]: Fri Jun 16 01:01:22 2017

    Slot 6 (CP0, active): Standby CP booted successfully with new firmware.

     

    [5]: Fri Jun 16 01:02:29 2017

    Slot 7 (CP1, active): Forced failover succeeded. New Active CP is running new firmware

     

    [6]: Fri Jun 16 01:03:13 2017

    Slot 7 (CP1, active): Firmware is being downloaded to standby CP. This step may take up to 30 minutes.

     

    [7]: Fri Jun 16 01:08:03 2017

    Slot 7 (CP1, active): Firmware has been downloaded successfully on Standby CP.

     

    [8]: Fri Jun 16 01:08:07 2017

    Slot 7 (CP1, active): Standby CP reboots.

     

    [9]: Fri Jun 16 01:10:04 2017

    Slot 7 (CP1, active): Standby CP booted successfully with new firmware.

     

    [10]: Fri Jun 16 01:10:04 2017

    Slot 7 (CP1, active): Firmware commit operation has started on both active and standby CPs.

     

    [11]: Fri Jun 16 01:10:05 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [12]: Fri Jun 16 01:13:53 2017

    Slot 7 (CP1, active): The commit operation has completed successfully.

     

    [13]: Fri Jun 16 01:13:53 2017

    Slot 7 (CP1, active): Firmware commit operation has completed successfully on active CP.

     

    [14]: Fri Jun 16 01:13:53 2017

    Slot 7 (CP1, active): Firmwaredownload command has completed successfully. Use firmwareshow to verify the firmware versions.

     

    [15]: Fri Jun 16 01:15:24 2017

    Slot 6 (CP0, standby): The commit operation has completed successfully.

     

    [16]: Wed Jun 21 02:40:48 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [17]: Wed Jun 21 02:40:48 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    [18]: Wed Jun 21 02:50:17 2017

    Slot 7 (CP1, standby): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [19]: Thu Aug 10 08:52:36 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [20]: Thu Aug 10 08:52:37 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [21]: Thu Aug 10 09:57:13 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [22]: Thu Aug 10 09:57:14 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    [23]: Thu Aug 10 10:06:46 2017

    Slot 7 (CP1, standby): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [24]: Thu Aug 10 10:58:02 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [25]: Thu Aug 10 10:58:03 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [26]: Thu Aug 10 12:02:10 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [27]: Thu Aug 10 12:02:10 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    [28]: Thu Aug 10 12:11:35 2017

    Slot 7 (CP1, standby): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [29]: Thu Aug 10 13:03:41 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [30]: Thu Aug 10 13:03:41 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [31]: Thu Aug 10 13:14:19 2017

    Slot 7 (CP1, active): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [32]: Fri Aug 11 04:18:46 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [33]: Fri Aug 11 04:18:47 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    [34]: Fri Aug 11 04:28:10 2017

    Slot 7 (CP1, standby): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [35]: Fri Aug 11 05:19:43 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [36]: Fri Aug 11 05:19:45 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [37]: Fri Aug 11 05:30:16 2017

    Slot 7 (CP1, active): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [38]: Fri Aug 11 06:24:25 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [39]: Fri Aug 11 06:24:25 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    [40]: Fri Aug 11 06:33:54 2017

    Slot 7 (CP1, standby): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [41]: Fri Aug 11 07:24:20 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [42]: Fri Aug 11 07:24:21 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [43]: Fri Aug 11 07:34:55 2017

    Slot 7 (CP1, active): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [44]: Fri Aug 11 08:29:20 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [45]: Fri Aug 11 08:29:21 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    [46]: Fri Aug 11 08:38:45 2017

    Slot 7 (CP1, standby): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [47]: Wed Sep 20 07:57:40 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [48]: Wed Sep 20 07:57:40 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [49]: Wed Sep 20 08:08:10 2017

    Slot 7 (CP1, active): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [50]: Wed Sep 20 09:28:14 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [51]: Wed Sep 20 09:37:23 2017

    Slot 7 (CP1, active): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [52]: Wed Sep 20 16:22:59 2017

    Slot 6 (CP0, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [53]: Wed Sep 20 16:22:59 2017

    Slot 6 (CP0, active): Firmware commit failed - No need to commit firmware. (0x41)

     

    [54]: Tue Sep 26 02:34:45 2017

    Slot 6 (CP0, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [55]: Tue Sep 26 02:34:45 2017

    Slot 6 (CP0, active): Firmware commit failed - No need to commit firmware. (0x41)

     

    [56]: Tue Sep 26 02:36:40 2017

    Slot 6 (CP0, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [57]: Tue Sep 26 02:36:40 2017

    Slot 6 (CP0, active): Firmware commit failed - No need to commit firmware. (0x41)

     

    [58]: Mon Oct 16 16:19:14 2017

    Slot 6 (CP0, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [59]: Mon Oct 16 16:19:14 2017

    Slot 6 (CP0, active): Firmware commit failed - No need to commit firmware. (0x41)

     

    [60]: Mon Oct 16 16:48:48 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [61]: Mon Oct 16 16:48:49 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [62]: Mon Oct 16 16:58:12 2017

    Slot 7 (CP1, active): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    [63]: Mon Oct 16 17:01:55 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [64]: Mon Oct 16 17:01:55 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [65]: Mon Oct 16 17:06:31 2017

    Slot 7 (CP1, standby): Firmware commit operation has started to restore the secondary partition.

     

    [66]: Mon Oct 16 17:06:31 2017

    Slot 7 (CP1, standby): The firmware commit operation has started. This may take up to 10 minutes.

     

    dcx85108-fcs01:FID128:admin> Repairing the secondary partition now.

    Please wait ...

    rsync: read errors mapping "/boot/bootrom.bin": Input/output error (5)

     

     

     

    -

    ------------

     

    After reseated and powered on CP blade. Now it’s in the state below and firmware commit is keep failing.

     

    WARNING: Firmwaredownload is in progress.

     

     

    dcx85108-fcs01:FID128:root> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      6  CP0        FOS      v8.1.1                                   ACTIVE *

                             v8.1.1

      7  CP1        FOS      v8.1.1                                   STANDBY

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

    dcx85108-fcs01:FID128:root>

     

     

     

    [55]: Tue Sep 26 02:34:45 2017

    Slot 6 (CP0, active): Firmware commit failed - No need to commit firmware. (0x41)

     

    [56]: Tue Sep 26 02:36:40 2017

    Slot 6 (CP0, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [57]: Tue Sep 26 02:36:40 2017

    Slot 6 (CP0, active): Firmware commit failed - No need to commit firmware. (0x41)

     

    [58]: Mon Oct 16 16:19:14 2017

    Slot 6 (CP0, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [59]: Mon Oct 16 16:19:14 2017

    Slot 6 (CP0, active): Firmware commit failed - No need to commit firmware. (0x41)

     

    dcx85108-fcs01:FID128:root>

     

     

     

     

    -

     

    -----------------

     

    Have you seen this? And any open issues?

    I have requested for remove and reinsert to clear out any errors.

     

    Provide any procedure or recovery steps.

     

     

    we tried the suggested scenario but unable to disable the CP blade. It gives the below error message:

    ========================================

    dcx85108-fcs01:FID128:admin> slotpoweroff 7
    Not supported on this slot

    dcx85108-fcs01:FID128:admin> bladedisable 7
    Blade 7 is being disabled...Failed
    Check if you typed a SW BLADE (Not CP) number and
    it is within the range, executing 'slotshow' command on the switch.
    Also, check if switch disabled or slot off or no blade,
    or if the blade is running Diagnostics.

    dcx85108-fcs01:FID128:admin> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      6  CP0        FOS      v8.1.1                                   ACTIVE *

                             v8.1.1

      7  CP1        FOS      v8.1.1                                   STANDBY

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

    dcx85108-fcs01:FID128:admin>

     

    dcx85108-fcs01:FID128:admin> slotshow

     

    Slot   Blade Type     ID    Status

    -----------------------------------

      1     SW BLADE     96     ENABLED

      2     UNKNOWN             VACANT

      3     SW BLADE    153     ENABLED

      4     UNKNOWN             VACANT

      5     CORE BLADE   98     ENABLED

      6     CP BLADE     50     ENABLED

      7     CP BLADE     50     ENABLED

      8     CORE BLADE   98     ENABLED

      9     UNKNOWN             VACANT

    10     SW BLADE     97     ENABLED

    11     UNKNOWN             VACANT

    12     UNKNOWN             VACANT

     

    dcx85108-fcs01:FID128:admin>

    =======================================

     

    I also tried doing firmwaresync but it also failed with below:

     

    dcx85108-fcs01:FID128:admin> firmwaresync

     

    This command will copy the firmware on the active CP blade to the

    standby CP blade but will require that existing telnet, secure telnet or

    SSH sessions to the standby CP blade to be restarted.

     

    This command may take up to 20 minutes.

     

     

    Do you want to continue (Y/N) [Y]: y

    Another firmware operation is running now.

    dcx85108-fcs01:FID128:admin>

     

    I think to perform any update we will have to stop the ongoing firmware download process.

     

    Can you look at this?

     

    Here are the detail from switch:

    ======================================

    dcx85108-fcs01:FID128:root> slotshow

     

    Slot   Blade Type     ID    Status

    -----------------------------------

      1     SW BLADE     96     ENABLED

      2     UNKNOWN             VACANT

      3     SW BLADE    153     ENABLED

      4     UNKNOWN             VACANT

      5     CORE BLADE   98     ENABLED

      6     CP BLADE     50     ENABLED

      7     CP BLADE     50     ENABLED

      8     CORE BLADE   98     ENABLED

      9     UNKNOWN             VACANT

    10     SW BLADE     97     ENABLED

    11     UNKNOWN             VACANT

    12     UNKNOWN             VACANT

     

    dcx85108-fcs01:FID128:root> firmwareshow

    Slot Name       Appl     Primary/Secondary Versions               Status

    --------------------------------------------------------------------------

      6  CP0        FOS      v8.1.1                                   STANDBY

                             v8.1.1

      7  CP1        FOS      v8.1.1                                   ACTIVE *

                             Unknown

    *  Local CP

     

    WARNING: Firmwaredownload is in progress.

     

    dcx85108-fcs01:FID128:root> firmwarecheck

    Validating integrity of firmware on root filesystem

    Please wait...

    ..5....T   /fabos/modules/blaster-module.ko

    ..5....T   /fabos/lib/libtracedb.so.1.0

    Firmware integrity check failed.

    dcx85108-fcs01:FID128:root>

     

    dcx85108-fcs01:FID128:admin> firmwaredownloadstatus

    [47]: Wed Sep 20 07:57:40 2017

    Slot 7 (CP1, active): Firmware commit operation has started to restore the secondary partition.

     

    [48]: Wed Sep 20 07:57:40 2017

    Slot 7 (CP1, active): The firmware commit operation has started. This may take up to 10 minutes.

     

    [49]: Wed Sep 20 08:08:10 2017

    Slot 7 (CP1, active): Firmware commit failed - Firmwarecommit failed. (0x1a)

     

    dcx85108-fcs01:FID128:admin>

    ========================================

     

    -Sunil

     


    #BrocadeFibreChannelNetworkingCommunity


  • 2.  Re: Issues with DCX8510 CP blade's secondary partition reported after upgrade from 8.1.0 to 8.1.1

    Posted 10-17-2017 07:59 AM

    Hi Sunil,

     

    look like firmware partitions on CP0 are corrupt - notice that from the below both read (of 8.1.1.) and writes are failing

     

    rsync: read errors mapping "/tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm": Input/output error (5)

    rsync: read errors mapping "/boot/bootrom.bin": Input/output error (5)

    rsync: read errors mapping "/fabos/modules/pluto-zentron-module.ko": Input/output error (5)

    rsync: read errors mapping "/tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm": Input/output error (5)

     

    ERROR: boot/bootrom.bin failed verification -- update discarded.

    ERROR: fabos/modules/pluto-zentron-module.ko failed verification -- update discarded.

    ERROR: tftpboot/SWBD88/kernel-debug-2.6.14.2-3282828861.ppc.rpm failed verification -- update

     

    during firmwarecommit.  As long as you have  a pending commit / firmware download (as determined by bootenv variables)

    you will not be able to proceed.  The slotpoweroff works on core and port blades, but not on CP, BTW.  

    Notice that firmwarecheck is also reporting errors - original install of 8.1.1 is probably not complete.

     

    dcx85108-fcs01:FID128:root> firmwarecheck

    Validating integrity of firmware on root filesystem

    Please wait...

    ..5....T   /fabos/modules/blaster-module.ko

    ..5....T   /fabos/lib/libtracedb.so.1.0

    Firmware integrity check failed.

    dcx85108-fcs01:FID128:root>

     

    I would open a support case for this to determine if the compact flash is really bad (swap out CP) or can be recovered. How full are the partitions, e.g. what does 'df' report as root?

     

     


    #BrocadeFibreChannelNetworkingCommunity


  • 3.  Re: Issues with DCX8510 CP blade's secondary partition reported after upgrade from 8.1.0 to 8.1.1

    Posted 10-18-2017 09:22 AM

    Well, here's a last ditch try. You can use the firmwarecleaninstall but here's the rub. It is not as clean as the name implies.

     

    First: Your dir WILL reboot. There is no way around it.

    Next: Even if you do a configupload and then configdownload with the 'all' switch, it's unlikely that your logical switches will come back up right. You may have to disable VF, then enable it, then reload the config again.

    Finally; Although this will clean up the file system, there is no guarantee that you aren't hitting a double bit error, or some other file system defect.

     

    Proceed with caution if you choose to use firmwarecleaninstall.


    #BrocadeFibreChannelNetworkingCommunity


  • 4.  Re: Issues with DCX8510 CP blade's secondary partition reported after upgrade from 8.1.0 to 8.1.1

    Posted 04-04-2019 05:39 AM

    Very old post, but this solution may help others.

     The issue is usually caused by a full compact flash on the CP.

    run ‘supportsave -R’ to cleanup the compact flash,

    once run,  try ‘firmwarecommit’ again.

     

    I highly recommend running supportsave -R. Before beginning firmware updates to avoid running into this problem


    #BrocadeFibreChannelNetworkingCommunity