DX Unified Infrastructure Management

 View Only
  • 1.  CABI 4.20 failing to start

    Posted Dec 04, 2020 07:06 AM
    Can anyone make sense as to why this is failing to start?

    I've performed a full reset (running ddl script, removing cabi/wasp and c folder..

    When redeploying cabi I get this in log:

    Dec 04 11:19:11:827 [main, cabi] extracting cabi installer ...
    Dec 04 11:20:47:904 [main, cabi] successfully extracted cabi installer at nimPath=/opt/nimsoft [time taken=1 minute(s) 36 second(s) (96076 ms)]
    Dec 04 11:20:47:907 [main, cabi] changing probe state from 'INSTALLING_CABI' to 'STOPPING_WASP'
    Dec 04 11:20:47:907 [main, cabi] deactivating wasp...
    Dec 04 11:20:47:912 [main, cabi] waiting for wasp to stop ...
    Dec 04 11:20:47:914 [main, cabi] Wasp status active=true
    at com.nimsoft.nimbus.NimRequest.sendImpersonate(NimRequest.java:264)
    at com.nimsoft.nimbus.NimRequest.sendImpersonate(NimRequest.java:237)
    at com.nimsoft.nimbus.NimRequest.send(NimRequest.java:208)
    at com.ca.cabi4uim.utilities.ProbeUtils.request1(ProbeUtils.java:383)
    at com.ca.cabi4uim.utilities.ProbeUtils.request(ProbeUtils.java:391)
    at com.ca.cabi4uim.utilities.ProbeUtils.waspConfigGet(ProbeUtils.java:234)
    at com.ca.cabi4uim.utilities.ProbeUtils.waspConfigGetAndHandleKeyNotFoundCondition(ProbeUtils.java:661)
    at com.ca.cabi4uim.utilities.ProbeUtils.isWaspSaaS(ProbeUtils.java:629)
    at com.ca.cabi4uim.Probe.setupcabi(Probe.java:371)
    at com.ca.cabi4uim.Probe.initOnTimer(Probe.java:700)
    at com.nimsoft.nimbus.NimProbe.doForever(NimProbe.java:418)
    at com.ca.cabi4uim.Probe.run(Probe.java:892)
    at com.ca.cabi4uim.Probe.main(Probe.java:123)
    Dec 04 11:21:03:032 [main, cabi] nimEx.getCode()=4, nimEx.getMsgForCode()=not found, nimEx.getMessage()=Received status (4) on response (for sendRcv) for cmd = 'probe_config_get' name = 'wasp'
    Dec 04 11:21:03:032 [main, cabi] ignoring the message as the key might not net set; logging it for tracking later
    Dec 04 11:21:03:034 [main, cabi] activating wasp...
    Dec 04 11:21:03:036 [main, cabi] ... activated wasp [time taken=0 minute(s) 0 second(s) (2 ms)]
    Dec 04 11:21:03:037 [main, cabi] java.lang.NullPointerException
    at com.ca.cabi4uim.utilities.Utils.doCreateUpgradePropFileRequiredByCABIInstaller(Utils.java:614)
    at com.ca.cabi4uim.Probe.writeUimProperties(Probe.java:293)
    at com.ca.cabi4uim.Probe.doConfigAndRunCabiInstaller(Probe.java:439)
    at com.ca.cabi4uim.Probe.setupCabiInBundledMode(Probe.java:388)
    at com.ca.cabi4uim.Probe.setupcabi(Probe.java:354)
    Dec 04 11:21:03:037 [main, cabi] setting the lastInstallStatus=4
    Dec 04 11:21:03:037 [main, cabi] successfully set the lastInstallStatus=4
    Dec 04 11:21:03:038 [main, cabi] srcFilePath=/opt/nimsoft/probes/service/cabi/cabi.log, srcFilePath.exists=true, tgtFilePath=/opt/nimsoft/probes/service/cabi/cabi_install_archive.log, tgtFilePath.exists=false
    Dec 4 11:21:04:278 [139924662785856] Controller: Probe 'cabi' (command = <startup java>) returns no-restart code (42)

    ------------------------------
    CA - UIM administrator
    ------------------------------


  • 2.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 05:48 AM
    Built an entirely new Server on a totally different OS (Original log was from an install on a Linux RHEL 7.8 Server).

    Not I'm getting the exact same error on a Windows 2016 build??

    ------------------------------
    CA - UIM administrator
    ------------------------------



  • 3.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 06:18 AM
    I think you need the delete the cabi tables in the UIM DB. I've just done this and CABI is now installed and running.

    You'll find the sql scripts in Nimsoft\probes\service\cabi\config\scripts\drop_tables\ on your CABI server.

    Some manual deleting of tables was needed when I did it.

    The section that deletes the QUARTZ tables contains an invalid IF EXISTS.


  • 4.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 06:34 AM
    Thanks Gary,
    I've ran a couple of scripts.  One from the path you mentioned, and another provided by support. I'll attach both.

    The one provided by support runs fine, but no rows are affected when it's ran.

    The one found on the CABI server errors reporting all tables starting with ca_uim.ji* doesn't exist.  It then continues reporting 0 rows affected.

    ------------------------------
    CA - UIM administrator
    ------------------------------

    Attachment(s)

    txt
    drop_mySQL.txt   13 KB 1 version
    txt
    js-pro-drop.txt   17 KB 1 version


  • 5.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 07:35 AM
    Seemed to get a little more out of the log on another attempt..

    Still ultimately failed with the same error:

    Dec 07 12:27:30:466 [main, cabi] extracting cabi installer ...
    Dec 07 12:29:24:335 [Thread-0, cabi] Buildfile: C:\Program Files (x86)\Nimsoft\c\ca_install\build.xml
    Dec 07 12:29:24:585 [Thread-0, cabi]
    Dec 07 12:29:24:585 [Thread-0, cabi] preinstallcheck:
    Dec 07 12:29:24:601 [Thread-0, cabi] [echo] UIM: Skipped Preinstall validation since CABI db is embedded in UIM DB
    Dec 07 12:29:24:632 [Thread-0, cabi]
    Dec 07 12:29:24:632 [Thread-0, cabi] copyandupdate_masterfile.init:
    Dec 07 12:29:24:632 [Thread-0, cabi] mysql:
    Dec 07 12:29:24:632 [Thread-0, cabi] [copy] Copying 1 file to C:\Program Files (x86)\Nimsoft\c\buildomatic
    Dec 07 12:29:24:726 [Thread-0, cabi] [propertyfile] Updating property file: C:\Program Files (x86)\Nimsoft\c\buildomatic\default_master.properties
    Dec 07 12:29:24:757 [Thread-0, cabi] [propertyfile] Updating property file: C:\Program Files (x86)\Nimsoft\c\buildomatic\default_master.properties
    Dec 07 12:29:24:773 [Thread-0, cabi] [propertyfile] Updating property file: C:\Program Files (x86)\Nimsoft\c\buildomatic\default_master.properties
    Dec 07 12:29:24:789 [Thread-0, cabi]
    Dec 07 12:29:24:789 [Thread-0, cabi] mysqlport:
    Dec 07 12:29:24:789 [Thread-0, cabi] setdefaults:
    Dec 07 12:29:24:804 [Thread-0, cabi] [propertyfile] Updating property file: C:\Program Files (x86)\Nimsoft\c\buildomatic\default_master.properties
    Dec 07 12:29:24:820 [Thread-0, cabi]
    Dec 07 12:29:24:820 [Thread-0, cabi] checkappserver:
    Dec 07 12:29:24:820 [Thread-0, cabi] sqlserver:
    Dec 07 12:29:24:820 [Thread-0, cabi] oracle:
    Dec 07 12:29:24:820 [Thread-0, cabi] postgresql:
    Dec 07 12:29:24:820 [Thread-0, cabi] other:
    Dec 07 12:29:24:820 [Thread-0, cabi] copyandupdate_masterfile:
    Dec 07 12:29:24:820 [Thread-0, cabi] [echo] Creation of default_master.properties file got completed successfully
    Dec 07 12:29:24:835 [Thread-0, cabi]
    Dec 07 12:29:24:835 [Thread-0, cabi] jsinstall-pro:
    Dec 07 12:29:24:851 [Thread-0, cabi]
    Dec 07 12:29:24:851 [Thread-0, cabi] install-jasper:
    Dec 07 12:29:24:851 [Thread-0, cabi] install-full-db:
    Dec 07 12:29:24:867 [Thread-0, cabi]
    Dec 07 12:29:24:867 [Thread-0, cabi] install-min-db:
    Dec 07 12:29:24:867 [Thread-0, cabi] [echo] Started CA Business Intelligence 7.5.0 installation without samples
    Dec 07 12:29:24:882 [Thread-0, cabi]
    Dec 07 12:29:24:882 [Thread-0, cabi] install-db-fresh-min:
    Dec 07 12:29:24:882 [Thread-0, cabi] [echo] Executing js-install without samples
    Dec 07 12:29:25:226 [Thread-0, cabi] [exec] Writing to log file: logs/js-install-pro_2020-12-07_12-29_10433.log
    Dec 07 12:29:25:226 [Thread-0, cabi] [exec] ----------------------------------------------------------------------
    Dec 07 12:29:25:226 [Thread-0, cabi] [exec] Running JasperReports Server install script at 2020-12-07_12-29
    Dec 07 12:29:25:226 [Thread-0, cabi] [exec] [minimal]
    Dec 07 12:29:25:226 [Thread-0, cabi] [exec] Running install-minimal-pro Ant task
    Dec 07 12:29:25:336 [Thread-0, cabi] [exec] Unable to locate tools.jar. Expected to find it in C:\Program Files (x86)\Nimsoft\jre\jre8u262b10\lib\tools.jar
    Dec 07 12:29:25:336 [Thread-0, cabi] [exec] Buildfile: C:\Program Files (x86)\Nimsoft\c\buildomatic\build.xml
    Dec 07 12:29:25:773 [Thread-0, cabi] [exec] [echo]
    Dec 07 12:29:25:773 [Thread-0, cabi] [exec] [create-ks] WARNING: A new encryption key and a new keystore are about to be created.
    at com.nimsoft.nimbus.NimRequest.sendImpersonate(NimRequest.java:264)
    at com.nimsoft.nimbus.NimRequest.sendImpersonate(NimRequest.java:237)
    at com.nimsoft.nimbus.NimRequest.send(NimRequest.java:208)
    at com.ca.cabi4uim.utilities.ProbeUtils.request1(ProbeUtils.java:386)
    at com.ca.cabi4uim.utilities.ProbeUtils.request(ProbeUtils.java:394)
    at com.ca.cabi4uim.utilities.ProbeUtils.waspConfigGet(ProbeUtils.java:237)
    at com.ca.cabi4uim.utilities.ProbeUtils.waspConfigGetAndHandleKeyNotFoundCondition(ProbeUtils.java:664)
    at com.ca.cabi4uim.utilities.ProbeUtils.isWaspSaaS(ProbeUtils.java:632)
    at com.ca.cabi4uim.Probe.setupcabi(Probe.java:371)
    at com.ca.cabi4uim.Probe.initOnTimer(Probe.java:702)
    at com.nimsoft.nimbus.NimProbe.doForever(NimProbe.java:418)
    at com.ca.cabi4uim.Probe.run(Probe.java:894)
    at com.ca.cabi4uim.Probe.main(Probe.java:123)
    Dec 07 12:29:28:023 [main, cabi] nimEx.getCode()=4, nimEx.getMsgForCode()=not found, nimEx.getMessage()=Received status (4) on response (for sendRcv) for cmd = 'probe_config_get' name = 'wasp'
    Dec 07 12:29:28:023 [main, cabi] ignoring the message as the key might not net set; logging it for tracking later
    Dec 07 12:29:28:039 [main, cabi] activating wasp...
    Dec 07 12:29:28:039 [main, cabi] ... activated wasp [time taken=0 minute(s) 0 second(s) (0 ms)]
    Dec 07 12:29:28:039 [main, cabi] (42) , CABI installer failed with exit code=1
    at com.ca.cabi4uim.installer.CabiInstaller.runCabiInstaller(CabiInstaller.java:247)
    at com.ca.cabi4uim.Probe.doConfigAndRunCabiInstaller(Probe.java:443)
    at com.ca.cabi4uim.Probe.setupCabiInBundledMode(Probe.java:388)
    at com.ca.cabi4uim.Probe.setupcabi(Probe.java:354)
    Dec 07 12:29:28:039 [main, cabi] setting the lastInstallStatus=4
    Dec 07 12:29:28:039 [main, cabi] successfully set the lastInstallStatus=4
    Dec 07 12:29:28:039 [main, cabi] srcFilePath=C:\Program Files (x86)\Nimsoft\probes\service\cabi\cabi.log, srcFilePath.exists=true, tgtFilePath=C:\Program Files (x86)\Nimsoft\probes\service\cabi\cabi_install_archive.log, tgtFilePath.exists=false
    Dec 7 12:29:29:204 [0076] Controller: Probe 'cabi' (command = <startup java>) returns no-restart code (42)

    ------------------------------
    CA - UIM administrator
    ------------------------------



  • 6.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 08:44 AM
    Check to see if the keystore files got created, location is in this KB
    UIM 20.3 cabi deployment fails. - Keystore may have been tempered with
    Article Id: 201551
    https://knowledge.broadcom.com/external/article?articleId=201551

    Beyond that check the cabi robot wasp log.

    ------------------------------
    Support Engineer
    Broadcom
    ------------------------------



  • 7.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 08:54 AM
    Thanks David,
    That error wasn't present in the log, https hadn't been enabled at that stage.

    Strangely, I've just redeployed cabi to the Linux Server after another full reset, and it's installed.  I genuinely don't know what I've done differently to the last x amount of attempts.  I've repeated this process so many times over the last few days, this time it just seems have gone through.

    ------------------------------
    CA - UIM administrator
    ------------------------------



  • 8.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 11:06 AM
    The "this time it just seems have gone through" is a consistent experience with mine.

    Expect further that with the CABI released with 20.10 and 20.30 that it will also go bad over time. Sometimes a restart of wasp or cabi or the robot fixes it, sometimes it involves deleting the work folder and bouncing, sometimes it's a remove, delete directories, reinstall effort. Since the release of 20.10 CABI starting without manual intervention is at about 50% success. 

    Not sure the impact but there was a significant change in reliability moving from a self signed certificate to a trusted authority issued one but I don't know why that might be - it shouldn't make a difference but it seems to have. Not fixed mind you but reliability on startup has moved from 50% to 75%ish.


  • 9.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 11:23 AM
    Glad it's not just me Garin!

    ------------------------------
    CA - UIM administrator
    ------------------------------



  • 10.  RE: CABI 4.20 failing to start

    Posted Dec 07, 2020 11:53 AM
    It is solidly a house of cards structure.

    I really shouldn't feel lucky when things work after a restart.