SDM 17.3.04, coracleanup, JAVA-error

[Baerbel Veit]

Hello Team,

I've installed SDM 17.3.0.4.

I tried coracleanup, but directly I get this JAVA error.
I defined th JAVA_HOME environment variable this is not missing.
I investigated some time looking for a solution, but unsuccessfully.
Does sombody knows this error and a solution?

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffe6da475e3, pid=7508, tid=3820
#
# JRE version: OpenJDK Runtime Environment (11.0.3+7) (build 11.0.3+7)
# Java VM: OpenJDK 64-Bit Server VM (11.0.3+7, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# C [ucrtbase.dll+0x275e3]
#
# Core dump will be written. Default location: C:\Program Files (x86)\CA\Service Desk Manager\bin\hs_err_pid7508.mdmp
#
# If you would like to submit a bug report, please visit:
# https://github.com/AdoptOpenJDK/openjdk-build/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- S U M M A R Y ------------

Command Line: -DLOG_CONFIG_DIR=C:/PROGRA~2/CA/SERVIC~1/site/cora_cfg -ea com.ca.Cora.util.ConsoleManager /SERVER WIN-HEDAC706DI3 /TYPE 2 /USERNAME sa /PASSWORD xxxxx /SCOPE USD /METHOD reinit /DATABASE mdb

Host: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz, 4 cores, 15G, Windows Server 2019 , 64 bit Build 17763 (10.0.17763.475)
Time: Thu Feb 18 12:04:21 2021 Mitteleuropäische Zeit elapsed time: 0 seconds (0d 0h 0m 0s)

--------------- T H R E A D ---------------

Current thread (0x0000018d8036e000): JavaThread "ApiAgent" [_thread_in_native, id=3820, stack(0x00000012a6400000,0x00000012a6500000)]

Stack: [0x00000012a6400000,0x00000012a6500000], sp=0x00000012a64fc698, free space=1009k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [ucrtbase.dll+0x275e3]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.ca.careg.CoraJava.JNIRegister(JIILjava/lang/String;Ljava/lang/String;[Ljava/lang/String;II)J+0
j com.ca.careg.CoraJava.register(JIILjava/lang/String;Ljava/lang/String;II)Ljava/lang/String;+36
j com.ca.Cora.util.ApiAgent.coraRegister(ILjava/lang/String;ILjava/lang/String;)Ljava/lang/String;+18
j com.ca.Cora.util.ApiAgent.reRegisterAssets()V+57
j com.ca.Cora.util.ApiAgent.doRegistrations()Z+65
j com.ca.Cora.util.ApiAgent.run()V+9
j java.lang.Thread.run()V+11 java.base@11.0.3
v ~StubRoutines::call_stub

siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), reading address 0xffffffff802b631a


Register to memory mapping:

RIP=0x00007ffe6da475e3 ucrtbase.dll
RAX=0x0000000000000043 is an unknown value
RBX={method} {0x0000018dff649da8} 'JNIRegister' '(JIILjava/lang/String;Ljava/lang/String;[Ljava/lang/String;II)J' in 'com/ca/careg/CoraJava'
RCX=0xffffffff802b631a is an unknown value
RDX=0x00007ffe46ab3a80 coraapi.dll
RSP=0x00000012a64fc698 is pointing into the stack for thread: 0x0000018d8036e000
RBP=0x00000012a64fee50 is pointing into the stack for thread: 0x0000018d8036e000
RSI=0x000000000001eb78 is an unknown value
RDI=0x000000070f400270 is an oop: [Ljava.lang.String;
{0x000000070f400270} - klass: 'java/lang/String'[]
- length: 1
R8 =0x00007ffe46ab4608 coraapi.dll
R9 =0x000000000000103d is an unknown value
R10=0x0000018d82ec90c0 points into unknown readable memory: 71 b6 15 f6 a8 77 46 4b
R11=0x00000012a64fed08 is pointing into the stack for thread: 0x0000018d8036e000
R12=0x0 is NULL
R13={method} {0x0000018dff649da8} 'JNIRegister' '(JIILjava/lang/String;Ljava/lang/String;[Ljava/lang/String;II)J' in 'com/ca/careg/CoraJava'
R14=0x00000012a64feeb8 is pointing into the stack for thread: 0x0000018d8036e000
R15=0x0000018d8036e000 is a thread


Registers:
RAX=0x0000000000000043, RBX=0x0000018dff649da0, RCX=0xffffffff802b631a, RDX=0x00007ffe46ab3a80
RSP=0x00000012a64fc698, RBP=0x00000012a64fee50, RSI=0x000000000001eb78, RDI=0x000000070f400270
R8 =0x00007ffe46ab4608, R9 =0x000000000000103d, R10=0x0000018d82ec90c0, R11=0x00000012a64


Regards
   Baerbel

------------------------------
Senior Consultant
MAXXYS AG
------------------------------

[Janos Mertz]

Hi Team,

 

We just got the same error from our customer that I was able to reproduce in my development machine. In the customer environment we have SDM 17.3.0.3 and I went back to the SDM 17.3.0.1 snapshot in my VM machine. We got the same error in both systems.

I used the following command:

CORACLEANUP /SERVER SDM173VM /TYPE 2 /USERNAME sa /PASSWORD ***** /SCOPE USD /METHOD reinit /DATABASE mdb

Result:

Assets Read = 0, Registration Successful = 0, Registration Failed = 0#

# A fatal error has been detected by the Java Runtime Environment:

#

#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffc7df72503, pid=2820, tid=6608

#

# JRE version: OpenJDK Runtime Environment (11.0.3+7) (build 11.0.3+7)

# Java VM: OpenJDK 64-Bit Server VM (11.0.3+7, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)

# Problematic frame:

# C  [ucrtbase.DLL+0x22503]

#

# Core dump will be written. Default location: C:\Program Files (x86)\CA\Service Desk Manager\bin\hs_err_pid2820.mdmp

#

# An error report file with more information is saved as:

# C:\Program Files (x86)\CA\Service Desk Manager\bin\hs_err_pid2820.log

#

# If you would like to submit a bug report, please visit:

#   https://github.com/AdoptOpenJDK/openjdk-build/issues

# The crash happened outside the Java Virtual Machine in native code.

# See problematic frame for where to report the bug.

#

 

Can any of you give me advice on how to solve this problem?

 

Thanks, and best regards

Janos Mertz

[Janos Mertz]

Hi Team,

 

We just got the same error from our customer that I was able to reproduce in my development machine. In the customer environment we have SDM 17.3.0.3 and I went back to the SDM 17.3.0.1 snapshot in my VM machine. We got the same error in both systems.

I used the following command:

CORACLEANUP /SERVER SDM173VM /TYPE 2 /USERNAME sa /PASSWORD ***** /SCOPE USD /METHOD reinit /DATABASE mdb

Result:

Assets Read = 0, Registration Successful = 0, Registration Failed = 0#

# A fatal error has been detected by the Java Runtime Environment:

#

#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffc7df72503, pid=2820, tid=6608

#

# JRE version: OpenJDK Runtime Environment (11.0.3+7) (build 11.0.3+7)

# Java VM: OpenJDK 64-Bit Server VM (11.0.3+7, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)

# Problematic frame:

# C  [ucrtbase.DLL+0x22503]

#

# Core dump will be written. Default location: C:\Program Files (x86)\CA\Service Desk Manager\bin\hs_err_pid2820.mdmp

#

# An error report file with more information is saved as:

# C:\Program Files (x86)\CA\Service Desk Manager\bin\hs_err_pid2820.log

#

# If you would like to submit a bug report, please visit:

#   https://github.com/AdoptOpenJDK/openjdk-build/issues

# The crash happened outside the Java Virtual Machine in native code.

# See problematic frame for where to report the bug.

#

 

Can any of you give me advice on how to solve this problem?

 

Thanks, and best regards

Janos Mertz

------------------------------
Senior Services Consultant
Enterprise Studio, HCL Technologies Austria GmbH

------------------------------

[Baerbel Veit]

Hello Janos,
thanks a lot​ to affirm the problem. At customer site it is impossible to go back to a prior version, Broadcom has to deliver a solution.

I did already create a ticket, I think you as well. The more tickets he better.

Regards
   Bärbel

------------------------------
Senior Consultant
MAXXYS AG
------------------------------

[Janos Mertz]

Hi Bärbel,

 

Thanks you!

I also created a support ticket a few days ago.

Support has confirmed to me that they can reproduce it.

For me, although it does not run due to an exception, the basic tables are deleted.

Tables like ca_asset ca, logical_asset, ca_logical_asset_property are all empty after coracleanup.

 

Thanks, and best regards

Janos

------------------------------
Senior Services Consultant
Enterprise Studio, HCL Technologies Austria GmbH
------------------------------

[Baerbel Veit]

Hello Janos,
fortunately at customer site the cora tables were not deleted, but the row-counts are different in all of the tables.

In my VMWare session the coratables are empty.

The customer has 735.600 CIs, perhaps this is the reason, why the rows were not deleted.

Anyway, due to the different row counts the cora is not clean.

Regards Baerbel

------------------------------
Senior Consultant
MAXXYS AG
------------------------------

[Janos Mertz]

Hi Team,

 

It's been a long story, but we got it going with the help of support.

I tested it in a newly installed R17.3 and an environment migrated from R14.1 to 17.3 and it worked in both with the following steps:

Note: the paths you use may differ from mine.

• Expand path variable: "set PATH= C:\Program Files (x86)\CA\Service Desk Manager\REPLACED\HYD-368_CUM_C.OLD \bin;%PATH%" 
• Edit the coracleanup.cmd in the directory C:\Program Files (x86)\CA\Service Desk Manager\REPLACED\HYD-368_CUM_C.OLD:

The content of the script:

@echo off

 

setlocal

set NX_TEMP_CMD=%TEMP%\setenv.cmd

nx_env -d NX_ROOT > %NX_TEMP_CMD%

nx_env -d NX_JRE_INSTALL_DIR >> %NX_TEMP_CMD%

call %NX_TEMP_CMD%

del %NX_TEMP_CMD%

 

REM /* Build argument list */

set args=

:getargs

set args=%args% %1

shift

if not .%1 == . goto :getargs

 

REM GR: Set additional paths

set NX_OLD_ROOT=C:/Program Files (x86)/CA/Service Desk Manager/REPLACED/HYD-368_CUM_C.OLD

set JAVA32BIT=C:/Program Files (x86)/CA/SC/JRE/1.8.0_112

set JAVA64BIT=%NX_JRE_INSTALL_DIR%

 

REM /* Invoke the tool */

 

"%JAVA32BIT%/bin/java" -DLOG_CONFIG_DIR="%NX_ROOT%/site/cora_cfg" -ea -classpath "%NX_ROOT%/java/lib/corajava.jar;%NX_ROOT%/java/lib/cora_util.jar;%NX_ROOT%/java/lib/log4j-1.2.17.jar;%NX_ROOT%/java/lib/ojdbc14.jar;%NX_OLD_ROOT%/java/lib/sqljdbc.jar" com.ca.Cora.util.ConsoleManager %args%

 

• Open a DOS box and navigate to the directory C:\Program Files (x86)\CA\Service Desk Manager\REPLACED\HYD-368_CUM_C.OLD \bin
• Run the coracleanup utility: coracleanup.cmd  /SERVER <Your-Server> /TYPE 2 /USERNAME <DB-User> /PASSWORD <DB-User-PW> /SCOPE USD /METHOD reint /DATABASE mdb

Hopefully it helps.

 

Best regards

Janos Mertz

------------------------------
Senior Services Consultant
Enterprise Studio, HCL Technologies Austria GmbH
------------------------------

[Paul Coccimiglio]

CORA will not work with 64bit JAVA that comes with SDM 17.3.

As you have discovered, you need to have a 32-bit JAVA installed and then modify the CORA.CMD file so that CORA uses the 32-bit JAVA version

------------------------------
Paul Coccimiglio
Principal Support Engineer
Broadcom Inc.
------------------------------

[Michael Mueller]

Hi Paul.
Are there any plans to migrate the cora stuff to 64 bit , same as for all other stuff in SDM recently?
At least this is my expectation.
Thanks
....Michael

------------------------------
Principal Services Consultant
HCL Enterprise Studio
Germany
------------------------------

[Giovanni Tamburlin]

Hi Team,
Hi Janos,

I'm trying to run coracleanup.cmd, beacuse I'm getting error " AHD05519:Error in MDB registration: (MDB and ca_owned_resource are not consistent; asset_source_uuid(DE89DDFBC3BB684B87A378F0D53A6C83) not found in ca_owned_resource table  ) " when I try to add a CI from CA Service Desk web interface.

The same error pops up if I try to import CI with GRloader.

Executing the command:

"C:/Program Files/CA/SC/JRE/ 1.8.0_74/bin/java" -DLOG_CONFIG_DIR="C:/PROGRA~2/CA/SERVIC~1/site/cora_cfg" -ea -classpath "C:/PROGRA~2/CA/SERVIC~1/java/lib/corajava.jar;C:/PROGRA~2/CA/SERVIC~1/java/lib/cora_util.jar;C:/PROGRA~2/CA/SERVIC~1/java/lib/log4j-1.2.17.jar;C:/PROGRA~2/CA/SERVIC~1/java/lib/ojdbc14.jar;C:/Program Files (x86)/CA/Service Desk Manager/java/lib/sqljdbc.jar" com.ca.Cora.util.ConsoleManager  /server TEST /type 2 /username mdbadmin /password password /scope USD /method reint /database dsmmdb

I get this error:

Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_software NOCHECK CONSTRAINT [$ca_di_r000002c700000000]'
Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_hardware NOCHECK CONSTRAINT [$ca_di_r000002ac00000000]'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_discovered_software where asset_source_uuid in (select asset_source_uuid from ca_asset_source where subschema_id=1 and asset_source_uuid not in (select asset_source_uuid from ca_owned_resource where asset_source_uuid is not NULL))'
Exception in doUpdate: 'No valid datahase connection found', statement: 'update ca_owned_resource set asset_source_uuid=NULL where asset_type_id=1 and asset_source_uuid is not NULL'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_asset_source where subschema_id=1 and asset_source_uuid not in (select asset_source_uuid from ca_owned_resource where asset_source_uuid is not NULL and asset_type_id!=1)'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_logical_asset_property where logical_asset_uuid not in (select distinct logical_asset_uuid from ca_asset_source where logical_asset_uuid is not null)'
reregisterAll null
Assets Read = 0, Registration Successful = 0, Registration Failed = 0Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_software CHECK CONSTRAINT [$ca_di_r000002c700000000]'
Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_hardware CHECK CONSTRAINT [$ca_di_r000002ac00000000]'
All Done.

Username and password for database are ok, I can successfully create a view, for example.

Any clues? Does the java version have to be strictly 1.8.0_112?

thanks for the support

Giovanni

[Baerbel Veit]

Hi Giovanni,
did you follow Janos' instructions ? 

I think it is the java version, at least only version 1.8.0_122 did work on my systems.

Regards Baerbel

------------------------------
Senior Consultant
MAXXYS AG
------------------------------

[Giovanni Tamburlin]

Hi Baerbel,

Yes, we followed Janos instructions, and tryed first with java version 1.8.0_112, but we get this error: 

Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_software NOCHECK CONSTRAINT [$ca_di_r000002c700000000]'
Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\Program Files (x86)\CA\Service Desk Manager\bin\corajava.dll: Can't load AMD 64-bit .dll on a IA 32-bit platform
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(Unknown Source)
        at java.lang.ClassLoader.loadLibrary(Unknown Source)
        at java.lang.Runtime.loadLibrary0(Unknown Source)
        at java.lang.System.loadLibrary(Unknown Source)
        at com.ca.careg.CoraJava.loadLibrary(CoraJava.java:46)
        at com.ca.careg.CoraJava.init(CoraJava.java:66)
        at com.ca.Cora.util.ApiAgent.<init>(ApiAgent.java:174)
        at com.ca.Cora.util.ConsoleManager.run(ConsoleManager.java:737)
        at com.ca.Cora.util.ConsoleManager.main(ConsoleManager.java:1143)
Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_hardware NOCHECK CONSTRAINT [$ca_di_r000002ac00000000]'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_discovered_software where asset_source_uuid in (select asset_source_uuid from ca_asset_source where subschema_id=1 and asset_source_uuid not in (select asset_source_uuid from ca_owned_resource where asset_source_uuid is not NULL))'
Exception in doUpdate: 'No valid datahase connection found', statement: 'update ca_owned_resource set asset_source_uuid=NULL where asset_type_id=1 and asset_source_uuid is not NULL'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_asset_source where subschema_id=1 and asset_source_uuid not in (select asset_source_uuid from ca_owned_resource where asset_source_uuid is not NULL and asset_type_id!=1)'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_logical_asset_property where logical_asset_uuid not in (select distinct logical_asset_uuid from ca_asset_source where logical_asset_uuid is not null)'
reregisterAll null

I can't understand if corajava.dll is complaining about some dll not being a 32 bit dll, or if corajava.dll is a 64 bit dll and we can't use it with java 32 bit.

If I try the command with java 11.0.3, I get this error:

Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_software NOCHECK CONSTRAINT [$ca_di_r000002c700000000]'
Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_hardware NOCHECK CONSTRAINT [$ca_di_r000002ac00000000]'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_discovered_software where asset_source_uuid in (select asset_source_uuid from ca_asset_source where subschema_id=1 and asset_source_uuid not in (select asset_source_uuid from ca_owned_resource where asset_source_uuid is not NULL))'
Exception in doUpdate: 'No valid datahase connection found', statement: 'update ca_owned_resource set asset_source_uuid=NULL where asset_type_id=1 and asset_source_uuid is not NULL'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_asset_source where subschema_id=1 and asset_source_uuid not in (select asset_source_uuid from ca_owned_resource where asset_source_uuid is not NULL and asset_type_id!=1)'
Exception in doUpdate: 'No valid datahase connection found', statement: 'delete from ca_logical_asset_property where logical_asset_uuid not in (select distinct logical_asset_uuid from ca_asset_source where logical_asset_uuid is not null)'
reregisterAll null
Assets Read = 0, Registration Successful = 0, Registration Failed = 0Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_software CHECK CONSTRAINT [$ca_di_r000002c700000000]'
Exception in doUpdate: 'No valid datahase connection found', statement: 'Alter table ca_discovered_hardware CHECK CONSTRAINT [$ca_di_r000002ac00000000]'
All Done.

It's the same error I get with java 1.8.0_74.

[Baerbel Veit]

Hi Giovanni,
have a look, if in the environment-variable path
  C:\Program Files (x86)\CA\Service Desk Manager\REPLACED\HYD-368_CUM_C.OLD\bin
is placed before
  C:\Program Files (x86)\CA\Service Desk Manager\bin
otherwise the 14.1 files in the REPLACED-directory will not be used.

Regards
   Baerbel

------------------------------
Senior Consultant
MAXXYS AG
------------------------------

[Giovanni Tamburlin]

Thanks for the hint, the problem wan't the path (sort of) but pointed us in the right direction.

Our installation was not an update from version 14 (at least I don't think so: I did not partecipate in the update process and I'm just starting to work on this server), so we do not have a folder with old files. At this point, I thought to use ojdbc14.jar and sqljdbc.jar was pointless, so I restored coracleanup.cmd at its original state, using ojdbc8.jar and mssql-jdbc-7.2.2.jre8.jar.

Then, I took a look at the cora dll files: they're all 64 bit dll (verified with sigcheck.exe, a MS sysinternal utility), so I thought I should use a 64 bit java version, but using java 11.0.3 64 bit gave me the ucrtbase.dll error. Thinking maybe I should use java 32 bit anyway, I tried with java 1.8.0_112 32 bit, but I got "Can't load AMD 64-bit .dll on a IA 32-bit platform" error.

So I used java 1.8.0_74, 64 bit version, and coracleanup.cmd started successfully and processed all the records.

I think this version of cora we have works only with java 8, 64 bit. Maybe we had an update from Servicedesk 17.1 to 17.3? I don't know.

This is the final version of our coracleanup.cmd:

@echo off
setlocal
set NX_TEMP_CMD=%TEMP%\setenv.cmd
nx_env -d NX_ROOT > %NX_TEMP_CMD%
nx_env -d NX_JRE_INSTALL_DIR >> %NX_TEMP_CMD%
call %NX_TEMP_CMD%
del %NX_TEMP_CMD%

REM /* Build argument list */
set args=
:getargs
set args=%args% %1
shift
if not .%1 == . goto :getargs

REM /* Invoke the tool */

set exec_java=C:/Program Files/CA/SC/JRE/1.8.0_74

echo on

"%exec_java%/bin/java" -DLOG_CONFIG_DIR="%NX_ROOT%/site/cora_cfg" -ea -classpath "%NX_ROOT%/java/lib/corajava.jar;%NX_ROOT%/java/lib/cora_util.jar;%NX_ROOT%/java/lib/log4j-1.2.17.jar;%NX_ROOT%/java/lib/ojdbc8.jar;%NX_ROOT%/java/lib/mssql-jdbc-7.2.2.jre8.jar" com.ca.Cora.util.ConsoleManager %args%

Baerbel thanks again for the fast reply, and the decisive hint, that was greatly appreciated.

[Baerbel Veit]

Hi Giovanni,
do you have someware an old SDM Version 14.1 installation? You can create the REPLACED\HYD-368_CUM_C.OLD directory and copy the 14.1 files to this directory .
I had the same situation at customer site, a clean 17.3 installation with the migrated customisations. Fortunately I still had the old 14.1 system and could copy the files. 
coracleanup did work as expected using Janos' modifications.

Regards
   Baerbel

------------------------------
Senior Consultant
MAXXYS AG
------------------------------

[Giovanni Tamburlin]

No need for old files from 14.1 installation (in my case, at least), coracleanup worked just fine the way I described in the last post, using java 1.8.0_74 64 bit.

I'm no more getting errors when I try to import CI, all solved.

Before runnig coracleanup, I was getting errors AHD05543 and AHD05519, but now importing CI is working ok.

 

Thanks again for the reply.