v21 upgrade: Issues preparing the certificates

[Keld Mollnitz]

Hi folks,

We have started to look into the v21 upgrade, so far without any luck.

We are using CA signed certificates, this means I have created a csr and downloaded a signed server certificate for the JCP.
I have run into so many different issues, too many to mention here, but the latest issue is that the JCP cannot start and throws the following error:

U00045014 Exception 'com.automic.agents.impl.TlsKeystoreReader$InvalidKeystoreException: "com.automic.agents.impl.TlsKeystoreReader$InvalidKeystoreException"' at 'com.automic.agents.impl.TlsKeystoreReader.tryToLoadKey():89'.
U00045015 The previous error was caused by 'com.automic.agents.impl.TlsKeystoreReader$InvalidKeystoreException: "null"' at 'com.automic.agents.impl.TlsKeystoreReader.tryToLoadKey():85'.
U00003432 Termination of Server 'UC4S#CP001' initiated.

The keystore can be opened with KeystoreExplorer and everything looks fine.

I have had a couple of Web-ex sessions with Support, but so far without any breakthrough and now it is becoming time critical.
@Carsten Schmitz, I see that you reported some issues with the upgrade also, and I also see that you got some help from @Markus Embacher. I would apreciate if any of you guys could help me fix this issue.


/Keld.​​

[Pothen Verghese]

Yes there are issues with v21. For us it is not the certificate issue as we have tested with self-signed certs. Some of things that are still there with v21.01 are:

a) the reporting utility dumps core while all other utilites do work.
b) the OS agents when started before the AE is fully UP, will shut down and will not reattempt connection, while other agents like SQL, RA etc run and do reattempt the connection once AE is UP/reachable.
c)  the OS agent on AE server itself will shutdown when the server is started. It will then have to be restarted from AWI. Sometimes on AWI the status of this agent is shown as running but actually it is down.
c) Under process monitoring if we monitor any schedule object, there are overlapping images of tasks under Name coln.

Most of these are on Linux/Oracle setup. But b) is for all platforms. So one should do proper testing for such minor glitches before doing actual migration.

Regards
Pothen

[Pete Wirfs]

@Pothen Verghese  I was having problems with some OS agent startups under V12, and the solution was to increase the startup delay settings for those agents.

​

------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------

[Pothen Verghese]

Hi Pete,

Actually I had no such problem with v12.x. All these agents were on 12.x and last month moved it to v21. Here is the snippet of the issue when the agent starts up and AE is not reachable/UP.


No such problem with the other two agents on the same host (sqlsvr12). The AE Server starts after 5 mins.

Regards
Pothen

[Michael Dolinek]

Hi @Keld Mollnitz

​Did you already have a look at the section ​TLS/SSL Troubleshooting in our documentation (Administering and Configuring > Security and System Hardening > TLS/SSL Communication and Encryption > TLS/SSL Troubleshooting)?
https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/_Common/Security/Security_TLS_Troubleshooting.htm

Please also check that the alias configured in the ucsrv.ini file (default is jetty) and the one used in the keystore for the keypair match.
You can also use KeyStore Explorer to check the alias of the keypair/certificate within the keystore.

There is also a blog-post "What Kind of Certificates Should I Use for Automic Automation v21?" from my colleague @Oana Botez
​https://academy.broadcom.com/blog/aiops/what-kind-of-certificates-should-i-use-for-automic-automation-v21

You might find the knowledge base article "Keystore parameters are not valid for the given keystore / Keystore Format" useful
https://knowledge.broadcom.com/external/article?articleId=232306

[Marcin Uracz]

@Keld Mollnitz ​​do you have a working keystore for the JCP from the older versions 12.1 -> 12.3  ( to secure REST API with HTTPS ), you can use the exact same keystore also for securing the WS connection, of course assuming that the WS (JCP) and REST processes will run on the same machine. 


For importing certs that are issued by Lets Encrypt I do the following: 

! Convert the PEM certs to PKCS12
"C:\Program Files\OpenSSL-Win64\bin\openssl" pkcs12 -export -in C:\[Path to certs]\fullchain.pem  -inkey C:\[Path to certs]\privkey.pem  -out C:\temp\ae.p12 -name ae -CAfile C:\[Path to certs]\fullchain.pem  -caname "Let's Encrypt Authority X3" -password pass:&pass#

! Import the certs to new keystore, change the alias
C:\apps\Java\bin\keytool -importkeystore -deststorepass &pass# -destkeypass &pass# -deststoretype pkcs12 -srckeystore C:\temp\ae.p12 -srcstorepass &pass# -srcstoretype PKCS12 -destkeystore &keystore#
! Adjust the alias as JCP cannot cope with any other in 12.3
C:\apps\Java\bin\keytool -changealias -alias "ae" -destalias "jetty" -keypass &pass# -keystore &keystore# -storepass &pass#
! List the certs
C:\apps\Java\bin\keytool -list -v -keystore &keystore#

I use the "jetty" alias so you can keep the defaults as they are in ucsrv.ini 



Adding to the list of issues from @Pothen Verghese
1. After cold start of the engine only 12.3 Windows agents reconnected the rest (Windows 21 , Unix 12.3 , TLS Gateway, SQL, RA) had to be restarted in order to come online
2. There seem to be some encoding issue with german characters in User Profile on the Priviliges list (the umlauts are not displayed properly). Example:

FileTransfer: Ohne Angabe eines Login-Objekts ausf�hren

Objekt-Eigenschaften: Ge�ffnet-Kennzeichen manuell zur�cksetzen

I have not seen this issue in any other views, or while using the **** characters in object names, titles etc. 

3. Some details like the command to start the TLS-Gateway agent in the README.txt attached to the zip package downloaded from the Engine was no 100% correct. 

java -Xmx2GB -jar uctlsgtw.jar

My OpenJDK complained that there is no GB option. 

​

------------------------------
Cheers,
Marcin
------------------------------

[Jared Kessans]

I started working on it probably in November but after running into so many issues related to understanding certificates that I decided to put it off for a couple of months while I finish refreshing some of my servers related to the non-OS agents.  

It didn't help when my coworker retired in December.

[Carsten Schmitz]

@Keld Mollnitz apologies, I didn't see your message until now. I have unsubscribed from any Broadcom community notifications. I will respond in more detail to the group email from Broadcom that has you as a recipient. ​

[Kaj Wierda]

The subject of this thread was changed on request by original poster as the certificate issues have been resolved.

------------------------------
Kaj Wierda
Sr. Product Line Manager | Automation

Broadcom Software
------------------------------