random ca.pem errors with file transfers?

[Daryl Brown]

We've been running into this issue for a while now in our 24.3 environment, and are still working with Broadcom Support to get to the bottom of it.

We have a ton of file transfers that we need to do between various agents, and while we've been in the process of validating that everything works before we migrate various workflows into this 24.3 environment, we've been running into a lot of file transfers erroring out like this:

20250110/174713.406 - U00063097 FT '3070179': Initiating connection to Agent 'VMAZ1UTUC4ARC06'.
20250110/174713.407 - U00063013 FT '3070179': Connecting to '10.86.95.12' at port '23020'.
20250110/174713.410 - U02000407 Initiating connection '1' to server using WebSocket URI: 'wss://10.86.95.12:23020/agent'.
20250110/174713.456 - U02000378 Loading certificates from directory: '/opt/aeluat/agent/bin/./security'.
20250110/174713.458 - U02000377 Certificate loaded from file 'NXUVARCS01001_ca.pem'.
20250110/174713.528 - U02000385 Web socket error: 'SSL_HANDSHAKE_EXCEPTION'.
20250110/174713.528 -           IOException: javax.net.ssl.SSLHandshakeException: signature check failed
20250110/174713.529 -            -> SSLHandshakeException: signature check failed
20250110/174713.529 -            -> ExtendedCertificateException: signature check failed
20250110/174713.529 -            -> SignatureException: Signature does not match.
20250110/174713.529 - U00000096 SSL Certificate validation error: 'VMAZ1UTUC4ARC06'.
20250110/174713.546 -           Root certificate details -> Issued To:'CN=AE Agent Certificate'   Issued By:'CN=AE Agent Certificate'   Valid to:'Mon Jan 02 12:58:05 EST 2045'   Serial:'1014500577571917941685596234770587331478683926940'
20250110/174713.548 - U02000230 FT '3070179': Thread 'Thread[FT3070179,5,main]' ended.
20250110/174713.585 - U00045014 Exception 'java.nio.file.AccessDeniedException: "./security/VMAZ1UTUC4ARC06_ca.pem"' at 'sun.nio.fs.UnixException.translateToIOException():90'.
20250110/174713.586 - U00003620 Routine 'com.uc4.ex.AdminRoutine' forces trace because of error.
20250110/174713.587 - U00003450 The TRACE file was opened with the switches '0000000000000000'.
20250110/174713.928 - U00003449 Output to the TRACE file is finished.

This isn't happening with every pair of agents, but these errors do occur consistently until we take action.  What's odd is that those same two agents are able to do file transfers with other agents just fine.

Broadcom Support has indicated that when we see errors like this, we're supposed to stop the agent + delete the files from the agent/bin/security/ folder(s) + reset the agent public key (from client 0) + restart the agent.  We've done this, and it has sometimes fixed the issue -- not always -- but we've also seen this "solution" cause other file transfers involving these agents that were previously working to now start throwing similar errors.  So I don't have a lot of faith in this solution right now.

Has anyone else encountered these issues in their environment?   If so, how did you address it?

[Ralf Schmidt]

Hello Daryl,

is it possible that there are several agents running on the system that log on to different automation systems and the agentSecurityFolder parameter in the INI files points to the same directory?

 

If so, it could be helpful to use different directories here.

Greetings

Ralf

[Daryl Brown]

We do have multiple agents installed on many of these servers (v12 agent for our current env, and v21/v24 agent for the env we're upgrading to), but all of the agents are using different install folders, so they would all have distinct agent/bin folders.

There are a handful of servers where we have both UAT and Prod agents installed (both v21/v24), but again, those agents would be using separate agent/bin folders.  The ca.pam errors we're getting aren't exclusive to those servers, though.  Some of the errors are occurring on servers that only have one (v24) agent installed on them.

[Daryl Brown]

Update:

We've made another discovery here.  The file transfer issues seem to occur when the two agents in question had made their initial connections to different AE servers to obtain their *ca.pem files in the agent/bin/security folder.  In these cases, the file transfers end up failing because the signatures in these two ca.pem files don't match.

When we've reset these agents (meaning cleared the agent/bin/security folder, reset the public key, and restarted the agent) and confirmed that both source and destination agents are using ca.pem files obtained from the same AE (regardless of which JCP that might have been), then our transfers work.

Now we've since obtained a new multi-server cert listing the names of both of our AEs in the same cert, and deployed this to both of our AEs.  I had expected this would result in agents' ca.pem files looking the same regardless of which AE they connect to now (since the cert on those AEs are the same), but we're still having the same problem -- the ca.pem files look different depending on which AE they connect to.

Has anyone else experienced this?  Or have any more insight on why the signatures on these ca.pem files would be different?

[Markus Embacher]

Hi Daryl,

I had a chat with @Oana Botez: the certificates you see in /agent/bin/security are internal certificates which the Automation Engine issues. Those certificates are used to encrypt the file transfer between agents. Every AE will have a different internal CA in order to create those internal certificates. That is why you are seeing different ca.pem files for every AE. If you would like to use your own CA for those internal certificates please have a look at:

https://docs.automic.com/documentation/webhelp/english/ALL/components/DOCU/24.3.0/Automic%20Automation%20Guides/Content/AWA/Variables/UC_AGENT_TLS_SETTINGS.htm

The issue you are facing is because of the following reason:

*) agent connects to AE1 and receives ca.pem file to store in /bin/security
*) now you connect the same agent to AE2, the agent does not receive a new ca.pem file, because the existing one is still valid
*) as soon as you run a file transfer the agent uses the existing ca.pem file, but it does not match the CA which AE2 is using and therefore the TLS handshake fails
*) after cleaning up the /bin/security folder the agent receives a new ca.pem file (signed by AE2) and the file transfer works as expected

Hope that help!

Regards, Markus

[Daryl Brown]

Thanks Markus, although I'm not sure I fully understand your reply...

If I try to do a transfer between two agents that are each using a ca.pem file from a different AE (which is a perfectly legitimate scenario), that shouldn't prevent the file transfer from working, should it?

We did make an additional discovery since my previous post, too: the "java.nio.file.AccessDeniedException" error is something we only see when dealing with java agents -- i.e., v24 agents.  Once we downgraded both agents involved in our file transfers back to v21, the problems went away.

We had opened case#36060655 with Broadcom support, and they did ultimately confirm some sort of defect, that is supposed to be fixed in 24.4.0.