Automic Workload Automation

 View Only
  • 1.  AE upgrade to v21 and TLS certiciates

    Posted Dec 02, 2022 09:29 AM

    Hey guys,

    Recently we upgraded one of our customer's Dev (sandbox) env to v21.

    All went smoothly with the upgrade and the certificates for the AE processes. Although that we had at least 3 java installed on that server.
    We are using Interla CA for the certificates. 

    Few days after the AE upgrade I decided to upgrade some of the agents and here is where the funky business began :
    On a UNIX host we have both OS Agent and RA FTP agent.
    CAU packeges that were part of the 21.0.4 pack were uploaded to the Package Manager.
    Agent upgrade was initiated on the RA FTP agent and... it never came back online.
    When i check the logs i found out that the new connection string was not updated in the .ini file, despite that i can see the inipatch.txt in the bin folder.
    Ive manually updated the .ini file and start the agent - it failed with the error for missing cert. This was expected.
    The certificate was updated in the java cacert and the agent properly started this time.

    The RA Avaloq agent is completely different hellhole.... ( LINK )

    On the next day i decided to upgrade the OS agent also. I knew that the cert was no updated in the OS truststore but ive again initiated the CAU.
    Surprisingly the agent was upgraded and connected succesfully and the checkmark for TLS was there.

    So... how that happen? I dont believe that there is a way of UNIX agent to use the Java cacert.

    During the CAU process i saw that jcp1.cer was also downloaded to the server.

    The agent also made few files himself that are following his name .

    DERLTDAPO0011_ca.pem
    DERLTDAPO0011.cert
    DERLTDAPO0011.pem

    In the logs i can see the following :

    20221201/114637.662 - U02000377 Certificate loaded from file '/etc/pki/tls/certs/ca-bundle.crt'.
    20221201/114637.662 - U02000378 Loading certificates from directory: './trustedCertFolder'.
    20221201/114637.663 - U02000377 Certificate loaded from file './trustedCertFolder/jcp1.cer'.
    20221201/114637.663 - U02000398 Loading certificates from the directory './security' that is specified in the parameter'AgentSecurityFolder'.\
    20221201/114637.663 - U02000376 Could not parse certificate './security/DERLTDAPO0011.pem'. Please make sure that the certificate is in PEM format.
    20221201/114637.664 - U02000377 Certificate loaded from file './security/DERLTDAPO0011.cert'.
    20221201/114637.664 - U02000377 Certificate loaded from file './security/DERLTDAPO0011_ca.pem'.
    20221201/114637.683 - U02000004 Connection to Server 'UC4DEV#CP003' successfully created.
    20221201/114637.684 - U02000308 Agent certificate file './security/DERLTDAPO0011.cert' was loaded, expiration date '20231127/210644'.
    20221201/114637.684 - U02000308 Agent certificate file './security/DERLTDAPO0011_ca.pem' was loaded, expiration date '20421120/102252'.

    Also I found out that the keystore file is no longer used (.kstr) but the option Reset Agent Public Key are deleting the <agentname).cert BUT only when the agent is offline, and afterwards the AE is sending another such file.

    20221201/120301.089 - U00029412 Signature verification failure caused by missing public key in database for the agent: 'DERLTDAPO0011'.
    20221201/120301.089 - U02000297 Agent doesn't have valid certificate, requesting new one from server.
    20221201/120301.174 - U02000298 New certificate stored at path './security/DERLTDAPO0011.cert', expiration '20231201/093621'.

    Hence my questions would be :

    - How the OS (UNIX) agent started without valid cert in the truststore?
    - What are these cert and pem files that the agent made himself?
    - As per the logs they are valid for 1y. Shall we initiate manual extension/re-validation of these certs when the time comes?
    - Did anyone else found issues when upgrading RA agents via CAU?

    P.S. There is some big issue with the community website... Second time i am writing this post.



    ------------------------------
    ------------------------------
    Automic SME @ DXC.Technology
    ------------------------------
    ------------------------------


  • 2.  RE: AE upgrade to v21 and TLS certiciates

    Posted Dec 03, 2022 08:51 PM
    Hi Krum,

    I'll try to answer based on my experience.

    - How the OS (UNIX) agent started without valid cert in the truststore?

    If you dont define SSLCerDir= SSLCertFile= on ini , Agent searches for certificates in the most common SSL stores

    for example : Certificate loaded from file '/etc/pki/tls/certs/ca-bundle.crt'

    If you don't use your own certificates, you may need to update it every year for the unix agent.
    Of course, if you don't want to use it, you have to fill in the other sections in ini.


    - What are these cert and pem files that the agent made himself?

    If your agent and engine certificate are correct, it will automatically create these files.
    my suggestion is to set a common location for these files in the ini file. agentSecurityFolder=
    You should not delete these files.

    - As per the logs they are valid for 1y. Shall we initiate manual extension/re-validation of these certs when the time comes?

    yes, you have to renew it every year. That's what the certificate is all about :)

    - Did anyone else found issues when upgrading RA agents via CAU?

    I generally do not prefer upgrade for major version changes. Migration is always the safest method :)

    ------------------------------
    Olgun Onur Ozmen
    https://www.linkedin.com/in/olgunonurozmen/
    ------------------------------



  • 3.  RE: AE upgrade to v21 and TLS certiciates

    Posted Dec 04, 2022 10:41 AM

    Hey Olgun,

    Thanks for your inputs.

    But that doesn't really clarify the situation.

    Maybe i did not explained properly.

    We have our TLS/SSL root certificate from the internal CA , called internalCA.crt . 
    As per the install/upgrade steps this was converted to PKCS#12 format and deployed on the AE server, and ucsrv.ini adjusted.
    This is working fine.

    Upgrading OS agents :
    No certificates were manually deployed on the target servers(agent hosts).
    No manual configurations of the agent .ini files were done.
    The upgrade was initiated by CAU and completed successfully.
    The agents connected and are already TLS enabled.

    According the documentation there are only two ways to enable TLS communication and we didnt do neither of them.

    There are 2 options to use Internal CA-signed certificates for secure connections between the Automic Server and the
    agents:
    ● Copy and configure all agents to use the CA root certificate
    ● Install the CA root certificate to use the already available Java or OS truststores


    The CAU process deployed jcp1.cer on the agent host and created the additional 3 files for the agent (<agentname>.cert, .pem, _ca.pem)
    All of them are in their default folders (./security) and the other SSLdir/file are also the OS default.

    So the CAU is skipping the step for import to OS truststore? 

    We have control over internalCA.crt and the Interla CA will provide us new when the current is about to expire. That is understandable.

    But how the others, meaning jcp1.cert and <agentname>.cert, .pem, _ca.pem are beign updated?

    When the <agentname>.cert ones are about to expire, do we need to switch off all agents and manually click (Reset Agent Public Key)?
    This seems ridiculous...

    Or they are auto-updated when the internalCA.crt is beign renewed?

    Moreover the expiration date is fluctuating from restart to restart.

    20221201/110704.705 - U02000298 New certificate stored at path './security/DERLTDAPO0011.cert', expiration '20231130/141531'.
    20221201/122058.392 - U02000298 New certificate stored at path './security/DERLTDAPO0011.cert', expiration '20231130/084551'.
    20221201/120301.048 - U02000308 Agent certificate file './security/DERLTDAPO0011.cert' was loaded, expiration date '20231127/210644'.
    20221201/120301.174 - U02000298 New certificate stored at path './security/DERLTDAPO0011.cert', expiration '20231201/093621'.


    I assume by "your own certifites" you are refering to self-signed, right?
    And would you elaborate on "I generally do not prefer upgrade for major version changes. Migration is always the safest method"?What do you mean by migration.



    ------------------------------
    ------------------------------
    Automic SME @ DXC.Technology
    ------------------------------
    ------------------------------



  • 4.  RE: AE upgrade to v21 and TLS certiciates

    Broadcom Employee
    Posted Dec 07, 2022 03:26 AM

    Hi Krum,

    here my answers in red :-)

    We have our TLS/SSL root certificate from the internal CA , called internalCA.crt .
    As per the install/upgrade steps this was converted to PKCS#12 format and deployed on the AE server, and ucsrv.ini adjusted.
    This is working fine.

    Upgrading OS agents :
    No certificates were manually deployed on the target servers(agent hosts).
    No manual configurations of the agent .ini files were done.
    The upgrade was initiated by CAU and completed successfully.
    The agents connected and are already TLS enabled.

    According the documentation there are only two ways to enable TLS communication and we didnt do neither of them.

    There are 2 options to use Internal CA-signed certificates for secure connections between the Automic Server and the
    agents:
    ● Copy and configure all agents to use the CA root certificate
    ● Install the CA root certificate to use the already available Java or OS truststores


    The CAU process deployed jcp1.cer on the agent host and created the additional 3 files for the agent (<agentname>.cert, .pem, _ca.pem)
    All of them are in their default folders (./security) and the other SSLdir/file are also the OS default.

    So the CAU is skipping the step for import to OS truststore?

    Definitely not. Part of the CAU process is that each package (before download, gets the JCP certificate jcp1.cert) which is recognized by the "new" agent and copied into the trustedCertFolder. Which basically enable the new Agent to connect to the JCP.
    The other certificate sources/folders are used every time, mean  trustedCertFolder= + (SSLCertFile=/SSLdir=) + agentSecurityFolder= folders are the sources from where the certificates will be loaded.   

    We have control over internalCA.crt and the Interla CA will provide us new when the current is about to expire. That is understandable.

    But how the others, meaning jcp1.cert and <agentname>.cert, .pem, _ca.pem are beign updated?

    • The jcp1.cert will be received only with CAU, just for the update case. But the idea is, that you have to care about the certificate on your own (how to roll it out). It's platform depend and usually the part of the standard infrastructure (SSLDir, Windows certificate management or Java Trust Store) ... This covers the case when Agent want to connect the JCP/AE. 
    • So now you have secure connection between JCP and Agent ... But for now you want to make sure that the Agent which want to connect to the AE is really the one you expect. "Agent authentication concept" (see more here). So as part of the authentication the Agent and AE will generate the key pair <agentname>.cert and <agentname>.pem ... So that AE can verify the Agent at logon.  
    • But there is another case, when the Agents want to start File-Transfer (it's peer to peer connection). Here also the <agentname>_ca.pem will be used.

    These certificates are part of the AE infrastructure (therefore the <agentname> prefix), so that you don't have to care about the connection/machine name for each File-Transfer ...  

    When the <agentname>.cert ones are about to expire, do we need to switch off all agents and manually click (Reset Agent Public Key)?
    This seems ridiculous...

    As mentioned these certificates are generated and controlled by the AE server, you can control it here: UC_AGENT_TLS_SETTINGS - Agent Certificate Management

    But to answer the question, this happens in background, automatically and is part of the "logon" process, therefore the message like:
    > U02000297 Agent doesn't have valid certificate, requesting new one from server. <


    The new certificate will be used after the agent tries to reconnect ... 

    Or they are auto-updated when the internalCA.crt is beign renewed?

    Moreover the expiration date is fluctuating from restart to restart.

    20221201/110704.705 - U02000298 New certificate stored at path './security/DERLTDAPO0011.cert', expiration '20231130/141531'.
    20221201/122058.392 - U02000298 New certificate stored at path './security/DERLTDAPO0011.cert', expiration '20231130/084551'.
    20221201/120301.048 - U02000308 Agent certificate file './security/DERLTDAPO0011.cert' was loaded, expiration date '20231127/210644'.
    20221201/120301.174 - U02000298 New certificate stored at path './security/DERLTDAPO0011.cert', expiration '20231201/093621'.


    I assume by "your own certifites" you are refering to self-signed, right?

    Once agent get connected the expiration date should not change (mean only in respect to the settings). Except you will reset the public key or delete the agent etc ... 

    And would you elaborate on "I generally do not prefer upgrade for major version changes. Migration is always the safest method"?What do you mean by migration.




  • 5.  RE: AE upgrade to v21 and TLS certiciates

    Posted Dec 07, 2022 05:28 AM

    Thanks!

    That shines some light.

    During the CAU the jcp1.cert is deployed and it acts as temporary certificate until we deploy the 'real' certificate in the respective truststore.

    This part was missing from the documentation and caused the confusion from our end.

    I was truly expecting the agents to not be able to connect even upgraded via CAU until we deploy the cert (as this was the official procedure described). And that happen with the RA agent but it can be due the not-updated .ini file.



    And shortly the previous used keystore (.kstr) file is now replaced with the three <agentname> ones. Logic is the same, more or less.



    ------------------------------
    ------------------------------
    Automic SME @ DXC.Technology
    ------------------------------
    ------------------------------