Layer7 Identity Management

Expand all | Collapse all

Resolution Steps for: Provisioning Certificates that expired on 6th Oct,2017 & Directory DSA certificates that expired on 25th Nov,2017

  • 1.  Resolution Steps for: Provisioning Certificates that expired on 6th Oct,2017 & Directory DSA certificates that expired on 25th Nov,2017

    Posted 10-07-2017 07:53 PM

    These  steps should help address this problem as noted in recent advisory quickly

     

    Symptoms if expired out of box certificates not been addressed

    • New provisioning server/connector server connections or connection updates for example when acquiring new endpoints may not work
    • If you have made a change on any of your IDM servers and then restart your server for the change to take effect, you may see server restart may not happen due to failing connections to the provisioning/connector servers etc. 
    • You may see the following messages in your IM application server logs:

          Caused by: java.security.cert.CertPathValidatorException: timestamp check failed 
             Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sat Nov 25 01:26:01 MST 2017

    • You may see the following messages in your DSA warn logs:

          [56] 20171127.100608.055 WARN : Certificate 'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem'       is outside of validity date range

          [56] 20171127.100608.055 WARN : Unable to get certificate from       'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem'

          [56] 20171127.100608.055 WARN : set_cert_stuff failed

          [56] 20171127.100608.055 WARN : Cannot get personality

    How to Confirm if certificates are out of the box and have expired

    1) Provisioning certificates

    On each provisioning server, running the below command will show if the expiration date is Oct 6.

    C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\data\tls>..\..\bin\openssl x509 -enddate -noout -in et2_cacert.pem

    WARNING: can't open config file: /usr/local/ssl/openssl.cnf

    notAfter=Oct  6 08:25:50 2017 GMT -> in this example, this means certificate expired on Oct 6

     

    2) DSA certificates

    ------------------------------

    On the machines where the Provisioning Directory is installed, open a command prompt and run the command:

    dxcertgen report

     

    This command will list all the certificates and their validity dates. Expired certificates will be marked as invalid. Here is an example:

     

    - <hostname>-impd-notify.pem -
    certificate : 1
    version     : 3
    serialNum   : 311
    issuer      : /C=US/ST=NY/L=Islandia/O=Identity Management/OU=Provisioning Servi
    ces
    notBefore   : Nov 28 18:26:00 2007 GMT
    notAfter    : Nov 25 18:26:00 2017 GMT
    subject     : /C=US/ST=NY/O=Identity Management/OU=Provisioning Services/CN=eta_
    server
    status      : *** INVALID ***

    - <hostname>-imps-router.pem -
    certificate : 1
    version     : 3
    serialNum   : 311
    issuer      : /C=US/ST=NY/L=Islandia/O=Identity Management/OU=Provisioning Servi
    ces
    notBefore   : Nov 28 18:26:00 2007 GMT
    notAfter    : Nov 25 18:26:00 2017 GMT
    subject     : /C=US/ST=NY/O=Identity Management/OU=Provisioning Services/CN=eta_
    server
    status      : *** INVALID ***

     

    If the certificates have not expired, take note of the notAfter date for future reference.

     

    Steps to replace certificates

    If certificates are OOTB and have expired, proceed as follows:

    • Note the attached ootb_certs.zip (relevant only for IDM 12.6X releases) and ootb_certs_SHA1.zip (relevant only for IDM 12.5X releases) on this post. 

     Replace Provisioning Server Router DSA certs

    On each Provisioning Server (where you typically have imps-router DSA running):

     

    • Go to" Attach" towards bottom of post, and extract the relevant zip attachment (either ootb_certs.zip if on 12.6X or ootb_certs_SHA1.zip if on 12.5X) on this post.
    • Navigate to the pd folder and copy the impd_trusted.pem file to DXHOME\config\ssld location, and overwrite the existing one.
    • From the same pd folder, rename the provided imps-router.pem to reflect the actual  local hostname, and copy that into your DXHOME\config\ssld\personalities location and overwrite the existing one.
    • Delete any other .pem files related to 'imps' and 'impd' you have in there.
    • Basically on each Provisioning Sever host, you will end up with only one router .pem file reflecting the local router name.
    • Restart your DSA performing 'dxserver stop all' followed by 'dxserver start all' command.

     

    Replace Provisioning Directory DSA certs

    On each Provisioning Directory Server (where you typically have impd-main, impd-inc, impd-co and impd-notify DSAs running):

     

    • Take the same impd_trusted.pem used above in 1), and copy it to your DXHOME\config\ssld location and overwrite the existing one.
    • From that same ootb_certs.zip/ootb_certs_SHA1.zip extraction and pd folder, rename the provided impd files (ex. hostname-impd-co.pem) to reflect your local data DSA names, and then copy the files into your DXHOME\config\ssld\personalities location and overwrite the existing ones.
    • Delete any other .pem files related to 'imps' and 'impd' you have in there.
    • Basically on each Provisioning Directory host, you will end up with only four impd .pem files reflecting the four local data DSA names.
    • Restart your DSAs performing 'dxserver stop all' followed by 'dxserver start all' command.

     

    Replace Provisioning Manager Certs

    For Prov Manager you replace in two places.

    1) from package path "prov/data/tls/" -> on the host under <Provisioning Manager>/data/tls/

    2) from package path "prov/data/tls/client/ -> on the host under <Provisioning Manager>/data/tls/client

    3) Restart Provisioning Manager.

     

    Replace Provisioning Server Certs

    For Prov Server you replace in just one place.

    1) from package path "prov/data/tls/" -> on the host under <Provisioning Server>/data/tls/

    2) Restart Provisioning Server.

     

    Now you can follow information in Update Your Provisioning Certificates - CA Identity Manager - 12.6.8 - CA Technologies    Documentation starting at:

    NOTE: For both of the above, if you are running Java/JRE 1.5, the provided keytool command in the documentation will not work as that version doesn't support '-importkeystore' option. Your workaround would be to upgrade Java/JRE to at least 1.7 and the command should work.

    NOTE: 'Use Case 2' also applies to IDM 12.5X release (or you can use this TEC1561732 for the same)

    Attachment(s)

    zip
    ootb_certs.zip   8.81MB 1 version
    zip
    ootb_certs_SHA1.zip   30K 1 version


  • 2.  Re: Steps to address expired 6 Oct 2017 Provisioning certificates in IdentityMinder

    Posted 10-07-2017 08:27 PM

    Thank You Palaka for the steps. There are lot of confusions among the customers about this update, Could you also clarify the below ?

     

    -  Does this applicable to the customers who does use SSL communication in the provisioning directory/server ONLY ? or regardless of communication protocol everyone has to apply this update ?

     

    - Also what all are the IDM component's communications are getting affected by this cert expiry ? for example, IDM talks to Provisioning directory which could be SSL or NON-SSL which is configured in IDM Manage console. what else ?

     

    Appreciate your clarification on this.

     

    Thanks

    Ashok



  • 3.  Re: Steps to address expired 6 Oct 2017 Provisioning certificates in IdentityMinder

    Posted 10-07-2017 08:58 PM

    Ashok,

     

    The main link where download and instructions are provided actually lists out what components of Identity Manager this is applicable to.

     

    A quick example would be, Provisioning Manager will fail to start as it will use SSL to authenticate to local Provisioning Router DSA to chain the request to remote Provisioning Directory (impd-main) DSA due to expired root CA. If Provisioning Server is offline you can image what other cascading problems it will bring.

     

    For now (due to lack of time), I would recommend everyone should look at the main link, see what components are effected and use their judgment to make a decision on whether this is applicable to their specific setup or not.

     

    I know this is not a detailed answer but hope this helps somehow.

     

    Thanks,

    Hitesh



  • 4.  Re: IdentityManager- Steps to resolve Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-09-2017 11:23 AM

    Additionally, this problem could cause the CA Directory replication for the Provisioning Directories to get out of synch.

    In case CA Directory Multi-write (MW) recovery is needed here is a doc and instructions:

    It appears that you may need to perform a MW Data recovery.

    See Page 9:

    http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec479507.aspx

     

    Select a DSA that is completely up to date, as the one that you are going to copy the data from.  In the below example I'm using the 'dx1' DSA as the DSA that IS up to date.

    Please follow the below recommended steps which was created as DSA 'dx2' (Host 2) being the corrupt DSA, and 'dx1' (Host 1) being the up to date DSA:

    The following commands would need to be performed via command prompt.

    For Linux/Unix environments, you will need to su to the dsa user (su - dsa) before running the commands.

    - Shutdown DSA 'dx2' (dxserver stop dx2)

    - Backup / Delete existing db, dp and tx file ($DXHOME/data) for the 'dx2' DSA.

    - Perform dxdisp on Host1 for Host1 (by running the follow command: dxdisp dx1)

    - Perform dxdisp on Host1 for Host2 (by running the follow command: dxdisp dx2)

    - Perform dxdisp on Host2 for Host2 (by running the follow command: dxdisp dx2)

    - Perform dxdisp on Host2 for Host1 (by running the follow command: dxdisp dx1)

    - Copy over dx1.DB file ($DXHOME/data) from Host 1 to the same location on the Host 2 server.

    - Rename DB file on Host 2 accordingly (renamed the copied database from dx1.db to dx2.db).

    - Start DSA 'dx2' (dxserver start dx2)

    - Confirm MW is working properly.



  • 5.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-10-2017 05:02 AM

    Hi Guys,

     

    I have taken over support for our IDM Environment and looking at the documentation provided by CA I am going to have to run the keytool to update our Java Connector Server:

     

    1. At the command prompt, navigate to the above folder location and run the following command:

    keytool -v -importkeystore -srckeystore eta2_server.p12 -srcstoretype PKCS12 -destkeystore ssl.keystore -deststoretype JKS

    1. Specify the following passwords when prompted: destkeystore password Value: Java Connector Server password. srckeystore password Value: secret Note: Type "yes" when prompted to overwrite the server certificate.

    I can not find the initial destkeystore password is there a way I can find this in the config files located: H:\Apps\CA - Backup\Identity Manager\Connector Server\jcs\conf?



  • 6.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-11-2017 01:15 PM

    Musangwe,

       in our docs we call out a default password:

    Update Your Provisioning Certificates - CA Identity Manager - 12.6.8 - CA Technologies Documentation 

     

       Have you tried with the default password? step 5 under connector express.

     

    Bill



  • 7.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-12-2017 12:13 AM

    Hi Bill,

     

    Thanks for the response I was able to locate the passwords.

    But I have another question in the documentation under Connector Xpress it talks about updating the keystore:

    1.    Ensure that Connector Xpress is closed.

    2.    From the downloaded certificate zip file, copy the eta2_server.p12 and eta2_client.p12 files from the conxp folder.

     

    3.  /conf

    We only have two certificates both expired:

     

    ·        Ca_fips

     

    ·        Root_ca

     

    But when I check the location

     

    Kind regards,

     

    Musangwe Kalowa

    AMS Identity Access Management Consultant



  • 8.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-11-2017 11:20 AM

    If CA SSO (Siteminder) is involved and customer has implemented full integration we must also update the certificate in the CA SSO (Siteminder) key database:

     

    CA SSO (Siteminder)

    Copy the new impd_trusted.pem from the provisioning server to the Policy Server.

     

    Import the new pem file from the certutil in SiteMinder/bin (your paths may differ)

    the name (-n below) of the cert may differ in your environment as well

     

    Lists the certificates in the store
    certutil -L -d C:\certdatabase

     

    This shows information specific to the certificate named "Provisioning Certificate" such as valid until date:
    certutil -L -d C:\certdatabase . -n "Provisioning Certificate"

     

    Updates the certificate with the cert copied from the prov server:
    certutil -A -n "Provisioning Certificate" -t "P,," -i C:\certificates\impd_trusted.pem -d C:\certdatabase

     

    Then verify the new cert:
    certutil -L -d C:\certdatabase . -n "Provisioning Certificate"



  • 9.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-12-2017 04:17 AM

    Hi Bill,

     

    Thanks for this information I believe our environment is fully integrated IAM 12.6.02 and Siteminder  12.52 sp01 cr04.

    I have read through some documentation about verifying if enviromenmtns are integrated and I came across one that ask one to look for the following:

    "d="cf128b97-1ec6-11b2-82a1-ca5535a7a489" version="126.2.0.213" copyright="2011" info_url="" support_url="http://supportconnect.ca.com" location="C:\Program Files (x86)\CA\Identity Manager" last_modified="2013-10-18 10:23:12">
       <vendor name="CA" id="3b7cec0e-1ec0-11b2-968e-ca5535a7a489" home_page="http://www.ca.com" email=""/>
       <feature name="Extensions for SiteMinder (if SiteMinder is installed locally)" last_modified="2013-10-18 10:24:29">
       <![CDATA[IdentityMinder Extensions files for Policy Server]]>
        <component ref_id="e8f35342-1ec6-11b2-84a6-ca5535a7a489" version="126.2.0.213" location="C:\Program Files (x86)\CA\Identity Manager\.\install_config_info\im-uninstall\uninstall.exe"/>
        <component ref_id="802fbd5d-1ee7-11b2-ba58-8ff1afdfe42a" version="126.2.0.213" location="C:\\AppData\Local\Temp\3\591755.tmp\framework.properties"/>
        <component ref_id="49055ea0-1ee8-11b2-9cff-8ff1afdfe42a" version="126.2.0.213" location=""/>
       </feature>  "

    https://support.ca.com/us/knowledge-base-articles.TEC535804.html

     

    But When following your work instructions I have tried listing all the cert on our Siteminder Policy servers and I am unable to locate and certdatabase.

     

    I also went further and looked for the following three files:

     

    1. a Cert database with the following three files:

    cert8.db

    key3.db

    secmod.db

    I cant locate them either.

    does that mean our environment is not fully integrated or just doesn't have the Certdabase, if so do I need to install one?



  • 10.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-12-2017 08:41 AM

    Musangwe,

        Again, I have to point to documentation for this:

    https://docops.ca.com/ca-single-sign-on/12-7/en/configuring/policy-server-configuration/configure-policy-server-data-storage-options/configure-an-ssl-connection-to-an-ldap-data-store

     

        I do not know your SM version, so this is for 12.7, no changes since 12.52.

     

    under the heading

    Create the Certificate Database Files

    step two says:

    Enter the following command:

    certutil -N -d certificate_database_directory

    • -N
      Creates the cert8.db, key3.db, and secmod.db certificate database files.
    • -d certificate_database_directory
      Specifies the directory in which the certutil tool is to create the certificate database files.

     

     

    if you have access to the policy server management console you can check the data tab

     

     

    If there are still troubles after this post, please open a support ticket to help.



  • 11.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-12-2017 10:17 PM

    Hi Bill,

     

    Thanks for all the help.

     

    No we do not have a certdatabase file I checked Policy Management console and that value is blank.

    Our siteminder environment is 12.52 it is using ODBC as storage and AD as the user store.

    Kind regards,

     

    Musangwe Kalowa

    AMS Identity Access Management Consultant



  • 12.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-16-2017 01:54 PM

    Hi Palaka,

     

     i have performed all the steps you mentioned.

    But when i am trying to start the provisioning server .

    The below message is appearing 


    "im_ps failed to start within 120 seconds"

    Please help me on these



  • 13.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 10-16-2017 02:10 PM

    Hi SabreeW75350521,

    This cannot be worked out in community thread as there could be many things that we might to check and go back/forth. I suggest you open a support case.



  • 14.  Re: IdentityManager- Steps to resolve OOTB Provisioning Certificates that expired on 6th Oct,2017

    Posted 11-06-2017 09:35 AM

    STATUS UPDATE SINCE LAST COMMUNICATION:

    Due to this problem, an installation or upgrade to existing 12.x SPs or CRs will fail.  As a result, we have replaced the following on our Downloads and Docops sites:

    12.6 SP4 CR4

    12.6 SP5 CR2

    12.6 SP6 CR1

    12.6 SP7 (12.6.7)

    12.6 SP8 (12.6.8)

    12.6 SP8 CR1

     

    The only changes made to these versions are updates to the certificates related to the CA Directory in the provisioning server and provisioning directory.  There are certificates for the other components, including the provisioning server, that will still need to be manually updated after install or upgrade.  (Follow instructions below in the Problem Resolution section to replace the certs after upgrade/install.)  

     

    If you have downloaded any of these prior versions, please discard them and use the new downloads for an upgrade or fresh install.  If you have already replaced your certs, no action is needed unless you plan a new install or upgrade to one of these versions.  The Docops pages for each of the refreshed CRs have been updated with instructions on how to determine the difference between the original and updated CRs.

     

    SPs can be located on http://support.ca.com Download Management area.  CRs are on http://docops.ca.com

     

    We have Provisioning Certificate Utilities to enable you to verify whether the CA Identity Manager components are using the new certificates.  There is also a link to these for each of the CRs on Docops and can be found here: https://docops.ca.com/ca-identity-manager/12-6-8/EN/upgrading/upgrade-provisioning-components/ca-identity-manager-certificate-utilities

    (The instructions and downloads are the same for each SP and CR)

     

    Communities post on this topic to follow: https://communities.ca.com/thread/241785381-steps-to-address-expired-6-oct-2017-provisioning-certificates-in-identityminder

     
    PROBLEM RESOLUTION:


    Follow the instructions in the Important Notice in the documentation set for your release at:

     

    CA Identity Manager 12.6.08 (SP8) - https://docops.ca.com/ca-identity-manager/12-6-8/EN/release-information/release-notes-12-6-08-cumulative-patches

    CA Identity Manager 12.6.07 (SP7) - https://docops.ca.com/ca-identity-manager/12-6-07/en/release-information/release-notes-12-6-07-cumulative-patches

    CA Identity Manager 12.6.06 (SP6) - https://docops.ca.com/ca-identity-manager/12-6-6/en/release-information/release-notes-12-6-06-cumulative-patches

    CA Identity Manager 12.6.05 (SP5) - https://docops.ca.com/ca-identity-manager/12-6-5/EN/release-information/release-notes-12-6-05-cumulative-patches

    CA Identity Manager 12.6.04 (SP4) –  https://docops.ca.com/display/CIM12604/Release+Notes+-+12.6.04+Cumulative+Patches

     

    If you are using a release prior to 12.6.04 (SP4), please refer to the instructions for 12.6.04 (SP4) as these instructions are consistent and apply to all prior releases.

    For End of Service version 12.5, please see the following tech doc:

    https://support.ca.com/us/knowledge-base-articles.TEC1561732.html

     

     

    If you have any questions about this Critical Alert, please contact CA Support.
     
    Thank you,

    CA Support Team
    1-800-225-5224

    http://support.ca.com



  • 15.  Re: Resolution Steps for: Provisioning Certificates that expired on 6th Oct,2017 & Directory DSA certificates that expired on 25th Nov,2017

    Posted 11-29-2017 08:32 AM

    Latest Notification:

     

    This is a follow up to prior notifications sent out about the CA Identity Manager certificate expiration that occurred on Oct. 6th.    If you have followed all of the steps provided in the previous notification and restarted the Identity Management, provisioning and directory services there should be no issue. Please note that as part of the instructions previously provided there are Directory certificates that also need to be replaced.  Since these expired on Nov 25th, please double check these in case they were missed.  If you restart and have not replaced all of the certs problems will arise.  Please see below for further information.

     

    PRODUCT(S) AFFECTED: CA Identity Manager                      RELEASE: 12.x

     

    PROBLEM DESCRIPTION:

    The CA provisioning server ships with out of the box certificates, including Directory DSA, that for version 12.x were due to expire on 6th October 2017 and Nov 25th 2017. If you are using your own certificate or have deployed 14.0 or later, this does not affect you, as from 14.0 GA, the provisioning certificate shipped with the product has been updated with a newer one.

     

    POTENTIAL SYMPTOMS:

    New provisioning server/connector server connections or connection updates for example, when acquiring new endpoints may not work

    If you have made a change on any of your IDM servers and then restart your server for the change to take effect, you may see server restart may not happen due to failing connections to the provisioning/connector servers, etc. 

                You may see the following messages in your IM application server logs:

    Caused by: java.security.cert.CertPathValidatorException: timestamp check failed. Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sat Nov 25 01:26:01 MST 2017

     

                   You may see the following messages in your DSA warn logs:

    [56] 20171127.100608.055 WARN : Certificate 'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem' is outside of validity date range

    [56] 20171127.100608.055 WARN : Unable to get certificate from  'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem'

                [56] 20171127.100608.055 WARN : set_cert_stuff failed

    [56] 20171127.100608.055 WARN : Cannot get personality

     

    IMPACT:
    For any 12.6.08 (SP8) and earlier releases, if you are using expired certificates, it will cause any of the following: requests being sent to a provisioning server to fail; directory replication failure; provisioning server service not starting; connector server requests failure; IM environment fails to start; IM provisioning directory communication failure.

     

    PROBLEM RESOLUTION:

    Please follow How to Confirm if Certificates are out of the box and have expired section at the following post: https://communities.ca.com/thread/241785381-steps-to-address-expired-6-oct-2017-provisioning-certificates-in-identityminder

    If expired, please follow Steps to Replace Certificates section in the same post.

     

    Alternatively:
    Follow the instructions in the Important Notice in the documentation set for your release at:

    CA Identity Manager 12.6.08 (SP8) - https://docops.ca.com/ca-identity-manager/12-6-8/EN/release-information/release-notes-12-6-08-cumulative-patches

    CA Identity Manager 12.6.07 (SP7) - https://docops.ca.com/ca-identity-manager/12-6-07/en/release-information/release-notes-12-6-07-cumulative-patches

    CA Identity Manager 12.6.06 (SP6) - https://docops.ca.com/ca-identity-manager/12-6-6/en/release-information/release-notes-12-6-06-cumulative-patches

    CA Identity Manager 12.6.05 (SP5) - https://docops.ca.com/ca-identity-manager/12-6-5/EN/release-information/release-notes-12-6-05-cumulative-patches

    CA Identity Manager 12.6.04 (SP4) –

    https://docops.ca.com/display/CIM12604/Release+Notes+-+12.6.04+Cumulative+Patches

     

    If you are using a release prior to 12.6.04 (SP4), please refer to the instructions for 12.6.04 (SP4) as these instructions are consistent and apply to all prior releases.

    For End of Service version 12.5, please see the following tech doc:

    https://support.ca.com/us/knowledge-base-articles.TEC1561732.html

     

     

    If you have any questions about this Critical Alert, please contact CA Support.
     
    Thank you,

    CA Support Team
    1-800-225-5224

    http://support.ca.com