Symantec SiteMinder

 View Only

Tech Tip : CA Single Sign-On : Data Protection, Key Management,Configuration & Common Issues

By Ujwol posted Aug 21, 2017 07:47 PM


The Policy Server and Agents utilizes various encryption keys to encrypt sensitive data stored & passed between CA SSO components in a CA Single Sign-On environment.


The following diagram gives an overview of the various encryption keys used in CA SSO environment :

What are the purpose of various encryption keys used by CA SSO ?


Policy Store Key

Policy store key is used to encrypt :

  • Sensitive data stored in policy store (e.g LDAP bind credential, trusted host shared secrets etc.)
  • Sensitive data stored in policy server management console ( e.g policy store/audit store/session store credential )
  • Key store data when policy store and key store are collocated.


Key Store Key

Key store key is used to encrypt the Agent Keys & Session Ticket Keys stored in Key Store when policy store and key store are not collocated. 

When separate key store is not configured, the Key store key is same as the Policy store key.


Agent Keys :

  • Agent keys are used to encrypt/decrypt CA Single Sign-On cookies sent to the browser
  • Agent keys are managed by the Policy Server, and distributed to agents periodically.


Session Keys

Session keys are used to encrypt :

  • Data sent to and from Policy server to web agent
  • Data sent to and from Policy server to Administrative UI


Session Ticket/Persistent Keys

  • Session ticket keys are used by the Policy Server to encrypt session tickets (specs). Session tickets contain

credentials and other information relating to a session (including user credentials). Agents embed session tickets in CA Single Sign-On cookies, but cannot access the contents since they do not have access to session ticket keys which never leave the Policy Server

  • CA SSO also provides the ability for user tracking so a site can remember that a user had a valid session with the site some time ago. If user tracking is enabled, the Identity Spec is generated every time a new Session Spec is generated. The Identity Spec is also encrypted with the Session Ticket Key known only to the Policy Server and passed back to the Web Agent. 
  • Session ticket keys are also used to encrypt the password services data (blob) stored in User Store.


More about Session Ticket Keys here :


What is the impact of resetting Persistent Key/ Session Ticket Key?

Resetting persistent Key has following impacts :

  • Existing logged in user sessions will not be valid anymore. User will have to re-login to establish a new session.
  • Existing password blob will be no more be valid, which means all the information related to password change, login tracking etc. is lost


Policy server Host Key

A built-in static (hard coded) key stored in Policy server binaries.

It is used to encrypt/decrypt data stored in EncryptionKey.txt.


Web agent Host Key

A built-in static (hard coded) key stored in Web agent binaries.

It is used to encrypt/decrypt shared secret stored in SmHost.conf along with the hostId of the machine.


Shared Secret

The shared secret is used to mutually authenticate the Agent and the Policy Server and to distribute the session keys from the Policy Server to the Agent.


What is stored in EncryptionKey.txt ?

Policy Store Key is stored as encrypted string in EncryptionKey.txt file.


Here is what happens during the policy server installation :



During Policy server startup :


Policy store key is stored in Policy server memory.


How is shared secret for Trusted Host created and stored ?

When registering a Trusted Host, a shared secret is auto generated. This auto generated shared secret is then encrypted using the combination of Web Agent Host Key and the HostId of the machine and stored in the SmHost.conf file on the web agent side.


On the policy server side also, the corresponding shared secret for the Trusted Host is encrypted with the Policy store key and stored in the policy store :


If these shared secret do not match, the agent to policy server handshake fails.


How is the communication channel between Policy server & Web Agent protected ?

CA SSO uses RC4 encryption with randomly-generated 128-bit session keys to encrypt data sent over a TCP connection between a Policy Server and an Agent. The shared secret is first used to mutually authenticate the Agent. Once the handshake (authentication) is complete Policy Server distributes the session keys to the Agent.


Encryption of CA SSO Cookies

There are three major types of cookies that may be sent with every request (there are other cookies that are used only during authentication, like FORMCRED cookies):

  • SMSESSION - contains session ticket, always present , encrypted with either static or dynamic agent keys depending upon configuration
  • SMIDENTITY - identity cookie. Present only if the "User Tracking" option is configured, always encrypted with static agent keys.
  • SMDATA - cookie that keeps user credentials. Present only if the "Allow Form Authentication Scheme to Save Credentials” option is configured, always encrypted with static agent keys. 


How and where is the Key store key stored while using separate key store ?

If a key store is configured as a separate store from the policy store, then the key store encryption key can be set manually from the Policy server management console --> Keys tab :


The provided value is then encrypted with the Policy store key and stored in the CA SSO registry as below :



What are the various FIPS mode supported by CA SSO ?


CA SSO can be configured to operate in three FIPS modes :

  • Compat Mode - read both FIPS/Non Fips always write non FIPS keys
  • Migration Mode - read both FIPS and non FIPS - always generate FIPS keys
  • FIPs Only Mode - only read/write FIPS keys

While operating in Compat Mode, it uses RC4-128 bit cipher (Session Keys) to encrypt traffic between Policy Server and Web Agent.
While operating in Migration Mode or FIPs Only Mode, it uses AES-128 bit cipher to encrypt traffic between Policy Server and Web Agent.
CA SSO embeds RSA 's Crypto-C ME v2.0 cryptographic library, which has been validated as meeting the FIPS 140-2 Security Requirements for Cryptographic Modules. The validation certificate number for this module is 608. CA SSO’ s Java-based APIs use a FIPS-compliant version of the Crypto-J cryptographic library.
In FIPS-only mode or Migration Mode, CA SSO uses the following algorithms:

  • AES Key Wrap for key encryption.
  • AES in OFB mode (HMAC-SHA 256) for channel encryption.
  • AES in CBC mode (HMAC-SHA 224) for encrypting tokens used to facilitate


Types of Agent Keys

The Policy Server provides the following two types of Agent keys:

  1. Dynamic
    Generated by Policy Server and distributed to connected Policy Servers and any associated Web Agents
    Can be rolled over at regular interval or manually by using the Key Management feature in the UI.

2. Static
Remain the same indefinitely
Can be generated or entered manually. When using a specified manual value , the static agent key will be derived based on the value provided. It won't be the exact same string.

Can be rolled over using the UI


There are in total 4 different agent keys , they are :

  • (Key Marker 1) An Old Key is a Dynamic key that contains the last value used for the Agent key before the current value.
  • (Key Marker 2) A Current Key is a Dynamic key that contains the value of the current Agent key.
  • (Key Marker 3) A Future Key is a Dynamic key that contains the next value that will be used as the Current key in an Agent key rollover.
  • (Key Marker 4) Static Key


Agent Keys can be exported by running following command :

smkeyexport -d<admin> -w<password> -okeys.txt


Sample Key export file :


#! SiteMinder Version 12.52
# Export Flags: Encrypted, Export Keys, Export Key store data.
objectclass: Root
Oid: 0a-00000000-0000-0000-0000-000000000000
KeyManagement: 1a-fa347804-9d33-11d3-8025-006008aaae5b
AgentKeys: 1b-a0b79a43-eca3-4090-9082-4a30604fd108, 1b-912d25e3-26c0-4103-9c08-397733217fee, 1b-87336696-825f-4c71-b919-2bf2cb61578f, 1b-68860af4-01ff-4227-a034-795df2b93c99
objectclass: KeyManagement
Oid: 1a-fa347804-9d33-11d3-8025-006008aaae5b
IsEnabled: true
ChangeFrequency: 2
ChangeValue: 0
NewKeyTime: 1502334000
OldKeyTime: 1502248172
FireHour: 3
PersistentKey: {RC2}VDPKLgZZDJ3mEjM3WzphnvBt2GCIQrNqa6TR174l279K6QLPC0dhZRlPNLvCp/A/
objectclass: AgentKey
Oid: 1b-a0b79a43-eca3-4090-9082-4a30604fd108
KeyMarker: 1
Key: {RC2}r0T7TDNWvPME3VOr6b+43YJjULngsqGsHcMBxsVnuk09Ijh7jsPe5+4xs/OccTvx
objectclass: AgentKey
Oid: 1b-912d25e3-26c0-4103-9c08-397733217fee
KeyMarker: 3
Key: {RC2}6XtDG3PqFgJ5t5JCXiy0S2Ohc6eIv5sNr6Pi06JfXR/hGfyJbvTUtnGfKcacX3kc
objectclass: AgentKey
Oid: 1b-87336696-825f-4c71-b919-2bf2cb61578f
KeyMarker: 2
Key: {RC2}ZDUJcusH5LBcutHqWdMNTxoL78LpXsRQ4OdLeZRIyXwJAzWZckh9H2uXxi9svAFX
objectclass: AgentKey
Oid: 1b-68860af4-01ff-4227-a034-795df2b93c99
KeyMarker: 4
Key: {RC2}5fIwrHQHpgb4ycaZcvYNmAQ2mY4PCgADZW3GMzlyxvUsF8F5nN1h0gEd9rOpNJmm


Note : 

  • If not using the clear text export option (-c) , the exported encrypted agent key values always comes as different. 
  • The leading {RC2} or {AES} string indicates the keys are encrypted.


How to configure Agent Keys & Session Ticket Key if using Multiple Policy store with Separate Key Store


If a network configuration is composed of multiple Policy Servers, policy stores, and master key stores, an administrator with appropriate privileges can specify the same static key and session ticket key for each policy store in order to facilitate one or more of the following:

  • Single sign-on across all Agents
  • Password Services with a common user directory


Common Issues with Key(s)

1. Duplicate Agent Keys



SSO fails with the following error in the web agent trace log :

Unable to decode SMSESSION cookie




To identify if your key store have duplicate set of agent keys, perform the key store export using the following     command :

smkeyexport -d<admin> -w<password> -okeys.txt


Then count the number of AgentKey in the export file. If there are more than 4 agent keys, then it means you have

duplicate set of agent keys.

If there are more than 4 agent keys, there will be no guarantee which set of keys an Agent will utilize if more than one set is delivered from the Key Store on Agent start up.
Consider a scenario, that there are two set of agent keys - set 1 & set 2. Now, if Web Agent 1 utilizes set 1 and Web Agent utilizes set 2, the SMSESSION cookie encrypted by one agent will not be decoded by another agent eventually breaking the SSO.


Common cause for duplicate agent keys :

When using dynamic agent keys, it is required that only ONE policy server is configured to generate agent keys. If multiple policy server generates agent keys , it will most likely end up with duplicate set of agent keys.

Key store could also end up with duplicate agent keys during agent key export and import

Refer : Tech Tip : CA Single Sign-On:: Policy Server : Best practice on importing Agent Keys  


Resolution :

    KB : Cleaning up duplicate agent keys How to Clean up a SiteMinder Key Store? 


2. "No Session" error in Administrative UI  and "Failed to decrypt persistent key error" in SMPS log.



  • "No Session" error in Administrative UI while trying to access Key Management --> Agent Key /Session Key Management option.

  • Policy server log  (smps.log ) shows error : "Failed to decrypt persistent key"

[2088/972][Wed Aug 09 2017 04:26:41][SmObjKeyManagement.cpp:459][ERROR][sm-Server-03080] Failed to decrypt persistent key


Common Cause:

The persistent key (session ticket key ) is encrypted using Policy store key (or Key Store Key if using separate key store). The encrypted Policy store key is stored in EncryptionKey.txt file on the policy server bin directory. So, this error indicates that the current Policy store key (EncryptionKey.txt) is unable to decrypt the persistent key from the key store.


Such a situation can occur if, for example, the EncryptionKey.txt file was changed or copied from another machine or the persistent key was created by the policy server with the different Policy store Key (EncryptionKey.txt)



1. (Preferred) If there is a backup of the original (valid) EncryptionKey.txt or the Persistent Key (in the key store) try reverting to it and see if that works.

2. If the prior solution does not work then proceed to do following steps which basically creates a new Persistent Key in the key store.


a) Stop Policy server

b) Stop Administrative UI

c) Set following registry :


DWORD key:


Value: 1

(If not already existing, create this registry. What this registry does is, even if the Policy server is unable to decrypt the existing persistent key , it will use empty persistent key to encrypt the sensitive data in the policy store )

d) Start Policy server 

e) Start Administrative UI

f) Login to the Administrative UI and navigate to Key Management --> Session Key Management tab

  ( You won't get "No session" error after setting above registry )

   Either click Rollover Now under Generate Random Session Ticket Key or Specify a Session Ticket Key and click

   Rollover Now button under it.

g) Once the new persistent key is created, either delete the registry key (AllowEmptyEncKey) created above or set the value to 0. For security reason , it is strongly advised to do not leave the AllowEmptyEncKey=1 in the production server.

h) Restart Policy server

i)  Restart Administrative UI


3. Key store export shows error "Unable to decrypt agent key with policy store / key store key"



While performing the key store export it shows following error :


C:\Users\Administrator>smkeyexport -dsiteminder -wsiteminder -oc:\keyenc3.txt
Unable to decrypt agent key with policy store / key store key
Unable to decrypt agent key with policy store / key store key
Unable to decrypt agent key with policy store / key store key
Unable to decrypt agent key with policy store / key store key


Common Cause:

This is similar to "Failed to decrypt persistent Key" error as discussed above.

Agent Keys are also encrypted with the Policy store key (Or Key store key if using separate key store) .

So if the Policy store key changes (change of EncryptionKey.txt) , the policy server will no longer be able to decrypt the Agent keys.


Resolution :

1. Stop policy server which is configured to generate agent keys.

2. Delete all existing agent keys directly from the key store


For RDBS : delete from smagentkey4

For LDAP , you may use the ldapmodify command or your GUI interface to sequentially select and delete all keys.

Example command:

# ldapmodify -D "cn=directory manager" -w dirmanagerpassword -h localhost
dn: smAgentKeyOID4=1b-4a79595f-9a40-1000-a34a-830cefdf0cb3, ou=PolicySvr4,ou=SiteMinder,ou=Netegrity,o=ghost
changetype: delete

(Note: The example commands are for example only and will need to be modified for your environment.)

3. Start Policy server

During the startup, if the policy server configured to generate agent keys doesn't find any agent keys, it will create it.

By default Policy server creates all 4 agent keys with the identical values :