Performance Notes for CA Identity Manager (IM)

[Alan_Baugher]

Hello All,

I have been updating my deck around performance of various solutions with regards to technical and business processes.

I collected over thirty (30) processes that range from rapid to those that require updates of an architecture.

 

I have refined the top high-value processes that can be deployed rapidly and with a listing of impacts.

I have put together an updated deck that lists these performance enhancements; along with some useful debugging processes; and a feasible method to document your complex business processes within the IM solution.

 

The only bottleneck in your solution stack should be database I/O or endpoint responses.  Don't let any other tier/component prevent the solution from being under utilzed.

If these processes don't give you the performance you are looking for, it will be necessary to dive a bit deeper or perhaps review the solution architecture.

 

 

Regards,

Alan Baugher

 

 

 

Edit 09/10/2015  -  Added updated IM performance deck with strategic steps as well.

Attachment(s)

Improve the performance of CA IdentityManager (IM)-2015-0...pdf   5.25 MB 1 version

Steps to Rapidly Improve the performance of IM - 2013-12-...pdf   4.65 MB 1 version

[Christopher Hackett]

alan_baugher:

Hello All,

I have been updating my deck around performance of various solutions with regards to technical and business processes.  

I collected over thirty (30) processes that range from rapid to those that require updates of an architecture.

 

I have refined the top high-value processes that can be deployed rapidly and with a listing of impacts.

I have put together an updated deck that lists these performance enhancements; along with some useful debugging processes; and a feasible method to document your complex business processes within the IM solution.

 

The only bottleneck in your solution stack should be database I/O or endpoint responses.  Don't let any other tier/component prevent the solution from being under utilzed.

If these processes don't give you the performance you are looking for, it will be necessary to dive a bit deeper or perhaps review the solution architecture.

 

 

Regards,

Alan Baugher

Thanks for sharing this with the community Alan!

[Anon Anon]

Thanks Alan for sharing, this helped us a lot

[Alan_Baugher]

Hi Itamar,

 

Thanks for the note.   I am continually updating this deck from my field notes & working with the support team.

 

Some of these items have made it into the IM bookshelf.

 

Now that IM r12.6.4 is out, (7/11/14) I will likely update my references.

 

 

The two (2) most common feedback responses I get back on usefulness are TP cleanup and Entropy that were of great help.

 

Cheers,

[Anon Anon]

well, heap configuration is very important as well thread pool.

 

so, in our project, we use WAS 7 and i ended up changing the default thread pools for the WebContainer and SIBJMSRAThreadPool to higher number as we noticed that not doing that can cause Out-Of-Memory errors

 

Also, one need to make sure the DB server (in our case Oracle 11gR2) has all the fix packs (ours had a memory leak)

 

as for heap, i found out that in WAS 7 running 12.6 SP2, you need to have the following heap settings:

 

Initial Heap Size: 1024

Maximum Heap Size: 4096

 

Java options:

 

-Xgcpolicy:gencon -Dsun.reflect.inflationThreshold=0 -Xdump:none -Dcom.sun.jndi.ldap.connect.pool.protocol=plain\tssl -Dcom.sun.jndi.ldap.connect.pool.debug=fine -Dcom.sun.jndi.ldap.connect.timeout=5000 -Dcom.sun.jndi.ldap.connect.pool.maxsize=300 -Dcom.sun.jndi.ldap.connect.pool.prefsize=128

 

note that i am using the gencon GC policy

 

http://publib.boulder.ibm.com/infocenter/realtime/v1r0/index.jsp?topic=%2Fcom.ibm.rt.doc.10%2Frealtime%2Frt_xoptions_gc_standard.html

 

and only using the LDAP connection pool settings.

 

using any of the other settings in the document actually cause a reduction in performance.

 

Also, starting from 12.6 SP4, you can now control how many threads you assign to the object feeder event (was hard coded to 30) so that also can improve your performance,

 

thanks

 

Itamar

[Alan_Baugher]

Thanks Itamar,

 

I have updated my deck with your input and others.

 

I have also include not only the quick and tactical "tweaks" but also some strategic views and possible re-architecture.

 

 

Cheers,

[Itamar Budin]

Any time

[Efren Yanez]

Alan,  - I´m grateful for the information you've shared 

 

Thanks

 

Efren

[Enrique_L._Torres]

Alan,

 

i have a specific question on the exposed IM and IMS tasks which can be set to DEBUG to track execution for troubleshooting, performance evaluation, etc. In the attached PDF (pg 30 on Advanced/Strategic Planning 03) you list the following:

Useful for tracking IM business logic from feed to PX rules:

• i. im.feeder = DEBUG {Must be added in Edit box}
• ii. ims.policyxpress = DEBUG
• iii. ims.tasktrack.custom = DEBUG {Must restart IME to fully capture debug at startup of IME}

At one time IM Dev published a list of all available classes with loggers enabled. Is the assumption that all IM and IMS classes have java.util.logging included, so any class can be set to DEBUG via the logging.jsp UI or updating the log4j xml?

[Alan_Baugher]

Hi Enrique,

 

That is my understanding.   If the class does not show up in the logging.jsp page as a drop down, I have directly added the class to the edit box and submitted.

I was able to then see that class reported in the J2EE logs.

 

The challenge I have found it how to ensure that "noise" is limited, to avoid overwhelming the administrator to debug.

 

One answer, was to directly update the log4j properties file with another "appender", and have ONLY select loggers go to that file.

I have one just for im.feeder, ims.policyxpress, ims.tasktrack.custom to allow a capture of the business flow.

I would like to update this process to have all the PX rules (currently in use) to create a screen LAH with a unique GUID and pass that GUID from event to event.

 

Only three (3) ways I have found to persist this data was to store it temporary

1) on the user profile (a I/O hit) or

2)  file (an I/O hit) or 

3) use the IME's Advanced Settings › Miscellaneous > User Defined Properties   {this will be available to the entire IME, but use it for one-id-and-one-use-case at a time}

 

However, since the attempt to capture business logic, the above methods do assist.

 

 

Cheers,

A.

[Itamar Budin]

Hi

 

Some more settings for WAS.

 

By default, WAS only support 50 concurrent connections for each JVM, you can change that using the following JVM properties (I have also included some other settings i have been using to fine tune connections)

 

-Dsun.net.inetaddr.ttl=0 -DdisableWSAddressCaching=true

-Dcom.ibm.websphere.webservices.http.connectionKeepAlive=true

-Dcom.ibm.websphere.webservices.http.maxConnection=1200

-Dcom.ibm.websphere.webservices.http.connectionIdleTimeout=6000

-Dcom.ibm.websphere.webservices.http.connectionPoolCleanUpTime=6000

-Dcom.ibm.websphere.webservices.http.connectionTimeout=0

[Alan_Baugher]

JBOSS Deployment Scanner  (Wildfly)    -   Small I/O performance improvement

 

The OOTB deployment of JBOSS enables a 5000 millisecond deployment scanner.   For the IMAG solutions, this process is not require.    Only upon start is the deployment scanner required.

 

To update the deployment scanner, one may edit the JBOSS_HOME\standalone\configuration\standalone*.xml

 

But it is easily updated via the JBOSS_HOME\bin\jboss-cli.bat (jboss-cli.sh) script

 

1)  JBOSS_HOME\bin\jboss-cli.bat     

2)  Type  connect   at the new prompt

3)  To view the current settings, type   /subsystem=deployment-scanner:read-resource(recursive=true)

4)   To update the interval from default of 5000 to -1, type the following:

/subsystem=deployment-scanner/scanner=default:write-attribute(name="scan-interval",value=-1)

 

 

REF:

Deployment Scanner configuration - JBoss AS 7.0 - Project Documentation Editor

8.4.8. Configure the Deployment Scanner with the Management CLI

[Sagi Gabay]

Another great tip - very useful and helpful. Thanks Alan !

 

Sagi

[Alan_Baugher]

Team,

 

Two (2) DB enhancements to review for CA IMAG solution stack.

 

How enhance the Task Persistence DB performance when running on Oracle 11gR2 DB  [Thanks to Itamar Budin]

https://communities.ca.com/docs/DOC-231169480

 

IMAG SQL Server Maintenance

https://communities.ca.com/thread/241759078

 

 

Provisioning Server Connection Improvements -  [Thanks to Itamar Budin]

 

Operation Details / Operation Details Expiration Time = Change from 96 to 4500

 

 

Operation Details / Maximum Operation Details = Change from 100 to 200000

 

 

And the usual suspect, logs.

Transaction Log / Level = change from 7 to 3

 

 

 

Regards,

A.

[Eric Laney]

The document you referenced on Transaction Persistence on Oracle [https://communities.ca.com/docs/DOC-231169480] gives me an authorization error when I try to view it.  Is that a document you can share?

[Itamar Budin]

Hi

 

please try now

 

thanks

 

Itamar

[Eric Laney]

Thanks, works great now.

[ausch02]

Great Content!

[Alan_Baugher]

Team,

 

Just a reminder; even on the vApp or standalone deployments; do not forget to add indexes to the IM Screens Tables.

See the readme under the CA Identity Suite samples / tool kit / examples.

 

NOTE:  These IM Screen Tables are built ONLY after an IME is created.   If you delete an IME, you will need to re-add these indexes upon creating the new IME.

 

 

 

 

<Paste in this section to allow easier search ability:>

 

config@vapp0001 VAPP-14.1.0 (192.168.242.146):/opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/samples/ObjectStore >
config@vapp0001 VAPP-14.1.0 (192.168.242.146):/opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/samples/ObjectStore > cat Readme.txt
The following files will add indices for Objectstore tables IM_SCREEN_LD & IM_SCREEN_FIELD_LD.

objectstore_db_oracle.sql
objectstore_db_sqlserver.sql

Please note that the tables must exist before attempting to run these files. Also, these tables are not created if no environment exists i.e. this is a fresh installation, hence the files should be run AFTER environment creation.config@vapp0001 VAPP-14.1.0 (192.168.242.146):/opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/samples/ObjectStore >
config@vapp0001 VAPP-14.1.0 (192.168.242.146):/opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/samples/ObjectStore > cat objectstore_db_oracle.sql
-- Adding indices for Objectstore tables IM_SCREEN_LD & IM_SCREEN_FIELD_LD
create index idx_IM_SCREEN_LD on IM_SCREEN_LD(REF_ID);
create index idx_IM_SCREEN_FIELD_LD on IM_SCREEN_FIELD_LD(REF_ID);
commit;
config@vapp0001 VAPP-14.1.0 (192.168.242.146):/opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/samples/ObjectStore >

 

 

 

EXAMPLE BEFORE:   (no indexes on the two IM Screen Tables)

 

 

CREATING INDEXES:

- Copy/Paste example from CA Identity Suite samples; update for your correct naming convention for these two (2) tables, e.g.    service_id.IM_SCREEN_LD   &    service_id.IM_SCREEN_FIELD_LD

 

 

 

AFTER EXAMPLE:   Indexes added

 

 

 

 

Test your startup & Run-n-Operate metrics before and after.

 

 

Cheers,

 

Alan

[Suresh Vaidyam]

Hi Alan,

The changes that are mentioned in PDF's for performance improvement in main post is this applicable to VAPP 14.3, or this are already implemented in VAPP14.3.
We are trying to get  screen  indexes done in Oracle DB.

Thanks
Suresh

[Alan Baugher]

Hi Suresh,

The IM screen indexes need to be added after an IME is created.
The official notes are still in the readme file in the IAMSuite samples folder.

If you have access to the Oracle SQL Developer UI, and the IM Database, you can paste the few lines into the UI and commit them.



------------------------------
Alan Baugher
ANA Technology Partner (anapartner.com)
------------------------------

[Alan Baugher]

If you are seeing delays with Startup and the other improvements have not address the delay between STEP4 and STEP5, you may wish to increase the bandwidth of the messaging bus.

The documentation has notes on adjustments for this via ejbs.

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-3/installing/install-on-jboss-or-wildfly/fine-tune-jboss-or-wildfly-configurations.html


The files are exposed for the 'config' userID to update on the vApp as well.

####  If you need to start quicker until performance tweaks resolve this issue ###

If the startup time is > 10 minutes, you can short cut the IMPS provisioning roles sync  (where Provisioning Role names are checked against the IM_ROLE table).      During STEP4, and after 10 minutes, stop the imps service  (imps stop ), wait 5 minutes, then restart imps service (imps start).

This will interrupt the provisioning server sync process, and allow the IME to continue to start.
- If all Provisioning Roles are created via the IME, this is low risk; as this is top-tier architecture and top-to-bottom data flow.
- If any Provisioning Role is created via the IMPS GUI, then please allow the sync to occur at least once to keep the tables in sync.

​​​

------------------------------
Alan Baugher
ANA Technology Partner (anapartner.com)
------------------------------

[Rafael Nicola Dias de Oliveira Sousa]

Hi Alan,
     How can I do that in VAPP 14.3 ? These files are read only.

[Alan Baugher]

Hi Rafael,

For the JVM arguements, there are three files that the 'config' ID has access to.  Any update here will overwrite about 90% of any predefined JVM settings.

/opt/CA/VirtualAppliance/custom/IdentityManager/jvm-args.conf
/opt/CA/VirtualAppliance/custom/IdentityGovernance/jvm-args.conf
/opt/CA/VirtualAppliance/custom/IdentityPortal/jvm-args.conf

{also a JVM custom configuration file for JCS:    /opt/CA/IdentityManager/ConnectorServer/data/jvm_options.conf }

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-suite/14-4/virtual-appliance/administering-virtual-appliance.html#concept.dita_484b93c7f06198e8b27adcc2537229358eb17777_CustomJVMArguments

For updates to the primary configuration files of  ca-standalone-full-ha.xml, you will need to use a Wildfly management ID.

The vApp has a sudo command for the 'config' user to add Wildfly application [-a switch] and/or management IDs.  [-m switch]

sudo /opt/CA/wildfly-idm/bin/add-user.sh -m -u jboss-admin -p Password01!

Then you can build your jboss-cli.sh scripts.   

IMPORTANT NOTE:   Always use "batch" mode, to avoid impacting startup of Wildfly with incorrect values.

Batch mode will rollback any changes that it can not accept.

/opt/CA/wildfly-idm/bin/jboss-cli.sh --connect --user=jboss-admin --password=Password01!  --file=im_jdbc_spy_for_tp_and_os.cli

Examples for update using jboss-cli.sh  CLI scripts (with batch mode).

https://anapartner.com/2020/04/26/advanced-oracle-jdbc-logging/

Cheers,

Alan

------------------------------
Alan Baugher
ANA Technology Partner (anapartner.com)
------------------------------

[Alan Baugher]

Note:  If you enable X11 libraries on the vApp, you may wish to use the GUI version of jboss-cli.sh  to review the ca-stanalone-full-ha.xml on the vApp; and make changes with low risk, as the tool will prevent changes that it will not accept.

/opt/CA/wildfly-idm/bin/jboss-cli.sh   --connect  --user=jboss-admin  --password=Password01!  --gui

https://anapartner.com/2021/06/21/using-x11-on-virtual-appliances/

------------------------------
Alan Baugher
ANA Technology Partner (anapartner.com)
------------------------------