Monitor the Identity Suite IM Notify Queue

[Alan_Baugher]

Team,

 

During testing of a large explore operation of endpoints/userstore with millions of records, if the IM Callback feature is enabled, we will see "audit" events sent to the IMPD Notify DSA (part of the IM Callback process).

 

If we wish to monitor progress, we can use various tools, e.g. ldap client tools or the IM VST.

 

One nice single command line is use of CA Directory's dxsearch binary (a "robust" version of "ldapsearch" type), and the switch to monitor a count.

 

 watch -n 2 "dxsearch -h `hostname` -p 20398 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=notify,dc=etadb dxEntryCount | grep dxEntryCount"

 

This process will show the total count under the notify DSA, and with a time-stamp to allow an accurate metric on how quickly the IME tier is processing these "audit" events, to be stored into the IM TP table, so they may be viewed with the IM VST (view submitted task) in the IM UI.

 

 

 

 

Ref:  Counting Entries - CA Directory - 12.0.14 - CA Technologies Documentation 

 

 

If the default log level of "INFO" is not require for audit events, this IM Callback process may be disabled, or reduced down to "ERROR" messages; to speed up the processing.

 

 

Restart the im_ps  service, to use this new log level.

 

Entries that are already in the notify DSA will still be processed.

[Legacy User]

Hey Alan ... just to be clear ... the IMPD notify DSA maps record-for-record to the inbound synch events in the VST if we leave the Log Level set to INFO, right? What are the deeper implications for the IM Callback process if we change the Log Level setting to ERROR?

 

... follow up ... Configure Synchronization in the Provisioning Manager

 

I had to be clear, refreshing my grasp of the Basic 101 info on Inbound Synchronization, on what Alan meant by changing the Log Level to improve performance on the IMPS service. For the clueless (uninformed or forgetful) the Log Level affects records written to the  PSHOME\logs\etanotify<date>.log file BUT does not affect the records written to the IMPD Notify DSA.

 

Mea culpa ... 

[Alan_Baugher]

Hi Enrique,

 

That is a good question.

 

I may need to do additional testing to see.

 

 

If I focus on the non-base64 encoded objects in the notify queue, I can query on the string "eTNotifyProv" to see two (2) objects that let me know the operation and the object itself that is being audited.

 

dxsearch -h `hostname` -p 20398 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=notify,dc=etadb | grep eTNotifyProv

 

Even with Error Mode checkbox, for the Explore operations, I do see the info messages still being populated in the Notify Queue, and thereby processed by the IME.

 

 

 

 

 

Will review additional processes to lower the impact to the J2EE CPU utilization.

- Testing with 600K identities.

 

 

 

 

Testing Lab:

 

 

SIDE NOTE:  The IAMCS/CCS components do NOT need to be deployed to a MS AD Domain Controller.  

-  Production/non-prod environments, would designated another server for these components.

-  My goals was to use a minimal server footprint for my lab/sandbox environment.

 

 

Scripts Used:

 

dsadd ou "ou=Office_001,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

START FOR /L %%i in (1,1,200000) DO dsadd user "cn=aa Test User%%i,ou=Office_001,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -q -samid aatestuser%%i -upn [e-mail] -fn aaTest -ln aaUser%%i -display "aaTest User%%i" -email [e-mail] -pwd P@ssw0rd -disabled no -pwdneverexpires no

 

dsadd ou "ou=Office_002,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

START FOR /L %%i in (1,1,200000) DO dsadd user "cn=BB Test User%%i,ou=Office_002,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -q -samid bbtestuser%%i -upn [e-mail] -fn BBTest -ln BBUser%%i -display "BBTest User%%i" -email [e-mail] -pwd P@ssw0rd -disabled no -pwdneverexpires no

 

dsadd ou "ou=Office_003,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

START FOR /L %%i in (1,1,200000) DO dsadd user "cn=cc Test User%%i,ou=Office_003,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -q -samid cctestuser%%i -upn [e-mail] -fn ccTest -ln ccUser%%i -display "ccTest User%%i" -email [e-mail] -pwd P@ssw0rd -disabled no -pwdneverexpires no

 

dsadd ou "ou=Groups,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

START FOR /L %%i in (1,1,10000) DO dsadd group "cn=GlobalGroup%%i,ou=Groups,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -desc "GlobalGroup%%i"

 

Here we can see that the AD DC is "quiet" with regards to CPU hit during the query operation.

- The MS AD lsass.exe process, that access the AD domain userstore, requires about 3.5GB RAM for 600K identities

- The IM Connector Tier, via the IAMCS (jcs) and the CCS (ccs) services, require minimal memory to manage the AD domain.

Using MS Sysinternals tool, Process Explorer, to monitor the MS Window running processes.

- Note:  Fantastic tool by Mark Russinovich & his team at MS.

Process Monitor - Windows Sysinternals | Microsoft Docs 

 

 

 

 

-Alan

[Legacy User]

Thanks for the additional detail and test plan. I updated my initial reply with some basic level reading on Inbound Sync which you explained above.

[Alan_Baugher]

Hi Enrique,

 

To clarify where the new functionality of "Inbound Sync Configurations" reside, I created exports of the J2EE tier and the IMPD DSAs.   

I then executed the changes with the new IM User Console/Settings/Provisioning Configurations/Inbound Notification Configuration

 

 

Configure Inbound Notifications - CA Identity Manager - 14.1 - CA Technologies Documentation 

 

 

I was able to discover that these settings are stored on the IMPD CO DSA.

- These settings are stored as an XML package in the attribute:  eTConfigPayload

- Under the IME configuration for the IMPS tier.

 

eTConfigName=BLS Connectivity Configuration,eTConfigContainerName=Configur
ation,eTNamespaceName=CommonObjects,dc=im,dc=etadb

 

The BLS Connectivity setting is aka IMPS "Identity Manager Setup" Setting.

  - However, the eTConfigPayload attribute is NOT fully viewable of all attributes, as set from the IM User Console.

 

 

Using an LDIFDELTA of before/after, I am able to view the base64 of this attribute:  eTConfigPayload

 

 

Using the LINUX command, base64, we can store & then view this attribute:

 

[dsa@vapp0001 ~/backup]$ base64 -d a.ldif

<BLSConfigItem><URLList><URL>http://192.168.242.146:8080/iam/im/ETACALLBACK/?env=identityEnv</URL></URLList><Misc><LogLevel>2</LogLevel><SuspendSending>No</SuspendSending><Password>{3DES}6dFlqaWCFp36FxWzfpLb9NKSJRisEsPKInbtIdpuzdIYZY+Hy4w9XNSbQE2isoDeS6VEU1cJI6kxj6u8DHsu5MGMKOiArA/PHBGL96ftoVh5ox3FN73K7YJn1FHe6ttDyIe4jYGdVbGtjmWe57Rh8HYAys1cp5xb5csSxrdAByGnzwGNlCeZJzlWX5aeP1As</Password></Misc><BLSEnvironments><BLSEnvironment><BLSEnvironmentURL>http://192.168.242.146:8080/iam/im/ETACALLBACK/?env=identityEnv</BLSEnvironmentURL><BLSEvents><BLSEvent>Add_Account</BLSEvent><BLSEvent>Delete_Provisioning_Object</BLSEvent><BLSEvent>Add_Global_User</BLSEvent><BLSEvent>Modify_Endpoint</BLSEvent><BLSEvent>Suspend_Account</BLSEvent><BLSEvent>Modify_Account</BLSEvent><BLSEvent>Delete_Endpoint_Type</BLSEvent><BLSEvent>Rename_Role</BLSEvent><BLSEvent>Delete_Provisioning_Role</BLSEvent><BLSEvent>Add_Explore_Container</BLSEvent><BLSEvent>Suspend_Global_User</BLSEvent><BLSEvent>Delete_Explore_Container</BLSEvent><BLSEvent>Modify_Explore_Container</BLSEvent><BLSEvent>Modify_Account_Template</BLSEvent><BLSEvent>Add_Explore_Object</BLSEvent><BLSEvent>Move_Provisioning_Object</BLSEvent><BLSEvent>Add_Provisioning_Role</BLSEvent><BLSEvent>Add_Endpoint</BLSEvent><BLSEvent>Delete_Account_Template</BLSEvent><BLSEvent>Rename_Provisioning_Object</BLSEvent><BLSEvent>Delete_Global_User</BLSEvent><BLSEvent>Delete_EndPoint</BLSEvent><BLSEvent>Modify_Global_User</BLSEvent><BLSEvent>Modify_Explore_Object</BLSEvent><BLSEvent>Modify_Provisioning_Role</BLSEvent><BLSEvent>Rename_Global_User</BLSEvent><BLSEvent>Unlock_Global_User</BLSEvent><BLSEvent>Modify_Endpoint_Type</BLSEvent><BLSEvent>Add_Provisioning_Object</BLSEvent><BLSEvent>Modify_Provisioning_Object</BLSEvent><BLSEvent>Unlock_Account</BLSEvent><BLSEvent>Resume_Global_User</BLSEvent><BLSEvent>Add_Endpoint_Container</BLSEvent><BLSEvent>Modify_Account_Password</BLSEvent><BLSEvent>Delete_Explore_Object</BLSEvent><BLSEvent>Modify_Global_User_Password</BLSEvent><BLSEvent>Delete_Account</BLSEvent><BLSEvent>Add_Account_Template</BLSEvent><BLSEvent>Delete_EndPoint_Container</BLSEvent><BLSEvent>Add_Endpoint_Type</BLSEvent><BLSEvent>Resume_Account</BLSEvent></BLSEvents></BLSEnvironment></BLSEnvironments></BLSConfigItem>

 

 

 

We can see that by having these values near the IM Provisioning Tier, the notify DSA will now be restricted of selected events, where the XML tag of  <BLSEvent> is the "blocking" description of the provisioning event.

 

The only way to modify this attribute, will be via the IM User Console.

- or using an LDAP client, e.g. Jxplorer.

 

 

To avoid the unnecessary audit events for EXPLORE operations, we will select these six "explore" events:

 

<BLSEvent>Add_Explore_Container</BLSEvent>

<BLSEvent>Delete_Explore_Container</BLSEvent>

<BLSEvent>Modify_Explore_Container</BLSEvent>

<BLSEvent>Add_Explore_Object</BLSEvent>

<BLSEvent>Modify_Explore_Object</BLSEvent>

<BLSEvent>Delete_Explore_Object</BLSEvent>

 

 

We can see these names of the events, are what were captured with the command:

 

dxsearch -h `hostname` -p 20398 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=notify,dc=etadb | grep eTNotifyProv

 

 

NOTE:   It is important to restart the Provisioning Server service, to allow it to be aware of the new Inbound Sync Configurations.

 - If not, then events will still be processed, and can be validated with the command above or the below example:

 

watch -n 2 "dxsearch -h `hostname` -p 20398 -c -x -D ainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=ntadb | grep eTNotifyProv | wc -l"

 

 

Cheers,

 

A.

 

Edit 3/3/2018.

 

Curious, but the attribute, eTConfigPayload, does not appear in the tool SoftTerra LDAP browser; and I am seeing Explore events.

-  Will research.

 

 

 

 

However, the command Line to view this base64 entry directly, does show the value set to ignore explore events.

 

dxsearch -LLL -h `hostname` -p 20396 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b "eTConfigName=BLS Connectivity Configuration,eTConfigContainerName=Configuration,eTNamespaceName=CommonObjects,dc=im,dc=etadb" eTConfigPayload | perl -p00e 's/\r?\n //g' | awk '{print $2}' | grep -v "eTConfigName=BLS" | base64 -d

 

 

Resolution:  Bounce all services to ensure no cache entries.

 

Additional check that notify is not being flood with explore operation.   View the IMPS/log/etatrans*.logs.

 

Before size and after size (growth), with roll-over at 100 MB.

 

 

 

Example if the EXPLORE is being "allowed" to be captured and not ignored.

-  Note string:   "Add_Explore_Object"

[Alan_Baugher]

Team,

 

Amit Sinha sinam09 provide this query of the etanotify*.logs to help identify the # of transactions by action type.

 

$ grep "Event:" etanotify*.log | cut -d" " -f5 | sort | uniq -c | sort -rn

 

 

That will provide a sorted table such as shown below:

 

S.no	Task	Count
1	Modify_Explore_Object	41022
2	Add_Explore_Object	331
3	End_Detail	58
4	Modify_Global_User	29
5	Modify_Account	29
6	Add_Account	20
7	Explore_Correlate	16
8	Provision_Account	5
9	Delete_Explore_Object	4
10	Suspend_Account	3
11	Resume_Account	3
12	Modify_Account_Password	2
13	Correlate_Account_To_Global_User	2
14	Modify_Configuration_Object	1

 

 

We can now see the value of masking the standard queries of Explore, especially the event "Modify_Explore_Object"

- These queries are still logged at the provisioning tier, so are not required to be duplicated at the top tier within the IME & IM database.

- This will streamline the IM notify queue from being overwhelmed by hourly/daily E&C operations.