Layer 7 Access Management

Expand all | Collapse all

Monitor the Identity Suite IM Notify Queue

  • 1.  Monitor the Identity Suite IM Notify Queue

    Posted 02-28-2018 01:33 PM

    Team,

     

    During testing of a large explore operation of endpoints/userstore with millions of records, if the IM Callback feature is enabled, we will see "audit" events sent to the IMPD Notify DSA (part of the IM Callback process).

     

    If we wish to monitor progress, we can use various tools, e.g. ldap client tools or the IM VST.

     

    One nice single command line is use of CA Directory's dxsearch binary (a "robust" version of "ldapsearch" type), and the switch to monitor a count.

     

     watch -n 2 "dxsearch -h `hostname` -p 20398 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=notify,dc=etadb dxEntryCount | grep dxEntryCount"

     

    This process will show the total count under the notify DSA, and with a time-stamp to allow an accurate metric on how quickly the IME tier is processing these "audit" events, to be stored into the IM TP table, so they may be viewed with the IM VST (view submitted task) in the IM UI.

     

     

     

     

    Ref:  Counting Entries - CA Directory - 12.0.14 - CA Technologies Documentation 

     

     

    If the default log level of "INFO" is not require for audit events, this IM Callback process may be disabled, or reduced down to "ERROR" messages; to speed up the processing.

     

     

    Restart the im_ps  service, to use this new log level.

     

    Entries that are already in the notify DSA will still be processed.



  • 2.  Re: Monitor the Identity Suite IM Notify Queue

    Posted 03-01-2018 11:39 AM

    Hey Alan ... just to be clear ... the IMPD notify DSA maps record-for-record to the inbound synch events in the VST if we leave the Log Level set to INFO, right? What are the deeper implications for the IM Callback process if we change the Log Level setting to ERROR?

     

    ... follow up ... Configure Synchronization in the Provisioning Manager

     

    I had to be clear, refreshing my grasp of the Basic 101 info on Inbound Synchronization, on what Alan meant by changing the Log Level to improve performance on the IMPS service. For the clueless (uninformed or forgetful) the Log Level affects records written to the  PSHOME\logs\etanotify<date>.log file BUT does not affect the records written to the IMPD Notify DSA.

     

    Mea culpa ... 



  • 3.  Re: Monitor the Identity Suite IM Notify Queue

    Posted 03-01-2018 11:54 AM

    Hi Enrique,

     

    That is a good question.

     

    I may need to do additional testing to see.

     

     

    If I focus on the non-base64 encoded objects in the notify queue, I can query on the string "eTNotifyProv" to see two (2) objects that let me know the operation and the object itself that is being audited.

     

    dxsearch -h `hostname` -p 20398 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=notify,dc=etadb | grep eTNotifyProv

     

    Even with Error Mode checkbox, for the Explore operations, I do see the info messages still being populated in the Notify Queue, and thereby processed by the IME.

     

     

     

     

     

    Will review additional processes to lower the impact to the J2EE CPU utilization.

    - Testing with 600K identities.

     

     

     

     

    Testing Lab:

     

     

    SIDE NOTE:  The IAMCS/CCS components do NOT need to be deployed to a MS AD Domain Controller.  

    -  Production/non-prod environments, would designated another server for these components.

    -  My goals was to use a minimal server footprint for my lab/sandbox environment.

     

     

    Scripts Used:

     

    dsadd ou "ou=Office_001,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

    START FOR /L %%i in (1,1,200000) DO dsadd user "cn=aa Test User%%i,ou=Office_001,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -q -samid aatestuser%%i -upn aatestuser%%i@exchange2012.lab -fn aaTest -ln aaUser%%i -display "aaTest User%%i" -email aatestuser%%i@exchange2012.lab -pwd P@ssw0rd -disabled no -pwdneverexpires no

     

    dsadd ou "ou=Office_002,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

    START FOR /L %%i in (1,1,200000) DO dsadd user "cn=BB Test User%%i,ou=Office_002,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -q -samid bbtestuser%%i -upn bbtestuser%%i@exchange2012.lab -fn BBTest -ln BBUser%%i -display "BBTest User%%i" -email bbtestuser%%i@exchange2012.lab -pwd P@ssw0rd -disabled no -pwdneverexpires no

     

    dsadd ou "ou=Office_003,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

    START FOR /L %%i in (1,1,200000) DO dsadd user "cn=cc Test User%%i,ou=Office_003,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -q -samid cctestuser%%i -upn cctestuser%%i@exchange2012.lab -fn ccTest -ln ccUser%%i -display "ccTest User%%i" -email cctestuser%%i@exchange2012.lab -pwd P@ssw0rd -disabled no -pwdneverexpires no

     

    dsadd ou "ou=Groups,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab"

    START FOR /L %%i in (1,1,10000) DO dsadd group "cn=GlobalGroup%%i,ou=Groups,ou=CompanyABC_Users_OU,dc=exchange2012,dc=lab" -desc "GlobalGroup%%i"

     

    Here we can see that the AD DC is "quiet" with regards to CPU hit during the query operation.

    - The MS AD lsass.exe process, that access the AD domain userstore, requires about 3.5GB RAM for 600K identities

    - The IM Connector Tier, via the IAMCS (jcs) and the CCS (ccs) services, require minimal memory to manage the AD domain.

    Using MS Sysinternals tool, Process Explorer, to monitor the MS Window running processes.

    - Note:  Fantastic tool by Mark Russinovich & his team at MS.

    Process Monitor - Windows Sysinternals | Microsoft Docs 

     

     

     

     

    -Alan



  • 4.  Re: Monitor the Identity Suite IM Notify Queue

    Posted 03-01-2018 11:59 AM

    Thanks for the additional detail and test plan. I updated my initial reply with some basic level reading on Inbound Sync which you explained above.



  • 5.  Re: Monitor the Identity Suite IM Notify Queue

    Posted 03-01-2018 06:57 PM

    Hi Enrique,

     

    To clarify where the new functionality of "Inbound Sync Configurations" reside, I created exports of the J2EE tier and the IMPD DSAs.   

    I then executed the changes with the new IM User Console/Settings/Provisioning Configurations/Inbound Notification Configuration

     

     

    Configure Inbound Notifications - CA Identity Manager - 14.1 - CA Technologies Documentation 

     

     

    I was able to discover that these settings are stored on the IMPD CO DSA.

    - These settings are stored as an XML package in the attribute:  eTConfigPayload

    - Under the IME configuration for the IMPS tier.

     

    eTConfigName=BLS Connectivity Configuration,eTConfigContainerName=Configur
    ation,eTNamespaceName=CommonObjects,dc=im,dc=etadb

     

    The BLS Connectivity setting is aka IMPS "Identity Manager Setup" Setting.

      - However, the eTConfigPayload attribute is NOT fully viewable of all attributes, as set from the IM User Console.

     

     

    Using an LDIFDELTA of before/after, I am able to view the base64 of this attribute:  eTConfigPayload

     

     

    Using the LINUX command, base64, we can store & then view this attribute:

     

    [dsa@vapp0001 ~/backup]$ base64 -d a.ldif

    <BLSConfigItem><URLList><URL>http://192.168.242.146:8080/iam/im/ETACALLBACK/?env=identityEnv</URL></URLList><Misc><LogLevel>2</LogLevel><SuspendSending>No</SuspendSending><Password>{3DES}6dFlqaWCFp36FxWzfpLb9NKSJRisEsPKInbtIdpuzdIYZY+Hy4w9XNSbQE2isoDeS6VEU1cJI6kxj6u8DHsu5MGMKOiArA/PHBGL96ftoVh5ox3FN73K7YJn1FHe6ttDyIe4jYGdVbGtjmWe57Rh8HYAys1cp5xb5csSxrdAByGnzwGNlCeZJzlWX5aeP1As</Password></Misc><BLSEnvironments><BLSEnvironment><BLSEnvironmentURL>http://192.168.242.146:8080/iam/im/ETACALLBACK/?env=identityEnv</BLSEnvironmentURL><BLSEvents><BLSEvent>Add_Account</BLSEvent><BLSEvent>Delete_Provisioning_Object</BLSEvent><BLSEvent>Add_Global_User</BLSEvent><BLSEvent>Modify_Endpoint</BLSEvent><BLSEvent>Suspend_Account</BLSEvent><BLSEvent>Modify_Account</BLSEvent><BLSEvent>Delete_Endpoint_Type</BLSEvent><BLSEvent>Rename_Role</BLSEvent><BLSEvent>Delete_Provisioning_Role</BLSEvent><BLSEvent>Add_Explore_Container</BLSEvent><BLSEvent>Suspend_Global_User</BLSEvent><BLSEvent>Delete_Explore_Container</BLSEvent><BLSEvent>Modify_Explore_Container</BLSEvent><BLSEvent>Modify_Account_Template</BLSEvent><BLSEvent>Add_Explore_Object</BLSEvent><BLSEvent>Move_Provisioning_Object</BLSEvent><BLSEvent>Add_Provisioning_Role</BLSEvent><BLSEvent>Add_Endpoint</BLSEvent><BLSEvent>Delete_Account_Template</BLSEvent><BLSEvent>Rename_Provisioning_Object</BLSEvent><BLSEvent>Delete_Global_User</BLSEvent><BLSEvent>Delete_EndPoint</BLSEvent><BLSEvent>Modify_Global_User</BLSEvent><BLSEvent>Modify_Explore_Object</BLSEvent><BLSEvent>Modify_Provisioning_Role</BLSEvent><BLSEvent>Rename_Global_User</BLSEvent><BLSEvent>Unlock_Global_User</BLSEvent><BLSEvent>Modify_Endpoint_Type</BLSEvent><BLSEvent>Add_Provisioning_Object</BLSEvent><BLSEvent>Modify_Provisioning_Object</BLSEvent><BLSEvent>Unlock_Account</BLSEvent><BLSEvent>Resume_Global_User</BLSEvent><BLSEvent>Add_Endpoint_Container</BLSEvent><BLSEvent>Modify_Account_Password</BLSEvent><BLSEvent>Delete_Explore_Object</BLSEvent><BLSEvent>Modify_Global_User_Password</BLSEvent><BLSEvent>Delete_Account</BLSEvent><BLSEvent>Add_Account_Template</BLSEvent><BLSEvent>Delete_EndPoint_Container</BLSEvent><BLSEvent>Add_Endpoint_Type</BLSEvent><BLSEvent>Resume_Account</BLSEvent></BLSEvents></BLSEnvironment></BLSEnvironments></BLSConfigItem>

     

     

     

    We can see that by having these values near the IM Provisioning Tier, the notify DSA will now be restricted of selected events, where the XML tag of  <BLSEvent> is the "blocking" description of the provisioning event.

     

    The only way to modify this attribute, will be via the IM User Console.

    - or using an LDAP client, e.g. Jxplorer.

     

     

    To avoid the unnecessary audit events for EXPLORE operations, we will select these six "explore" events:

     

    <BLSEvent>Add_Explore_Container</BLSEvent>

    <BLSEvent>Delete_Explore_Container</BLSEvent>

    <BLSEvent>Modify_Explore_Container</BLSEvent>

    <BLSEvent>Add_Explore_Object</BLSEvent>

    <BLSEvent>Modify_Explore_Object</BLSEvent>

    <BLSEvent>Delete_Explore_Object</BLSEvent>

     

     

    We can see these names of the events, are what were captured with the command:

     

    dxsearch -h `hostname` -p 20398 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=notify,dc=etadb | grep eTNotifyProv

     

     

    NOTE:   It is important to restart the Provisioning Server service, to allow it to be aware of the new Inbound Sync Configurations.

     - If not, then events will still be processed, and can be validated with the command above or the below example:

     

    watch -n 2 "dxsearch -h `hostname` -p 20398 -c -x -D ainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b dc=ntadb | grep eTNotifyProv | wc -l"

     

     

    Cheers,

     

    A.

     

    Edit 3/3/2018.

     

    Curious, but the attribute, eTConfigPayload, does not appear in the tool SoftTerra LDAP browser; and I am seeing Explore events.

    -  Will research.

     

     

     

     

    However, the command Line to view this base64 entry directly, does show the value set to ignore explore events.

     

    dxsearch -LLL -h `hostname` -p 20396 -c -x -D "eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb" -w "Password01" -b "eTConfigName=BLS Connectivity Configuration,eTConfigContainerName=Configuration,eTNamespaceName=CommonObjects,dc=im,dc=etadb" eTConfigPayload | perl -p00e 's/\r?\n //g' | awk '{print $2}' | grep -v "eTConfigName=BLS" | base64 -d

     

     

    Resolution:  Bounce all services to ensure no cache entries.

     

    Additional check that notify is not being flood with explore operation.   View the IMPS/log/etatrans*.logs.

     

    Before size and after size (growth), with roll-over at 100 MB.

     

     

     

    Example if the EXPLORE is being "allowed" to be captured and not ignored.

    -  Note string:   "Add_Explore_Object"



  • 6.  Re: Monitor the Identity Suite IM Notify Queue

    Posted 04-05-2018 10:37 AM

    Team,

     

    Amit Sinha sinam09 provide this query of the etanotify*.logs to help identify the # of transactions by action type.

     

    $ grep "Event:" etanotify*.log | cut -d" " -f5 | sort | uniq -c | sort -rn

     

     

    That will provide a sorted table such as shown below:

     

    S.no

    Task

    Count

    1

    Modify_Explore_Object

    41022

    2

    Add_Explore_Object

    331

    3

    End_Detail

    58

    4

    Modify_Global_User

    29

    5

    Modify_Account

    29

    6

    Add_Account

    20

    7

    Explore_Correlate

    16

    8

    Provision_Account

    5

    9

    Delete_Explore_Object

    4

    10

    Suspend_Account

    3

    11

    Resume_Account

    3

    12

    Modify_Account_Password

    2

    13

    Correlate_Account_To_Global_User

    2

    14

    Modify_Configuration_Object

    1

     

     

    We can now see the value of masking the standard queries of Explore, especially the event "Modify_Explore_Object"

    - These queries are still logged at the provisioning tier, so are not required to be duplicated at the top tier within the IME & IM database.

    - This will streamline the IM notify queue from being overwhelmed by hourly/daily E&C operations.