Symantec Identity Management

Expand all | Collapse all

CA IM Exchange Agent 2007/2010/2013 Performance

  • 1.  CA IM Exchange Agent 2007/2010/2013 Performance

    Posted 12-05-2013 10:27 PM

    Hello All,

     

    Edit: 2017-02-17 -  Just to update this note, and re-enforce the concept that updating the default MS Powershell quota limit from 18 to 100 connections, will decrease the load on the J2EE, IMPS, and AD servers (due to "backlog" load) and lower the duration to load 1,000 of new users via the bulk loader.      This process works for BOTH agent and agent-less connections (as they both use Powershell API) to manage MS Exchange access.   

     

    Goal:   Reduce bulk load of 1000 new users from 40 minutes to less than 5 minutes.

     

     

    ### ###

     

    While working with the CA IM Exchange Agent for 2007/2010/2012 (not the agentless version); I have noticed a few tweaks that can increase bulk load performance of "day-one/brightright" access that will create an active directory account + an exchange mailbox via IM Provisioning Roles.

     

    There are three (3) challenges that exist with the MS AD/Exchange architecture that must be addressed to improve performance.  Lucky for us, the methods to address them exists.

    The mid tier component of the CA IM stack is called the Provisioning Server, the provisioning server will create an AD account per MS standards in a rapid two (2) step process:   1) Create a non-active account; 2) Update the password and make the AD account active.

    Challenge 1:

    Depending on which AD DC was chosen for the primary endpoint for AD, there will be a replication period to sync new user account to all peer AD DC, default period is 15 minutes, but client may have AD sync policies that lower this to 5 min or so.   Regardless, there is some X minutes between replication of data.

    Challenge 2:  

    The MS Exchange architecture require that prior to creation of a mailbox, the AD account must exist AND be active.   The MS Exchange architecture does NOT query ALL DC, but only the "closest" DCs (per lan segments).   If replication of the AD account has not occur to the DCs that Exchange uses; then there will be a delay.  If the AD account does display but is not active; then there will be a delay.

    Challenge 3:

    Large bulk loads of 500-10000+ are easliy handled by the J2EE JVM + IM business logic + IM Database + IM userstore very rapidly.  

    Delays are usually seen during the addtion of IM Provisioning Roles that grant or create access on downstream endpoints (AD, UNIX, Exchange,Mainframe,DB, etc.); these delay are due to how rapid those systems can respond to large data loads.   Identifing the bottle-necks at this tier is a requirement for increasing performance for bulk loading.  

    The challenge for the MS Exchange architecture is a default thottle policy of eighteen (18) connections.   When X connections arrive on the Exchange Server; the CA IM Exchange agent will setup X number of connections to the MS Exchange mailbox server.  If X is greater than 18, then these extra connections must "wait" until a session is free.  If the wait period is greater than the default timeout of 30 seconds for the IM Exchange agent, then mailbox creation will fail.

     

    How to address these challenges:

    1) To address challenge #1 new functionality was added into the IM Provisioning Server than can be enabled so that when an account is created on one (1) DC, that DC FQDN can be sent over as a variable to be consumed by the CA IM Exchange agent.   This information is within the current IM bookshelfs.  This will let the MS Exchange "know" which DC to query for the new AD account.

    Enable the following environmental variable on the IMPS server(s):    ADS_E2K_SEND_DC=1      and bounce the im_ccs services.

     

    2) To address challenge #3; expanding the default timeouts and increase the MS Exchange throttle limits for the selected proxy service account; used to create Exchange mailboxes.

    On the MS Exchange Server(s);  Increase timeout of HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\Identity Manager\Ex2k7AgentTimeout = 600           seconds

    On IMPS servers(all): Set the environmental variable, then bounce the im_ccs service.     ADS_CONFIRM_MAILBOX = 600                seconds

     

    Exchange 20XX throttle limits.

    IM solution OOTB has uses a service account for IM on the Exchange server with 18 max concurrent login to create/modify Exchange accounts.

    Recommendation #1: Increase the throttle sessions to 100 to accommodate IM bulk feeds that are sent in batches with a batch switch of 100.

    1. Exchange Admin may create a new Throttling policy to be used by select user accounts

    Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100

    2. Exchange Admin would then apply this new throttling policy for the IM service account on the Exchange server.

    Example: Set-Mailbox "User Name" -ThrottlingPolicy MaxPowershell

     

    Justification:

    Scenario: 2000 creations from IME BLC (Bulk Loader Client)

     

    Exchange able to create user mailbox in 20 seconds. {conservative estimate}

    If Timeouts are bumped to 600 seconds but left with the default of 18 session pool:

    2000 * 20 / 18 = 40,000 seconds / 18 = 2222 seconds = < 40 minutes (Expect 5-10% failure due to timeout over 600 seconds)

    If Timeouts are bumped to 600 seconds AND throttle is increased to 100 session pool:

    2000 * 20 / 100 = 40,000 seconds / 100 = 400 seconds = < 5 minutes [Expect no failures]

     

     

    Get the performance you deserve and need.   

     

     

    Regards,

    Alan Baugher



  • 2.  RE: CA IM Exchange Agent 2007/2010/2013 Performance

    Posted 12-06-2013 07:02 PM
    alan_baugher:

    Hello All,

    While working with the CA IM Exchange Agent for 2007/2010/2012 (not the agentless version); I have noticed a few tweaks that can increase bulk load performance of "day-one/brightright" access that will create an active directory account + an exchange mailbox via IM Provisioning Roles.

     

    There are three (3) challenges that exist with the MS AD/Exchange architecture that must be addressed to improve performance.  Lucky for us, the methods to address them exists.

    The mid tier component of the CA IM stack is called the Provisioning Server, the provisioning server will create an AD account per MS standards in a rapid two (2) step process:   1) Create a non-active account; 2) Update the password and make the AD account active.

    Challenge 1:

    Depending on which AD DC was chosen for the primary endpoint for AD, there will be a replication period to sync new user account to all peer AD DC, default period is 15 minutes, but client may have AD sync policies that lower this to 5 min or so.   Regardless, there is some X minutes between replication of data.

    Challenge 2:  

    The MS Exchange architecture require that prior to creation of a mailbox, the AD account must exist AND be active.   The MS Exchange architecture does NOT query ALL DC, but only the "closest" DCs (per lan segments).   If replication of the AD account has not occur to the DCs that Exchange uses; then there will be a delay.  If the AD account does display but is not active; then there will be a delay.

    Challenge 3:

    Large bulk loads of 500-10000+ are easliy handled by the J2EE JVM + IM business logic + IM Database + IM userstore very rapidly.  

    Delays are usually seen during the addtion of IM Provisioning Roles that grant or create access on downstream endpoints (AD, UNIX, Exchange,Mainframe,DB, etc.); these delay are due to how rapid those systems can respond to large data loads.   Identifing the bottle-necks at this tier is a requirement for increasing performance for bulk loading.  

    The challenge for the MS Exchange architecture is a default thottle policy of eighteen (18) connections.   When X connections arrive on the Exchange Server; the CA IM Exchange agent will setup X number of connections to the MS Exchange mailbox server.  If X is greater than 18, then these extra connections must "wait" until a session is free.  If the wait period is greater than the default timeout of 30 seconds for the IM Exchange agent, then mailbox creation will fail.

     

    How to address these challenges:

    1) To address challenge #1&#2; new functionality was added into the IM Provisioning Server than can be enabled so that when an account is created on one (1) DC, that DC FQDN can be sent over as a variable to be consumed by the CA IM Exchange agent.   This information is within the current IM bookshelfs.  This will let the MS Exchange "know" which DC to query for the new AD account.

    Enable the following environmental variable on the IMPS server(s):    ADS_E2K_SEND_DC=1      and bounce the im_ccs services.

     

    2) To address challenge #3; expanding the default timeouts and increase the MS Exchange throttle limits for the selected proxy service account; used to create Exchange mailboxes.

    On the MS Exchange Server(s);  Increase timeout of HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\Identity Manager\Ex2k7AgentTimeout = 600           seconds

    On IMPS servers(all): Set the environmental variable, then bounce the im_ccs service.     ADS_CONFIRM_MAILBOX = 600                seconds

     

    Exchange 20XX throttle limits.

    IM solution OOTB has uses a service account for IM on the Exchange server with 18 max concurrent login to create/modify Exchange accounts.

    Recommendation #1: Increase the throttle sessions to 100 to accommodate IM bulk feeds that are sent in batches with a batch switch of 100.

    1. Exchange Admin may create a new Throttling policy to be used by select user accounts

    Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100

    2. Exchange Admin would then apply this new throttling policy for the IM service account on the Exchange server.

    Example: Set-Mailbox "User Name" -ThrottlingPolicy MaxPowershell

     

    Justification:

    Scenario: 2000 creations from IME BLC.

    Exchange able to create user mailbox in 20 seconds. {conservative estimate}

    If Timeouts are bumped to 600 seconds but left with the default of 18 session pool:

    2000 * 20 / 18 = 40,000 seconds / 18 = 2222 seconds = < 40 minutes (Expect 5-10% failure due to timeout over 600 seconds)

    If Timeouts are bumped to 600 seconds AND throttle is increased to 100 session pool:

    2000 * 20 / 100 = 40,000 seconds / 100 = 400 seconds = < 5 minutes [Expect no faliures]

     

    Get the performance you deserve and need.   

     

     

    Regards,

    Alan Baugher

     


    Great stuff!  Thanks for sharing this with the community Alan!



  • 3.  RE: CA IM Exchange Agent 2007/2010/2013 Performance

    Posted 05-28-2014 03:50 AM

    "(1) DC, that DC FQDN can be sent over as a variable to be consumed by the CA IM Exchange agent.   This information is within the current IM bookshelfs. "

    Hi Alan, This is very useful and I tried to find the information on above but did not have any results.
    if possible, can kindly share the link to above information?
     



  • 4.  RE: [CA IdentityMinder (formerly CA Identity Manager) General Discussion] R

    Posted 05-28-2014 01:06 PM
    NP. Here is a link and I pulled the content for this one valuable environmental variable.
    Note: When the instructions mention “IM_CCS computer” ; this is typically the IMPS server(s) unless you have set up additional “Connector Servers” to address internal firewall domains.



    https://support.ca.com/cadocs/0/CA%20IdentityMinder%2012%206%202-ENU/Bookshelf_Files/HTML/idocs/index.htm?toc.htm?1826645.html


    Configure Exchange 2007/2010 Preferred Domain Controller Settings
    In some managed Exchange 2007 and 2010 environments, the preferred Domain Controller used by Exchange servers is different from the Domain Controller used by the Active Directory connector. As a result, the Active Directory replication latency can introduce mailbox creation failure. To prevent mailbox creation failure, configure the Exchange Server so that it communicates directly with the Active Directory connector-preferred Domain Controller.
    To resolve the issue, set the ADS_E2K_SEND_DC system environment variable on the IM_CCS computer to 1.
    Note: By default the value of the ADS_E2K_SEND_DC system environment variable value is 0.


    Alan Baugher, CISSP, ITIL, Master Certified IT Architect
    CA Principal Architect
    Tel: 636.578.6072 / 636.336.6605
    CC Bridge # 18663766162 x6365786072
    1-631-342-3333

    From: CA Security Global User CommunityMessage Boards [mailto:CommunityAdmin@communities-mail.ca.com]
    Sent: Wednesday, May 28, 2014 2:50 AM
    To: mb_message.2252815.118319333@myca-email.ca.com
    Subject: [CA IdentityMinder (formerly CA Identity Manager) General Discussion] RE: CA IM Exchange Agent 2007/2010/2013 Performance


    "(1) DC, that DC FQDN can be sent over as a variable to be consumed by the CA IM Exchange agent. This information is within the current IM bookshelfs. "

    Hi Alan, This is very useful and I tried to find the information on above but did not have any results.
    if possible, can kindly share the link to above information?

    Posted by:timo82
    --
    CA Communities Message Boards
    118321873
    mb_message.2252815.118319333@myca-email.ca.com<mailto:mb_message.2252815.118319333@myca-email.ca.com>
    https://communities.ca.com