Hello All,
Edit: 2017-02-17 - Just to update this note, and re-enforce the concept that updating the default MS Powershell quota limit from 18 to 100 connections, will decrease the load on the J2EE, IMPS, and AD servers (due to "backlog" load) and lower the duration to load 1,000 of new users via the bulk loader. This process works for BOTH agent and agent-less connections (as they both use Powershell API) to manage MS Exchange access.
Goal: Reduce bulk load of 1000 new users from 40 minutes to less than 5 minutes.
### ###
While working with the CA IM Exchange Agent for 2007/2010/2012 (not the agentless version); I have noticed a few tweaks that can increase bulk load performance of "day-one/brightright" access that will create an active directory account + an exchange mailbox via IM Provisioning Roles.
There are three (3) challenges that exist with the MS AD/Exchange architecture that must be addressed to improve performance. Lucky for us, the methods to address them exists.
The mid tier component of the CA IM stack is called the Provisioning Server, the provisioning server will create an AD account per MS standards in a rapid two (2) step process: 1) Create a non-active account; 2) Update the password and make the AD account active.
Challenge 1:
Depending on which AD DC was chosen for the primary endpoint for AD, there will be a replication period to sync new user account to all peer AD DC, default period is 15 minutes, but client may have AD sync policies that lower this to 5 min or so. Regardless, there is some X minutes between replication of data.
Challenge 2:
The MS Exchange architecture require that prior to creation of a mailbox, the AD account must exist AND be active. The MS Exchange architecture does NOT query ALL DC, but only the "closest" DCs (per lan segments). If replication of the AD account has not occur to the DCs that Exchange uses; then there will be a delay. If the AD account does display but is not active; then there will be a delay.
Challenge 3:
Large bulk loads of 500-10000+ are easliy handled by the J2EE JVM + IM business logic + IM Database + IM userstore very rapidly.
Delays are usually seen during the addtion of IM Provisioning Roles that grant or create access on downstream endpoints (AD, UNIX, Exchange,Mainframe,DB, etc.); these delay are due to how rapid those systems can respond to large data loads. Identifing the bottle-necks at this tier is a requirement for increasing performance for bulk loading.
The challenge for the MS Exchange architecture is a default thottle policy of eighteen (18) connections. When X connections arrive on the Exchange Server; the CA IM Exchange agent will setup X number of connections to the MS Exchange mailbox server. If X is greater than 18, then these extra connections must "wait" until a session is free. If the wait period is greater than the default timeout of 30 seconds for the IM Exchange agent, then mailbox creation will fail.
How to address these challenges:
1) To address challenge #1 new functionality was added into the IM Provisioning Server than can be enabled so that when an account is created on one (1) DC, that DC FQDN can be sent over as a variable to be consumed by the CA IM Exchange agent. This information is within the current IM bookshelfs. This will let the MS Exchange "know" which DC to query for the new AD account.
Enable the following environmental variable on the IMPS server(s): ADS_E2K_SEND_DC=1 and bounce the im_ccs services.
2) To address challenge #3; expanding the default timeouts and increase the MS Exchange throttle limits for the selected proxy service account; used to create Exchange mailboxes.
On the MS Exchange Server(s); Increase timeout of HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\Identity Manager\Ex2k7AgentTimeout = 600 seconds
On IMPS servers(all): Set the environmental variable, then bounce the im_ccs service. ADS_CONFIRM_MAILBOX = 600 seconds
Exchange 20XX throttle limits.
IM solution OOTB has uses a service account for IM on the Exchange server with 18 max concurrent login to create/modify Exchange accounts.
Recommendation #1: Increase the throttle sessions to 100 to accommodate IM bulk feeds that are sent in batches with a batch switch of 100.
1. Exchange Admin may create a new Throttling policy to be used by select user accounts
Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100
2. Exchange Admin would then apply this new throttling policy for the IM service account on the Exchange server.
Example: Set-Mailbox "User Name" -ThrottlingPolicy MaxPowershell
Justification:
Scenario: 2000 creations from IME BLC (Bulk Loader Client)
Exchange able to create user mailbox in 20 seconds. {conservative estimate}
If Timeouts are bumped to 600 seconds but left with the default of 18 session pool:
2000 * 20 / 18 = 40,000 seconds / 18 = 2222 seconds = < 40 minutes (Expect 5-10% failure due to timeout over 600 seconds)
If Timeouts are bumped to 600 seconds AND throttle is increased to 100 session pool:
2000 * 20 / 100 = 40,000 seconds / 100 = 400 seconds = < 5 minutes [Expect no failures]
Get the performance you deserve and need.
Regards,
Alan Baugher