Team,
If you plan to use the IMPS CLI tool of etautil, please consider using the input file switch for a performance gain.
Comparison examples:
Slowest: etautil used for 100,000 updates. Where etautil command is used on every line.
etautil.exe" -o -d im -u etaadmin -p Password01 ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000001
etautil.exe" -o -d im -u etaadmin -p Password01 ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000002
etautil.exe" -o -d im -u etaadmin -p Password01 ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000003
...
etautil.exe" -o -d im -u etaadmin -p Password01 ACTION_VERB_+_BASE_DN_+_ITEM_HERE_100000
The reason is the etautil process has a built-in error checking routine that executes to validate the input text or file.
If this error checking routine takes 1 second, then 100,000 lines would add this many seconds to processing of the job as overhead.
Observed Rate: 1 account per 4 seconds
100,000 rows x 4 (sec/id) = 400,000 seconds = 112 hours ( about 5 days)
*** ***
Faster: etautil used for 100,000 updates. Where etautil is called ONCE and uses a file for input.
Create an input file, e.g. input.txt where each line now has the content + a SEMICOLON for each line.
input.txt
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000001;
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000002;
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000003;
...
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_100000
And call this input file ONCE with etautil.
@echo on
"C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin\etautil.exe" -o -d im-u etaadmin-p Password01 -f input.txt
pause
Since etautil is only called once, it will only perform its error checking routine once for a 100 K row input file.
If any part of this file is not formatted correctly, etautil will stop and error out.
If there are incorrect attributes that don't exist in the IMPD user store, the etautil script will report this error but continue.
Observed Rate: 15-17 accounts per second
100,000 rows x (1/15 rows/sec) = 6667 seconds = 2 hours
*** ***
Even Faster: PARALLEL Processing
etautil used for 100,000 updates. Where input file is broken into two (2) sections of 1/2 size AND etautil is called ONCE for each file and uses the files for input.
Create a single input file, e.g. input.txt where each line now has the content + a SEMICOLON for each line.
input.txt
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000001;
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000002;
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_000003;
...
ACTION_VERB_+_BASE_DN_+_ITEM_HERE_100000
Then split this file into two (2) each sections: input_01.txt and input_02.txt
And call each file ONCE with etautil in two (2) different batch processes
@echo on
"C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin\etautil.exe" -o -d im-u etaadmin-p Password01 -f input_01.txt
pause
@echo on
"C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin\etautil.exe" -o -d im-u etaadmin-p Password01 -f input_02.txt
pause
The IMPS server AND IMPD userstore will be able to handle up-to 8-16 parallel scripts before a 4vCPU IMPS server is 100% constrained.
Again: Since etautil is only called once per script, it will only perform its error checking routine once for a 1/2 of the 100 K row input file.
If any part of this file is not formatted correctly, etautil will stop and error out.
If there are incorrect attributes that don't exist in the IMPD user store, the etautil script will report this error but continue.
Observed Rate: 15-17 accounts per second x 2
100,000 rows x (1/30 rows/sec) = 3334 seconds = < 1 hour
*** ***
Fastest: IAMCS Load Balancing - Avoid single thread connectors to endpoints + PARALLEL Processing
Ensure that the IAMCS "Routing Rules" allow for Round Robin Load Balancing to the CCS and JCS connector servers.
This can be validated by monitoring the CCS logs or JCS logs, when two (2) separate transactions are pushed through for a single endpoint.
Transaction 1, should show up on IMPS Server #1
Transaction 2, should show up on IMPS Server #2 (If not, then IAMCS is setup in FAILOVER model)
Observed Rate: 15-17 accounts per second x 2 x 2
100,000 rows x (1/60 rows/sec) = 1667 seconds = 28 minutes
*** ***
- Notes:
etautil Gotchas -
1) Ensure the attribute values are wrapped with single or double quotes if there are spaces or UTF characters in the values
2) Ensure when using the input file, that each line ends with a semi-colon character; except for the very last line. This lets etautil know there are addition lines to process and when to stop.
3) Use the -DYN switch if any of the content contains a dynamic endpoint, e.g. CX connectors. Not needed for standard OOTB connectors
2016-05-21 Edit: Added reference deck to support observations and testing methodology
2018-03-15 Edit: Add clarity to the IM provisioning tier "Routing Rules" that provide the load-balancing feature.
Clarity on the IM Provisioning Routing Rules for IAMCS/CCS Connectors
- Note: IAMCS (with embedded CCS or remote managed single CCS service) may only have a 1:1 relationship with a single CCS service. See note above for more information.
Cheers,
A.