I am always looking for better methods to streamline business processes, and lower effort to projects or TCO for customers. One process that intrigued me was the TDM methodology with regards to "test data". If "test data" within development or other non-production environments does NOT represent what a project team will experience in a customer production environment, then the likelihood of show-stopper issues, high-level defects/issues, and remediation efforts will occur.
While using automated test tools will preemptively identify these challenges, the question is how to resolve them to prevent impact to a project. As time goes on a for a project/program, the business logic for development and production environments will drift.
There are various tools and processes that can assist to limit the "drift" via CVS systems, or automated dev-ops processes focused on "business logic promotion" from environment to environment. There are even TDM automation tools, that address this need/market space.
For projects that do not have a TDM solution, we can still take the same concepts and apply them.
"How to export non-PII business logic/user data from production to lower environments"
- This is typically a 3 or 4 step process
- 1) Extract the business logic/user data
If possible, at this time of extraction, exclude any PII or Password or other sensitive data.
Masking of user fields (that will not impact use-case testing) is also a possibility, e.g. replace all firstname/lastname with generic data.
-2 ) "Massage" the data => Implies removing the "environmental factors", e.g. hostnames, service IDs, password hash, etc.
These "environmental factors" may be using the business rule logic or within the provisioning user store.
Do NOT leave in this data, to avoid a "development environment" communication to a production endpoint!!!
- This also avoids "service IDs" & hostname being used to prevent "password lockout" of a production service ID.
- 3) Import the "cleaned TDM" data to development or other non-production systems.
- 4) Optional, but useful to check for any business logic that may be external, e.g. custom JAR files
Use WinMerge or Beyond Compare to review for any deltas between environments.
Only allow "environmental factors", e.g. hostnames, IP addresses, service IDs, password hashes, to differ.
Please review my enclosed examples and provide feed back.
I am enclosing the following:
- a TDM process overview deck,
- a working example of using the Identity Suite vApp, Samba Network Share, and use of data extract tools
- text file of the working bash shell script for the Identity Suite vApp & a MS Windows 2012/2016 server (IAMCS/CCS server) & IM r14 tool: ImportExportUtility
Note: This script will extract the IME, the DSA and covert them to LDIF format; and use TDM approach to exclude sensitive data.
- text file of managing a TDM extract of MS AD LDS or MS Active Directory using MS tools: LDFIDE / CSVDE
Pulled out the steps for IM/IP/DIR to clarify the process
Step 1: Export the latest production IME & directory.xml (both corp & prov)
Value: Will identify/isolate the use-cases to IM tasks & business rules (PX), to lower migration effort and any risk.
Step 2. Export the LDIF of the provisioning Store
Value: Will identify managed endpoints, business logic (AT), Roles; to lower migration effort and any risk.
Please remove all sensitive data (PII), e.g. any password fields attached to the Global User record. [ See –x option to exclude attributes]
Example: Use dsa account & offline (-z) or a shutdown DSA to export. Repeat for main, co, inc. May ignore notify.
dxdumpdb -z -f $DXHOME/backup/ca-prov-srv-01-impd-main_TDM_No_Passwords_<date-stamp>.ldif -x eTPassword,eTEncryptedPassword,eTExitAuthPassword,eTSelfAdminPassword,eTPreviousPassword,eTPropagatePassword,eTIMPasswordData,eTSyncPassword,eTPropagatePassword,eTPSAgentChangePassword,eTTestPassword ca-prov-srv-01-impd-main
Step 3. Export the LDIF of the corporate userstore [May remove ALL user records except sample set, e.g. Any service IDs and 5 sample user IDs]
Value: Will identify mapping & transformation of attributes; to lower migration effort and any risk
Please remove all sensitive data (PII), e.g. any password field attached to the uid/cn record [ See –x option to exclude attributes]
Example: Use dsa account & offline (-z) or a shutdown DSA to export.
dxdumpdb -z -f $DXHOME/backup/UserStore_userstore-01_TDM_No_Pwd_or_Policies_<date-stamp>.ldif -x userPassword,IdentityPolicy,createTimestamp,modifiersName,modifyTimestamp UserStore_userstore-01
Step 4: Zip the CA Directory config folder from one server.
Value: Will contain any custom directory schema for the corporate user store.
Step 5. OPTIONAL: Zip of their IM J2EE folder from root of one or all nodes
Value: Identify &isolate any custom configurations/jar files
Additional Value: Health Check Process: Zip of other nodes, will help to quickly identify any mis-configurations or missing supporting files between nodes.
Step 6. Export the latest JSON for Identity Portal.
Step 7: Open a CA support ticket, to securely upload the large files (they will be too large via email). Use FileZilla or any other client tool that has re-start functionality to upload/download
Thanks for sharing this with the community Alan!
TDM (Test Data Management) with CA Identity Suite/vApp