Test Data Manager Release Announcement
We are excited to announce the General Availability of Test Data Manager 4.8
Release 4.8 delivers a number of powerful new features, including Horizontally Scalable Portal Masking, which helps to efficiently mask high volumes of data without affecting TDM Portal performance. This new version can be downloaded directly from the Support Site. Once logged in you will be able to select the latest version from Download Management. To find the files more quickly, search for ‘Test Data Manager’ in the Search field.
What is Horizontally Scalable Portal Masking?
In TDM Release 4.6, we introduced an end-to-end workflow to identify, tag and mask sensitive data stored in various tables across disparate but connected data sources – all of that in an intuitive and easy to use TDM Portal UI. Masking is typically based on resource- intensive operations that deal with huge volumes of data. This often leads to the presence of long-running jobs in the job queue, which can cause other jobs to stall. It also requires administrators to allocate significant hardware resources for the single Portal machine, because all these resource-intensive processes are running on the TDM Portal machine itself.
In TDM 4.8 we are introducing Horizontally Scalable Portal Masking architecture that prevents that from happening - and in fact much more. Horizontally Scalable Portal Masking is based on a network of connected masking engines that can run on different machines independent of the machine where Portal is deployed. These engines are available to take up masking jobs triggered from TDM Portal which ensures that resource-intensive tasks can run on machines other than the one running Portal. In addition to that, a big masking job could be split into smaller sub-jobs (based on schema or table level) that will be distributed between available masking engines and run in parallel to achieve horizontal scalability and increased performance. Once all sub-jobs complete their work, the logs and audit files are consolidated and available in TDM Portal to provide a smooth and transparent user experience.
New TDM Portal Homepage and Insights Dashboard
The first thing a user sees in TDM Portal after successful login is the Home Page – in TDM 4.8 we focused on re-design of this prominent and frequently visited page. First of all, TDM Portal now includes the “Getting Started” banner, for easier on-boarding of new TDM users – the "Getting Started” banner guides users through the logically connected steps, which must be completed in order to effectively use TDM Portal, within a few minutes. These are: Create new project, Define connections to data sources and Create new data model.
Besides the “Getting Started” banner, the primary function of which is to help users during on-boarding, there is also a re-designed “Quick Links” panel displayed on the TDM Portal Home Page. The “Quick Links” panel provides a convenient and easy way to trigger common TDM functions like Data Masking or Find & Reserve.
Administrators and Project administrator can togle between the Home Page view and the Insights Dashboard view, which helps to track TDM activities and utilization via a set of metrics presented as Dashboard Tiles. Using the Insights Dashboard, Administrator users can easily see patterns in the usage of TDM, and use this for internal reporting and operations tuning.
TDM 4.8 provides the following metrics: Running Jobs, Job Queue Time, Job Type Chart, Recently Created Projects, Connection Profiles and Generators and Find & Reserve statistics. All these metrics are available as Dashboard Tiles which can be re-arranged or hidden.
Find & Reserve: Show More Properties than Model Keys
The “Model Data” table in the "My Reservations” section is no longer limited to displaying model key columns only – in TDM 4.8 the “Model data” table displays the other columns from the root entity in the Find & Reserve model. This change allows testers to see all data in the reserved entity and easily reserve data for their test needs. This enhancement applies also to the reserved data exported as a CSV file.
Find & Reserve: DB2/AS400, LUW and AIX support
Find & Reserve functionality now supports more DB2 platform flavours than DB2 on zOS support introduced in TDM 4.6. TDM 4.8 adds support for DB2/AS400, DB2 LUW and DB2 AIX. This enables the use of the TDM Find & Reserve feature (vital for tester self-service scenarios) across different variants of DB2, and also empowers testers to use the right data they need for their tests.
License Management in TDM Portal
TDM Portal is now the central element from which to manage product activation, using either a license key or via Portfolio License Agreement terms. If the TDM product is not activated by license yet, TDM Portal displays the Product Activation Wizard, which helps with product activation, and includes the ability to upload the license key with a redesigned UI.
TDM Repository Upgrade via Portal
TDM’s internal repository structure is now automatically upgraded during the first start-up of TDM Portal 4.8 service. This process is completely transparent and requires no manual steps. A dedicated schema-management utility is available, to control the process of repository upgrade manually and also to retrieve a detailed log of repository upgrade progress.
TDM Usage Telemetry
TDM is able to collect usage data across its different components – including Portal, Datamaker, GT Subset and Fast Data Masker. This usage data can be exported via TDM Portal so you can access this and create various reports about TDM usage. For more information on this feature, please refer to TDM Documentation.
Active Directory & LDAP Configuration Improvements
Active Directory or LDAP connection settings are no longer limited to the mandatory setting of User or Group Container attributes – these fields are no longer defined as required. This change allows users to setup connection to Active Directory and LDAP in a more flexible way that is no longer limited by the requirement to link it with a single User or Group Container.
Data Model & PII Discovery Improvements
We have listened to the feedback we received from our customers regarding the Data Modal and PII Discovery flows. Based on that, TDM 4.8 introduces the following enhancements in the respective TDM Portal screens.
Data Discovery scan level– as well as discovering relations based on keys defined at the database level, the TDM Portal Data Discovery process can now discover relationships that are defined on the application level. Data Discovery uses heuristics-based data content and column matching, however in some cases the key-based relationships may be sufficient. TDM 4.8 now provides the ability to pick the Data Discovery scan level, to either use key relationships only (which is the default) or to perform an advanced scan based on content.
Column name based PII classifiers– the PII Discovery process in Portal now uses classifiers that can determine the nature of the data based on the column name. This, together with the already available PII Discovery based on the data content provides more accurate PII Discovery results, and also provides an option to run a quick PII Discovery based on column names only.
PII Discovery scan level– PII Discovery settings allow users to define the scan level based on the percentage of data to be scanned. However, a scan level of 10% could result in millions of rows for very large datasets, or just hundreds for smaller datasets. To be able to control the level of the scan on a more granular level, TDM 4.8 provides a way to define the absolute limit of rows to be scanned for a table. This can significantly reduce the time required to perform PII scan, while still keeping the ability to control scan level.
TDM Masking with Brightside
The TDM documentation contains a step-by-step guide to performing TDM mainframe masking operations using theBrightsideproject, which allows users to perform mainframe operations via a command-line interface available for distributed platforms like Windows, Linux or MacOS. This enables a unique method for users to easily trigger masking jobs, and also to integrate mainframe masking into industry standard CI/CD platforms like Jenkins.
For more information, you can find TDM 4.8 release notes here.