Revolutionizing TDM Synthetic Data Generation: Introducing Data Assistant Plus
Authored by Uttam Anand, Customer Engagement Engineer at Broadcom
Synthetic Data Generation
Synthetic data is fabricated but realistic data created using rules, algorithms, or models to mimic the structure and characteristics of real-world data, without containing any actual sensitive or Personal information (PII Data). Synthetic data generation in Test Data Management (TDM) works by creating structured, rule-based test data that mimics the shape and logic of real production data, without containing any actual user or business information. The process begins with defining the data model or schema of the application database under test, such as customer records, transactions, or policy details. This schema includes field types, constraints, relationships, and dependencies. One of the key steps in Broadcom TDM is PII scanning, which enables users to identify and tag columns containing sensitive data. This ensures that personally identifiable information (PII) is properly flagged and proper custom rules are applied to it. Based on this, TDM tools apply data generation rules, business logic, and validation constraints to produce meaningful, coherent, and usable test data.
For example, if a customer’s age must be greater than 18 to open an account, the generator enforces this rule while producing the test data. Additionally, we can help maintain data integrity across databases, e.g., account numbers in SQLS & Oracle. Broadcom TDM's synthetic data feature can simulate real-world distributions (like 80% of users being active and 20% inactive), generate edge cases (e.g., max-length names or future-dated transactions), and include negative test data for robustness.
The generated synthetic data can be injected into test environments on demand, integrated with CI/CD pipelines, and used across functional, performance, and security testing. Because the data is not derived from production systems, it eliminates the risk of data breaches and simplifies compliance with privacy regulations like GDPR or HIPAA. Overall, synthetic data generation allows teams to create clean, controlled, and secure test datasets that accelerate development, improve quality, and ensure that testing is both comprehensive and legally safe.
Introducing Data Assistant Plus in Broadcom TDM
Test Data Management has long been a critical, yet often challenging, aspect of the software development lifecycle. Generating realistic, consistent, and compliant test data can be a significant bottleneck, particularly for complex applications. We are excited to announce a new capability in Broadcom’s TDM that aims to transform this process: the Data Assistant, powered by a heuristic-based model.
The TDM Data Assistant is designed to address key pain points in traditional synthetic data generation and data management. Let's delve into the challenges it solves and the innovative features it brings to the table.
Challenge #1: The Burden of Synthetic Data Generation Rule Setup
Traditionally, setting up rules for synthetic data generation has been a laborious and time-consuming task. Imagine needing to define rules for each column across numerous tables – a process that can consume significant development time.
The TDM Data Assistant tackles this head-on. We've significantly automated and accelerated this process by adopting a heuristic model. This intelligent approach allows for a high degree of accuracy in deriving appropriate data generation rules for tables and columns. The result? The ability to create consistent synthetic data while meticulously maintaining crucial key relationships (Primary Keys and Foreign Keys), ensuring data integrity.
Challenge #2: Ensure consistent distribution of data between environments / Crafting UI-Workable Scenarios with Precision
Consider a scenario where a user interface (UI) requires specific data to function correctly. For instance, out of 1000 tables, the first table might need 1 row, table 2 might need 150 rows, and other tables require varying numbers of rows to make the UI work as expected. Manually ensuring the right amount of data in each table for a specific UI scenario is incredibly complex and prone to error.
The TDM Data Assistant, with its innovative data clone-based approach, elegantly solves this. It can intelligently select a specific table and column for cloning, then generate data for that table with appropriate rules. This capability empowers you to overcome the challenge of precisely populating tables to ensure your UI functions seamlessly.
Challenge #3 & #4: Streamlining Data Cloning and Slicing
Beyond synthetic data generation, the TDM Data Assistant extends its capabilities to data cloning and slicing. We have taken data cloning a step further by making it accessible via the TDM Portal, streamlining the process significantly. Data slicing refers to the process of creating smaller, more manageable subsets of production data for testing purposes. Working with a smaller subset of data, rather than the entire production database, significantly reduces storage requirements and the time needed to refresh data for testing.
Furthermore, using the Clone-based approach, data subsetting can also be performed directly from the TDM Portal. This addresses a traditional challenge: running a subset generator from the portal can be incredibly expensive and time-consuming, especially when dealing with scenarios involving 15-20 levels of joins. The TDM Data Assistant mitigates this by offering a more efficient and less resource-intensive method.
Introducing CRAWLER: Navigating Complex Data Models
To further enhance data management for highly intricate systems, we've integrated CRAWLER into the Data Assistant. While not entirely new to TDM (it's also present in GTSubset and Javellen), its inclusion here is pivotal.
CRAWLER is designed to excel in working with complex data models, particularly those with 8-12-15 levels of joins. It helps in understanding and extracting data from these deeply intertwined structures. Importantly, using the Data Assistant with CRAWLER allows these operations to be performed conveniently via the TDM Portal.
Data Assistant Templates: Simplify and Standardize
The TDM Data Assistant leverages templates to further simplify and standardize data generation and management. We offer two primary types:
-
Datatype-Based Template: This template considers the underlying data types, providing a foundation for consistent data generation.
-
Clone: For creating copies of existing data.
-
CloneEncrypt: For secure cloning with encryption.
-
CRAWLER: For navigating complex data models.
-
Tag-Based Template: This allows for more flexible and business-oriented tagging of data elements.
It's important to differentiate between Clone and CRAWLER. While Clone focuses on replicating data, CRAWLER is specifically employed when dealing with highly complex data models that require deep traversal and understanding of relationships.
Conclusion
The TDM Data Assistant marks a significant leap forward in test data management. By leveraging a heuristic-based model, offering intelligent cloning and subsetting capabilities, and integrating the powerful CRAWLER functionality, it empowers teams to generate high-quality, relevant test data with unprecedented efficiency and accuracy. This translates to faster development cycles, more robust testing, and ultimately, higher quality software.
Reference:
https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-11/provisioning-test-data/generate-synthetic-test-data.html
https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-11/provisioning-test-data/data-assistant-templates.html