Blog by Shamim Ahmed, CTO for DevOps Solutions, Broadcom
Model-based testing (or MBT) has recently emerged as a popular approach to transforming and optimizing application testing. MBT offers significant benefits such as improvement of test coverage, reduction of testing effort and improvement of quality.
MBT includes a rich variety of techniques and capabilities that may appear overwhelming to initial adopters. My customers often ask me how to get started with the basics of MBT and how to gradually improve their maturity and capabilities.
I am not a big fan of “maturity models”, I like “capability models” instead. This capability model is an attempt to provide some guidance on how to implement the different capabilities of MBT in a progressive manner and accrue corresponding benefits (described by specific metrics). This article is not however meant to provide an in-depth treatment on the different MBT capabilities – I will provide external references to such techniques for further exploration.
In this blog, I will use the capabilities supported by Agile Requirements Designer by Broadcom (and related products) as a reference for exemplification of concepts -- similar capabilities exist in other MBT tools as well.
The MBT Capability Model Summary
The following Figure 1 summarizes our point of view on the MBT Capability Model.
Figure 1: MBT Capability Model summary
The different levels of the MBT model are described as follows. I use the canonical Crawl-Walk-Run-Soar analogy to describe the evolution of capabilities in these levels.
The “Initial” level (Level 0) represents the state before the adoption of MBT. In this state, typically test designs are created manually, and it takes significant effort to perform analysis of impact from requirement changes.
The Basic Level (Level 1)
The basic level is the starting point for MBT adoption. Here the key focus is on the following capabilities:
1. Development of requirements in the form of models in an MBT tool.
This is the most important step in the MBT journey – the modeling of requirements. There are a variety of modeling techniques (for example see here) that may be used for MBT. One of the key approaches we recommend is behavioral modeling. In such models we express the intended behavior of the application as an active flow chart (see example in the Figure 2 below for a simple model of login process in ARD tool)
Figure 2: Requirements model in MBT
Such models may be created for different levels of requirements, for example at the business process level, feature or story level and linked to each other. Models may be described at different levels of abstraction with high level flows (e.g. developed by Business Analysts or Product Owners) supported by deeper sub-flows (e.g. developed by Testers), etc. See Figure 3 below.
Figure 3: Detailed modeling in MBT
Visual models such as the above provide a collaborative platform for disambiguating application requirements and the behavior of the application. For more details on the modeling process, please see here.
2. Automated generation of test cases and automated change impact analysis and update of tests
Once the model is defined, test cases can be generated automatically from the model. This eliminates the effort to manually define and document test cases. But more importantly, these tests can be refreshed (or updated) whenever the model is updated with any changes. MBT tools will identify tests that have been impacted/updated, new tests that were created and tests that have been deprecated as a result of change. For example, change impact analysis may be used to generate a set of lean build verification tests tied only to the changes in a development build.
3. Generation of Behavior-Driven Development (BDD) test cases
Application development teams that use BDD as part of Test Driven Development may benefit from MBT by having the BDD test cases automatically generated from the behavioral model without having to write (and maintain) the BDD scenarios manually (for example BDD feature files in Gherkin).
4. Automated optimization of tests and scientific test coverage
One of the best capabilities of MBT is the automated optimization of tests that provides the maximum coverage with the least number of steps. MBT offers multiple test optimization options which can also be tuned based on additional parameters such as test importance and risk. As part of the optimization, MBT also offers a more precise calculation of the test coverage (and test redundancies) than manually possible.
5. Integration with Requirements/Test Management Tools and automated traceability of tests to requirements
Since the tests are generated from the model, MBT enables automatic maintenance of traceability between requirements and tests. No sperate manual work is required to maintain the traceability. If user stories (or features) are mapped to models, this allows the automated maintenance of traceability between such requirements and the generated tests. In addition, text-based requirements (such as features/user stories) can be enriched with information from the model (such as flow diagrams, decision tables, etc). See Figure 4 below
Figure 4: Integration of MBT models with requirements and test cases
Key Metrics impacted at Basic Level
The following are the key metrics that will be significantly impacted by the capabilities in Level 1:
|
Metric
|
Comments
|
|
Test design time/effort ↓↓
|
Significant reduction in test design time (compared to manual design, often over 30%) due to automation of the test design
|
|
Test coverage ↑
Code coverage ↑ and Code quality ↑ (using model-based BDD)
|
Notable increase in test coverage (algorithmically validated) since test design algorithms explore every possible test path through a given model.
Similarly if MBT is used to generate BDD test cases, it may help to improve code coverage (and quality) during development
|
|
Change impact analysis effort ↓↓↓
Test maintenance effort ↓↓↓
|
Orders of magnitude reduction in change impact analysis effort since the model automatically computes the impact of any change in the model. This allows impacted tests to be refreshed automatically without laborious manual updates
|
|
Requirements traceability to tests ↑
|
Automatically generates traceability from requirements to tests, hence reduces effort for creating and maintaining such traceability
|
|
Effort/elapsed time for test execution ↓
Over-testing/redundancy ↓↓
Overall testing efficiency ↑
|
Effort for test execution is reduced since optimization provides the least number of tests required to get the maximum coverage. Change-impact based testing further reduces testing effort
At the same time, over-testing or test redundancies are lowered since they are detected by the model
|
|
In-process defect containment ↑
|
Generally, defect containment increases because the model generates the most optimized (or change-affected) test thereby enabling testers to focus
|
The Automated Level (Level 2)
At Level 2, the emphasis is on generation of automated tests and automating the process of test automation. Here the key focus is on the following capabilities:
1.Tie models to test automation engines to generate automated test scripts
This involves embedding automation script-lets in model steps to invoke test automation execution engines. Typically this involves invoking actions in the automation framework action library using the keyword-driven test automation technique. This enables the MBT tool to generate automation test scripts corresponding to the optimized tests it generates. MBT tools such as ARD support a variety of test automation engines such as Selenium, HP-UFT, Appium, Eggplant, etc.
This capability not only reduces the time to create automated test scripts, but dramatically reduces the effort to maintain such scripts when changes are made to the behavior of the application (reflected in model changes)
2. Automation of BDD test cases
Developers using MBT to generate BDD tests (as described in the Level 1 capability) can generate automated BDD tests (e.g. Cucumber scripts)
3. Integration with Continuous Integration (CI) processes and tools
Once we are able to generate automated tests from MBT, we can tie such tests to CI processes and tools to enable agile development and testing. For example automated BDD tests can be invoked (e.g. by a CI tool such as Jenkins) as part of every build. Similarly automated integration tests can be executed as part of integration testing following the build. Another example is dynamically generated build verification test scripts that are tied to the changes incorporated in the build – this would replace statically (or manually defined) build verification test suites.
Key Metrics impacted at Automated Level
The following are the key metrics that will be significantly impacted by the capabilities in Level 2:
|
Metric
|
Comments
|
|
Test automation effort ↓
Test automation maintenance effort ↓↓
|
This is the top metric impacted by Level 2 capabilities. Since automated test scripts are generated automatically from the model, it significantly reduces the effort to create such scripts.
But more importantly, the automated script maintenance effort is reduced dramatically since it eliminates the laborious manual effort to do so
|
|
Test automation level (or % of tests automated) ↑↑
|
Due to the reduction of test automation (and maintenance) effort, we see significant improvement of test automation level, often by as high as 70%
|
|
Effort/elapsed time for test execution ↓↓
Overall testing efficiency ↑
|
The elapsed time for tests execution reduces as a natural benefit from test automation. However, MBT allows further reduction of such test automation execution time by ensuring that the most optimized (or impacted) set of tests are automated (and executed) -- as opposed to executing all the tests.
This helps to improve overall testing efficiency.
|
|
Lead time for earliest feedback from QA to Dev ↓
|
This is a key metric for supporting agile development and testing. Automation of tests and their tie-in to CI tools enables rapid test feedback to developers, which is a key tenet of agile delivery
|
|
Code (unit test) coverage ↑↑ (when using model-based BDD)
Defects leaked beyond Dev ↓
|
When using model-based BDD to generate automated BDD test cases, it accelerates the execution of larger number of such BDD tests in rapid time, thereby improving code coverage and quality even further.
This could also help in improving in-process defect containment
|
The Integrated Level (Level 3)
The integrated level is where MBT achieves its full potential by enabling integration of models with associated testing processes and tool such as: test data management, service virtualization, performance testing, etc. Here the key focus is on the following capabilities:
1. Integration with Test Data Management (TDM)
Test coverage by itself is not completely effective without appropriate test data coverage. This is especially important for automated test execution which gives us the ability to run a large number of tests quickly using a variety of test data sets, thereby improving overall test effectiveness.
Generation of test data, however, is generally a laborious and time-consuming activity, and often a major bottleneck in the testing process. Matter of fact test data management is typically overlooked early in the development lifecycle, especially during unit and integration tests. Developers and software developers in test (SDETs) typically manually fabricate some limited test data to just get by, thereby limiting the effectiveness of their tests.
Automation of test data management using tools such as Test Data Manager provides testers the ability to address this bottleneck. Tying model-based testing (in ARD) to test data management (using TDM) provides significantly higher conjugated benefits of both!
This is done in ARD by embedding test data rules and constraints as part of the model itself. These constraints are expressed using a variety of built in functions tied to the model steps. These rules are then used by ARD to automatically and synthetically generate test data to accompany the test cases. This allows test data management to shift-left by supporting adequate data generation during the early stages of the lifecycle (such as unit and integration tests).
In addition, this allows effective test data matching, which enables an optimized set of test data to be populated matched to the specific needs of the optimized test set generated from ARD. See Figure 5 below.
Figure 5: Integration of MBT models with test data
2. Integration with Service Virtualization
Service virtualization is an innovative technique that enables the removal of dependency constraints between applications during development and testing. Service Virtualization (SV) is the leading tool in this space. ARD integrates with Service Virtualization to allow the generation of data driven synthetic virtual services from the model using request-response (RR) pairs.
A synthetic virtual service is created to represent an endpoint (or dependency) that does not yet exit – for example, a component that is still under development. The RR pairs can either be created manually, algorithmically (using in-built computational functions in ARD) or by accessing data from TDM. For example, let’s say, the “Payment” step in the model in Figure 3 requires an invocation to an external payment gateway that we’d like to virtualize. We define the RR pairs for the service and export them to SV for generation of the virtual service. See Figure 6 below.
Figure 6: Generation of synthetic virtual services from a model
3. Integration with Performance Testing
In addition to generating functional tests, models may also be used to generate performance test scripts for the Taurus framework. Developers can execute these scripts either locally (as part of unit tests) or over the cloud (during load tests) using Blazemeter.
Figure 7 below is an extension of Figure 4 to illustrate the full range of integration and types of outputs that can be automatically created from models in ARD. The key value-add is that all of these outputs can be refreshed automatically every time there is a change in the model. This allows the entire agile testing process to be change-impact driven, which is a key imperative for agile lifecycles. The model precisely (and automatically) analyzes the impact of change and re-generates all the impact assets at the touch of a button without human effort required to perform change analysis.
Figure 7: Integration between ARD and other tools
Key Metrics impacted at Integrated Level
The following are the key metrics that will be significantly impacted by the capabilities in Level 3:
|
Metric
|
Comments
|
|
Test data coverage
Effort/time for test data creation ↓
|
Significant improve in test data coverage and reduction of test data creation effort (by >~50)
|
|
Wait time/effort for dev/test environment setup specially in the Dev and Integration environments ↓
|
Generation of synthetic data and virtual services significantly speeds up the velocity of test environments (and reduces test feedback time) often by greater than 60%
|
|
Code quality from performance perspective and algorithmic performance improvements ↑↑
|
Early performance testing ensures high algorithmic performance
|
The Intelligent Level (Level 4)
The Intelligent level is where MBT is infused with advanced analytics and cognitive capabilities to enable dynamic optimization and adaptive testing based on intelligence gleaned from application lifecycle data. Here the key focus is on the following capabilities:
1. Dynamic and adaptive optimization of tests and other test assets based on lifecycle data
In the previous levels, MBT models provide optimized outputs (such as test cases and test data) using static optimization techniques – based on human input such as priorities, risk, etc. By leveraging data analytics and machine learning, an adaptive framework dynamically optimizes these assets based on lifecycle data such as: build failure trends, defect detection and escape patterns, root causes of defects, changes in environment topologies etc, see Figure 7.
Figure 7: Dynamic adaptive optimization of tests based on lifecycle intelligence
This capability is underpinned by advanced analytics and AI framework that continuously collects data across the application lifecycle and identifies patterns to enable predictive optimization of tests.
Figure 8: Intelligent testing framework
2. Predictive quality
The use of predictive analytics allows us to be more predictive about our quality initiatives. Testing is transformed from a defect finding function to a predictive and decision support function where we are able to better automatically forecast things such as: what defects are likely to be found where, how much effort testing is likely to consume and what quality risk levels exist in the release. See Figure 9 below.
Figure 9: Predictive testing capability
Key Metrics impacted at Integrated Level
The following are the key metrics that will be significantly impacted by the capabilities in Level 4:
|
Metric
|
Comments
|
|
Test optimization ↑↑
|
Typical optimization improvement (compared to static test optimization) > 80%
|
|
Overall application quality improvement ↑↑
|
Near flawless application quality
|
|
Higher quality predictability ↑↑
|
Precise identification of trouble spots and focus on testing initiatives
|
Summary and looking forward
Hopefully, this article has provided readers with some insight into the power of model-based testing and how to progressively embrace the different capabilities of MBT supported by Broadcom ARD and related solutions. This is not the end of the capability journey, however. As MBT capabilities continue to evolve (beyond “Soar” to “Fly”, “Escape Velocity” etc. :-), we expect to see greater impact and benefits from such capabilities.