6.1.1 Test Tool Classification

Test tools can have one or more of the following purposes depending on the context:

• Improve the efficiency of test activities by automating repetitive tasks or tasks that require significant resources when done manually (e.g., test execution, regression testing)

• Improve the efficiency of test activities by supporting manual test activities throughout the test process (see section 1.4)

• Improve the quality of test activities by allowing for more consistent testing and a higher level of defect reproducibility

• Automate activities that cannot be executed manually (e.g., large scale performance testing)

• Increase reliability of testing (e.g., by automating large data comparisons or simulating behavior)

Tools can be classified based on several criteria such as purpose, pricing, licensing model (e.g., commercial or open source), and technology used. Tools are classified in this syllabus according to the test activities that they support.

Some tools clearly support only or mainly one activity; others may support more than one activity, but are classified under the activity with which they are most closely associated. Tools from a single provider, especially those that have been designed to work together, may be provided as an integrated suite.

Some types of test tools can be intrusive, which means that they may affect the actual outcome of the test. For example, the actual response times for an application may be different due to the extra instructions that are executed by a performance testing tool, or the amount of code coverage achieved may be distorted due to the use of a coverage tool. The consequence of using intrusive tools is called the probe effect.

Some tools offer support that is typically more appropriate for developers (e.g., tools that are used during component and integration testing). Such tools are marked with “(D)” in the sections below.

Tool support for management of testing and testware

Management tools may apply to any test activities over the entire software development lifecycle. Examples of tools that support management of testing and testware include:

• Test management tools and application lifecycle management tools (ALM)

• Requirements management tools (e.g., traceability to test objects)

• Defect management tools

• Configuration management tools

• Continuous integration tools (D)

Tool support for static testing

Static testing tools are associated with the activities and benefits described in chapter 3. Examples of such tool include:

• Static analysis tools (D)

Tool support for test design and implementation

Test design tools aid in the creation of maintainable work products in test design and implementation, including test cases, test procedures and test data. Examples of such tools include:

• Model-Based testing tools

• Test data preparation tools

In some cases, tools that support test design and implementation may also support test execution and logging, or provide their outputs directly to other tools that support test execution and logging.

Tool support for test execution and logging

Many tools exist to support and enhance test execution and logging activities. Examples of these tools include:

• Test execution tools (e.g., to run regression tests)

• Coverage tools (e.g., requirements coverage, code coverage (D))

• Test harnesses (D)

Tool support for performance measurement and dynamic analysis

Performance measurement and dynamic analysis tools are essential in supporting performance and load testing activities, as these activities cannot effectively be done manually. Examples of these tools include:

• Performance testing tools

• Dynamic analysis tools (D)

Tool support for specialized testing needs

In addition to tools that support the general test process, there are many other tools that support more specific testing for non-functional characteristics.