6.1.2 Benefits and Risks of Test Automation

Simply acquiring a tool does not guarantee success. Each new tool introduced into an organization will require effort to achieve real and lasting benefits. There are potential benefits and opportunities with the use of tools in testing, but there are also risks. This is particularly true of test execution tools (which is often referred to as test automation).

Potential benefits of using tools to support test execution include:

• Reduction in repetitive manual work (e.g., running regression tests, environment set up/tear down tasks, re-entering the same test data, and checking against coding standards), thus saving time

• Greater consistency and repeatability (e.g., test data is created in a coherent manner, tests are executed by a tool in the same order with the same frequency, and tests are consistently derived from requirements)

• More objective assessment (e.g., static measures, coverage)

• Easier access to information about testing (e.g., statistics and graphs about test progress, defect rates and performance)

Potential risks of using tools to support testing include:

• Expectations for the tool may be unrealistic (including functionality and ease of use)

• The time, cost and effort for the initial introduction of a tool may be under-estimated (including training and external expertise)

• The time and effort needed to achieve significant and continuing benefits from the tool may be under-estimated (including the need for changes in the test process and continuous improvement in the way the tool is used)

• The effort required to maintain the test work products generated by the tool may be underestimated

• The tool may be relied on too much (seen as a replacement for test design or execution, or the use of automated testing where manual testing would be better)

• Version control of test work products may be neglected

• Relationships and interoperability issues between critical tools may be neglected, such as requirements management tools, configuration management tools, defect management tools and tools from multiple vendors

• The tool vendor may go out of business, retire the tool, or sell the tool to a different vendor

• The vendor may provide a poor response for support, upgrades, and defect fixes

• An open source project may be suspended

• A new platform or technology may not be supported by the tool

• There may be no clear ownership of the tool (e.g., for mentoring, updates, etc.)