Manual Intervention
The book has now been published and the content of this chapter has likely changed substanstially.Please see page 250 of xUnit Test Patterns for the latest information.
A test requires a person to perform some manual action each time it is run.
Symptoms
The person running the test must do something manually either before the test is run or part way through the test run otherwise the test fails. A person may need to verify the results of the test manually.
Impact
Automated tests are all about getting early feedback on problems introduced into the software. If the cost of getting that feedback is too high in the form of Manual Intervention, we likely won't run the tests very often and we won't get the feedback very often. If we don't get that feedback very often, we'll probably introduce lots of problems between test runs and that will lead to Frequent Debugging (page X) and High Test Maintenance Cost (page X).
Manual Intervention also makes it impractical to have a fully-automated Integration Build[SCM] and regression test process.
Causes
The causes of Manual Intervention are as varied as the kinds of things our software does or encounters. The following are some general categories of the kinds of things that require Manual Intervention. This list is by all means not exhaustive.
Cause: Manual Fixture Setup
Symptoms
A person has to set up the test environment manually before the automated tests can be run. This may take the form of configuring servers, starting server processes or running scripts to set up a Prebuilt Fixture (page X).
Root Cause
This is typically caused by a lack of attention to automating the fixture setup phase of the test. It may also be caused by excessive coupling between components in the system under test (SUT) that prevent us from testing a majority of the code in the system inside the development environment.
Possible Solution
When using a Prebuilt Fixture it is important not to modify anything in the fixture to ensure we don't have Unrepeatable Tests (see Erratic Test on page X). This is called an Immutable Shared Fixture (see Shared Fixture on page X).
Cause: Manual Result Verification
Symptoms
We can run the tests but they almost always pass even when we know that the SUT is not returning the correct results.
Root Cause
If the tests we write are not Self-Checking Test (see Goals of Test Automation on page X), we can be given a false sense of security because tests will only fail if an error/exception is thrown.
Possible Solution
We need to ensure our tests are all self-checking by including result verification logic such as calls to Assertion Methods (page X) with the Test Methods (page X).
Cause: Manual Event Injection
Symptoms
A person must intervene during test execution to perform some manual action before the test can proceed.
Root Cause
Many events in a SUT are hard to generate under program control. Examples include unplugging network cables, bringing down database connections and pressing buttons on a user interface.
Impact
If a person needs to do something manually, it both increases the effort to run the test and ensures that the test cannot be run unattended. This torpedoes any attempt to do a fully-automated build & test cycle.
Possible Solution
The best solution is to find ways to test the software that do not require a real person to do the manual actions. If the events are reported to the SUT through asynchronous events, we can have the Test Method invoke the SUT directly passing it a simulated event object. If the SUT experiences the situation as a synchronous response from some other part of the system, we can get control of the indirect inputs by replacing some part of the SUT with a Test Stub (page X) that simulates the circumstances to which we want to expose the SUT.
Further Reading
Refer to the chapter Using Test Doubles for a much more detailed description of how to get control of the indirect inputs of the SUT.Copyright © 2003-2008 Gerard Meszaros all rights reserved