Frequent Debugging

Symptoms

A test run results in a test failure or a test error. The output of the Test Runner (page X) is insufficient for us to determine the problem so we have to use an interactive debugger (or print statements sprinkled throughout the code) to determine where things are going wrong.

If this case is an exception, we needn't worry about it but if most test failures require this kind of debugging then we have a case of Frequent Debugging.

Causes

Frequent Debugging is caused by a lack of Defect Localization (see Goals of Test Automation on page X) in our suite of automated tests. The failed tests should tell us what went wrong either through their individual failure messages (see Assertion Message (page X)) or through the pattern of test failures. If they don't:

• We may be missing the detailed unit tests that would point out a logic error inside an individual class.
• We may be missing the component tests for a cluster of classes (i.e. a component) that would point out an integration error between the individual classes. This can happen when we use Mock Objects (page X) extensively to replace depended-on objects but the unit tests of the depended-on objects don't match the way the Mock Objects are being programmed to behave.

I've encountered this problem most frequently when I had written the higher level (functional or component) tests but had failed to write all the unit tests for the individual methods. (Some people would call this approach test-first development to distinguish it from test-driven development where every little bit of code is pulled into existence by a failing unit test.)

Frequent Debugging can also be caused by Infrequently Run Tests (see Production Bugs on page X). If we run our tests after every little change we make to the software we can remember what we changed since the last time we ran the tests. That means that when a test fails we don't have to spend a lot of time trouble-shooting the software to discover where the bug is; we know where it is because we remember putting it there!

Impact

Manual debugging is a slow, tedious process. It is easy to overlook subtle indications of the problem and spend many hours tracking down a single logic error. It reduces productivity and makes development schedules much less predictable because a single manual debugging session could extend the time required to develop the software by half a day or more.

Solution Patterns

If we are missing the customer tests for a piece of functionality and manual user testing has revealed a problem not exposed by any automated tests, we probably have a case of Untested Requirement (see Production Bugs). We can ask ourselves what kind of automated test would have prevented the manual debugging session. Better yet, once we have identified the problem, we can write the test that exposes it. Then we can use the failing test to do test-driven bug fixing! If we suspect this to be a widespread problem, we can create a development task to identify and write any additional tests that would be required to fill the gap we just exposed.

Doing true test-driven development is the best way to avoid the circumstances that lead to Frequent Debugging. We should start as close as possible to the skin of the application and do storytest-driven development writing unit tests for individual classes as well as component tests for the collections of related classes to ensure we have good Defect Localization.

Page generated at Wed Feb 09 16:39:50 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved