Test Suite Object

Given that we have created Test Methods (page X) containing our test logic and placed them on a Testcase Class (page X), it would be nice to be able to run these tests as a single user operation.

How It Works

We define a Composite[GOF] Testcase Object (page X) called a Test Suite Object to hold the collection of individual Testcase Objects to execute. When we want to run all the tests in the test suite, the Test Runner (page X) asks the Test Suite Object to run all its tests.

Why We Do This

Treating test suites as first-class objects makes it easier for the Test Runner of the Test Automation Framework (page X) to manipulate tests in the test suite. With or without a Test Suite Object, the Test Runner would have to hold some kind of collection of Testcase Objects (so that we could iterate over them, count them, etc.); by making the collection "smart", it becomes simple to add other uses such as the Suite of Suites.

Variation: Testcase Class Suite

To run all the Test Methods in a single Testcase Class we simply build a Test Suite Object for the Testcase Class and add one Testcase Object for each Test Method. This allows us to run all the Test Methods simply by passing the Testcase Class name to the Test Runner.

Variation: Suite of Suites

We can build up larger Named Test Suites (page X) by composing smaller test suites into a tree structure. The Composite pattern makes this invisible to the Test Runner allowing it to treat a Suite of Suites exactly the same way it treats a simple Testcase Class Suite or a single Testcase Object.

Implementation Notes

As a Composite object, each Test Suite Object implements the same interface as a simple Testcase Object so that neither the Test Runner nor the Test Suite Object needs to be aware of whether it is holding a reference to a single test or an entire suite. This makes it easier to implement any operations that involve iterating across all the tests such a counting, running, displaying, etc..

Before we can do anything with our Test Suite Object, we must construct it. We have several options to choose from:

• Test Discovery (page X) - We can let the Test Automation Framework discover our Testcase Classes and Test Methods for us.
• Test Enumeration (page X) - We can write code that enumerates which Test Methods we want to include in a Test Suite Object. This usually involves creating a Test Suite Factory (see Test Enumeration).
• Test Selection (page X) - We can specify which subset of the Testcase Objects we want to include from an existing Test Suite Object.

Variation: Test Suite Procedure

Sometimes we have to write code in programming or scripting languages that do not support objects. Given that we have written a number of Test Methods, we need to give the Test Runner some way to find the tests. A Test Suite Procedure is how we can enumerate all the tests we want to run by invoking each test in turn. The calls to each test are hard-coded within the body of the Test Suite Object. Of course, a Test Suite Procedure may call several other Test Suite Procedures to realize a Suite of Suites.

The biggest single disadvantage of this approach is that it forces us into Test Enumeration and that increases the effort of writing tests as well as the likelihood of Lost Tests (see Production Bugs on page X). Because we are not treating our code as "data", we lose the ability to manipulate them at runtime. This means we cannot have a Graphical Test Runner (see Test Runner) with a hierarchy (tree) view of our Suite of Suites.

Example: Test Suite Object

Most members of the xUnit family implement Test Discovery so there isn't much of an example of Test Suite Object to see. The main evidence of the existence of Test Suite Objects appears in the Test Tree Explorer (see Test Runner) when we "drill down" into the Test Suite Object to expose the Testcase Objects it contains. Here's an example from the JUnit Graphical Test Runner built into Eclipse.

Example: Suite of Suites built using Test Enumeration

Here is an example of using Test Enumeration to construct a Suite of Suites.

public class AllTests {

   public static Test suite() {
      TestSuite suite = new TestSuite("Test for allJunitTests");
      suite.addTestSuite( com.clrstream.camug.example.test.InvoiceTest.class);
      suite.addTest(com.clrstream.ex7.test.AllTests.suite());
      suite.addTest(com.clrstream.ex8.test.AllTests.suite());
      suite.addTestSuite( com.xunitpatterns.guardassertion.Example.class);
      return suite;
   }
}
Example AllTests embedded from java/allJunitTests/AllTests.java

The first and last lines add the Test Suite Objects created from a single Testcase Class while the middle two lines each call the Test Suite Factory for another Suite of Suites. The Test Suite Object we return is likely at least three levels deep:

1. The Test Suite Object we instantiated and populated before returning,
2. the AllTests Test Suite Objects returned by the two calls to factory methods, and
3. the Test Suite Objects for each of the Testcase Classes aggregated into those Test Suite Objects.

This is illustrated in the following tree of objects:

TestSuite("Test for allJunitTests");
   TestSuite("com.clrstream.camug.example.test.InvoiceTest")
      TestCase("testInvoice_addLineItem")
      ...TestCase("testRemoveLineItemsForProduct_oneOfTwo")
   TestSuite("com.clrstream.ex7.test.AllTests")
      TestSuite("com.clrstream.ex7.test.TimeDisplayTest")
         TestCase("testDisplayCurrentTime_AtMidnight")
         TestCase("testDisplayCurrentTime_AtOneMinAfterMidnight")
         TestCase("testDisplayCurrentTime_AtOneMinuteBeforeNoon")
         TestCase("testDisplayCurrentTime_AtNoon")
         ...     TestSuite("com.clrstream.ex7.test.TimeDisplaySolutionTest")
         TestCase("testDisplayCurrentTime_AtMidnight")
         TestCase("testDisplayCurrentTime_AtOneMinAfterMidnight")
         TestCase("testDisplayCurrentTime_AtOneMinuteBeforeNoon")
         TestCase("testDisplayCurrentTime_AtNoon")
         ...TestSuite("com.clrstream.ex8.test.AllTests")
      TestSuite("com.clrstream.ex8.FlightMgntFacadeTest")
         TestCase("testAddFlight")
         TestCase("testAddFlightLogging")
         TestCase("testRemoveFlight")
         TestCase("testRemoveFlightLogging")
         ...     TestSuite("com.xunitpatterns.guardassertion.Example") TestCase("testWithConditionals")
         TestCase("testWithoutConditionals")
         ...
Example AllTestsTree embedded from java/allJunitTests/AllTestsObjectStructure.txt

Note also that this class doesn't subclass any other class; it does need to import TestSuite and the classes it is using as Test Suite Factoriess.

Example: Test Suite Procedure

In the early days of agile software development, before there were any agile project management tools, I built a set of Excel spreadsheets for managing tasks and user stories. To make life simpler, I automated frequently done tasks such as sorting all stories by release and iteration, sorting tasks by iteration and status, and so on. Eventually, I got bold enough to write a macro (a program, really) that would sum up the estimated and actual effort of all the tasks for each story. At this point, the code was starting to get somewhat complex and hard to maintain. A common problem was that if one of the named ranges used by the sorting macros was accidently deleted, they would error.

Unfortunately, there was no xUnit framework for VBA so all this was done without Tests as Safety Net (see Goals of Test Automation on page X). Here is the main program of the reporting macro. All output was written to a new sheet in the workbook.

'Main Macro
  
Sub summarizeActivities()
   Call VerifyVersionCompatability
   Call initialize
   Call SortByActivity
   
   For row = firstTaskDataRow To lastTaskDataRow
      If numberOfNumberlessTasks < MaxNumberlessTasks Then
         thisActivity = ActiveSheet.Cells(row, TaskActivityColumn).Value
         If thisActivity <> currentActivity Then
            Call finalizeCurrentActivityTotals
            currentActivity = thisActivity
            Call initializeCurrentActivityTotals
         End If
         Call accumulateActivityTotals(row)
      Else
         lastTaskDataRow = row   ' end the For loop right away
      End If
   Next row
   Call cleanUp
End Sub
Example UntestedVbaMacro embedded from VBA/ClearTrack.vba

When one has no tests and no Test Automation Framework, we do what we can to introduce some kind of regression testing. In this case, it was enough of a challenge (and a win) just to be able to exercise all the macros. If they ran to completion it was a much better indication that I hadn't broken anything major than not running them at all. The following is an example of the various Test Suite Procedures and the Test Methods they called:

Sub TestAll() Call TestAllStoryMacros
    Call TestAllTaskMacros
    Call TestReportingMacros
    Call TestToolbarMenus  'All The Same
End Sub

Sub TestAllStoryMacros() Call TestActivitySorting
    Call TestStoryHiding
    Call ReportSuccess("All Story Macros")
End Sub

Sub TestActivitySorting() Call SortStoriesbyAreaAndNumber
    Call SortActivitiesByIteration
    Call SortActivitiesByIterationAndOrder
    Call SortActivitiesByNumber
    Call SortActivitiesByPercentDone
End Sub

Sub TestReportingMacros() Call summarizeActivities
End Sub
Example TestSuiteProcedure embedded from VBA/ClearTrackTest.vba

The first Test Suite Procedure is a Suite of Suites while the second Test Suite Procedure is is the equivalent of a single Test Suite Object. The third Sub is the Test Method for exercising all the sorting macros. The last Sub exercises the summarizeActivities macro using a Prebuilt Fixture (page X). Because VBA is based on Visual Basic 5, it has no classes therefore we have no Testcase Class and no runtime Testcase Objects. (For those who might be wondering where the verify outcome phase of the test is, there isn't one in this test. It is not a Self-Checking Test (see Goals of Test Automation) and it is not a Single Condition Test (see Principles of Test Automation on page X) either. Shame on me!)

Page generated at Wed Feb 09 16:39:47 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved