Custom Assertion
The book has now been published and the content of this chapter has likely changed substanstially.Please see page 474 of xUnit Test Patterns for the latest information.
Also known as: Bespoke Assertion
How do we make tests self-checking when we have test-specific equality logic?
How do we reduce Test Code Duplication when we have the same assertion logic in many tests?
How do we avoid Conditional Test Logic?
Create a purpose-built Assertion Method that compares only those attributes of the object that define test-specific equality.
Sketch Custom Assertion embedded from Custom Assertion.gif
Most members of the XUnit family provide a reasonably rich set of Assertion Methods (page X). But sooner or later, we'll find ourselves saying "this test would be so much easier to write if I just had an assertion that did ...". So why not write it ourself?
The reasons for writing a Custom Assertion are many but the technique is pretty much the same regardless. We hide the complexity of whatever it takes to prove the system is behaving correctly behind an Assertion Method with an Intent Revealing Name[SBPP].
How It Works
We encapsulate the mechanics of verifying that something is true (an assertion) behind an intent-revealing name. We do this by factoring out all the common assertion code within the tests into a Custom Assertion that implements the verification logic. A Custom Equality Assertion takes an Expected Object (see State Verification on page X) and the actual object as its parameters.
A key characteristic of Custom Assertions is that they receive everything they need to pass or fail the test as parameters. Other than causing the test to fail, they have no side effects.
Most typically, we'll create Custom Assertions through refactoring by identifying common patterns of assertions in our tests. When test driving we might just go ahead and call a non-existent Custom Assertion because it makes writing our test easier; this let's us focus on what needs to be tested rather than the mechanics of how the test would be carried out.
When To Use It
We should consider creating a Custom Assertion whenever any of the following are true:
- We find ourselves writing (or cloning) the same assertion logic in test after test (Test Code Duplication (page X).)
- We find ourselves writing Conditional Test Logic (page X) in the result verification part of our tests. That is, our calls to Assertion Methods are embedded in if statements or loops.
- The result verification parts of our tests are suffering from Obscure Test (page X) because we are using procedural rather than declarative result verification in the tests.
- We find ourselves doing Frequent Debugging (page X) whenever assertions fail because they do not provide enough information.
A key reason for moving the assertion logic out of the tests and into Custom Assertions is to Minimize Untestable Code (see Principles of Test Automation on page X). Once the verification logic has been moved into a Custom Assertion, we can write Custom Assertion Tests (see Custom Assertion on page X) to prove the verification logic is working properly. Another important benefit of using Custom Assertions is that it helps avoid Obscure Test and helps make tests Communicate Intent (see Principles of Test Automation). And that will help lead to robust, easily maintained tests.
If the verification logic must interact with the SUT to determine the actual outcome, we can use a Verification Method (see Custom Assertion) instead of a Custom Assertion. If the setup and exercise parts of the tests are also the same except for the value of actual/expected objects, we should consider using a Parameterized Test (page X). The primary advantage of Custom Assertions over both of these techniques is reusability; the same Custom Assertion can be reused in many different circumstances because it is independent of its context (because its only contact with the outside world is through its parameter list.)
We most commonly write Custom Assertions that are Equality Assertions (see Assertion Method) but there is no reason why we cannot write other kinds.
Variation: Custom Equality Assertion
For custom equality assertions, the Custom Assertions must be passed an Expected Object and the actual object to be verified. It should also take an Assertion Message (page X) to avoid playing Assertion Roulette (page X). It is essentially an equals method implemented as a Foreign Method[Fowler].
Variation: Object Attribute Equality Assertion
We often run across Custom Assertions that take one actual object and several different Expected Objects that need to be compared with specific attributes of the actual object. (The set of attributes to be compared with is implied by the name of the Custom Assertion.) The key difference between these and a Verification Method is that the latter interacts with the SUT while the Object Attribute Equality Assertion only looks at the objects passed in as parameters.
Variation: Domain Assertion
All the built-in Assertion Methods are domain independent. Custom Equality Assertions implement test-specific equality but still only compare two objects. There is another style of Custom Assertion that helps contribute to the definition of a "domain-specific" Higher Level Language (see Principles of Test Automation); the Domain Assertion.
A Domain Assertion is a Stated Outcome Assertion (see Assertion Method) that states something that should be true in domain-specific terms. It helps elevate the test into "business speak".
Variation: Diagnostic Assertion
Sometimes we find ourselves doing Frequent Debugging whenever a test fails because the assertions are only telling us that something is wrong but not specifically what (e.g. these two objects are not equal but it isn't clear what isn't equal about them). We can write a special kind of Custom Assertion that may look just like one of the built-in assertions but which can provide more information about what is different between the expected and actual values than a built-in assertion because it is specific to our types. (E.g. It would tell us which attribute(s) are different or where long strings differ, etc.).
On one project, we were comparing string variables containing XML. Whenever a test failed, we had to bring up two String inspectors and scroll through them looking for the difference. Finally, we got smart and included the logic in a Custom Assertion that told us where the first difference between the two XML strings occurred. The small amount of time we spent writing the diagnostic custom assertion was paid back many times over as we ran our tests.
Variation: Verification Method
In customer tests a lot of the complexity of verifying the outcome is related to interacting with the SUT, Verification Methods are a form of Custom Assertion that interact directly with the system under test (SUT) thus relieving their callers from this task. This simplifies the tests significantly and leads to a more "declarative" style of outcome specification. After the Custom Assertion has been written, subsequent tests that result in the same outcome are much quicker to write. In some cases, it may be advantageous to incorporate even the exercise SUT phase of the test into the Verification Method. This is one step short of a full Parameterized Test that incorporates all the test logic in a reusable Test Utility Method (page X).
Implementation Notes
The Custom Assertion is typically implemented as a set of calls to the various built-in Assertion Methods. Depending on how we plan to use it in our tests, we may also want to include the standard Equality Assertion template to ensure correct behavior with null parameters. Because the Custom Assertion is itself an Assertion Method, it should not have any side effects nor should it call the SUT. (If it needs to do that, it would be a Verification Method.)
Variation: Custom Assertion Test
Testing zealots would also write a Custom Assertion Tests (a Self-Checking Test (see Goals of Test Automation on page X) for Custom Assertions) to verify the Custom Assertion. The benefits of doing so are obvious: increased confidence in our tests. And in most cases it isn't very hard to do because Assertion Methods take all their arguments as parameters.
We can treat the Custom Assertion as the SUT simply by calling it with various arguments and verifying that it fails in the right cases. Single Outcome Assertions (see Assertion Method) only need a single test because they don't take any parameters (other than possibly an Assertion Message. Stated Outcome Assertions need one test for each possible value (or boundary value). Equality Assertions need one that compares two objects deemed to be equivalent, one for comparing an object with itself, and one test for each attribute whose inequality should cause the assertion to fail. Attributes that don't affect equality can be verified in one additional test since the Equality Assertion should not raise an error for any of them.
The Custom Assertions follow the normal Simple Success Test (see Test Method on page X) and Expected Exception Test (see Test Method) templates with one minor difference: Because the Assertion Method is the SUT, the exercise SUT and verify outcome phases of the Four-Phase Test (page X) are combined into a single phase.
Each test consists of setting up the Expected Object and the actual object and then calling the Custom Assertion. If the objects should be equivalent, that's all there is to it. (The Test Automation Framework (page X) would catch any assertion failures and fail the test.) For the tests where we expect the Custom Assertion to fail, write the test as an Expected Exception Test (except that the "Exercise SUT" and "Verify Outcome" phases of the Four-Phase Test are combined into the single call to the Custom Assertion.)
The simplest way to build the objects to be compared for a specific test is to do something similar to One Bad Attribute (see Derived Value on page X); build the first object and make a deep copy of it. For success tests, modify any of the attributes that should not be compared. For each failure test, modify one attribute that should be grounds for failing the assertion.
A brief warning about a possible complication in a few members of the xUnit family: if they don't do all the test failure handling in the Test Runner (page X), calls to fail or built in assertions may put messages into the failure log even if we catch the error or exception in our Custom Assertion Test. The only way to get around this is to use an "Encapsulated Test Runner" to run each test by itself and verify that the one test failed with the expected error message.
Motivating Example
The following is an example where several test methods repeat the same series of assertions.
public void testInvoice_addOneLineItem_quantity1_b() {
// Exercise
inv.addItemQuantity(product, QUANTITY);
// Verify
List lineItems = inv.getLineItems();
assertEquals("number of items", lineItems.size(), 1);
// Verify only item
LineItem expItem = new LineItem(inv, product, QUANTITY);
LineItem actual = (LineItem)lineItems.get(0);
assertEquals(expItem.getInv(), actual.getInv());
assertEquals(expItem.getProd(), actual.getProd());
assertEquals(expItem.getQuantity(), actual.getQuantity());
}
public void testRemoveLineItemsForProduct_oneOfTwo() {
// setup:
Invoice inv = createAnonInvoice();
inv.addItemQuantity(product, QUANTITY);
inv.addItemQuantity(anotherProduct, QUANTITY);
LineItem expItem = new LineItem(inv, product, QUANTITY);
// Exercise
inv.removeLineItemForProduct(anotherProduct);
// Verify
List lineItems = inv.getLineItems();
assertEquals("number of items", lineItems.size(), 1);
LineItem actual = (LineItem)lineItems.get(0);
assertEquals(expItem.getInv(), actual.getInv());
assertEquals(expItem.getProd(), actual.getProd());
assertEquals(expItem.getQuantity(), actual.getQuantity());
}
//
// Adding TWO line items
//
public void testInvoice_addTwoLineItems_sameProduct() {
Invoice inv = createAnonInvoice();
LineItem expItem1 = new LineItem(inv, product, QUANTITY1);
LineItem expItem2 = new LineItem(inv, product, QUANTITY2);
// Exercise
inv.addItemQuantity(product, QUANTITY1);
inv.addItemQuantity(product, QUANTITY2);
// Verify
List lineItems = inv.getLineItems();
assertEquals("number of items", lineItems.size(), 2);
// verify first item
LineItem actual = (LineItem)lineItems.get(0);
assertEquals(expItem1.getInv(), actual.getInv());
assertEquals(expItem1.getProd(), actual.getProd());
assertEquals(expItem1.getQuantity(), actual.getQuantity());
// verify second item
actual = (LineItem)lineItems.get(1);
assertEquals(expItem2.getInv(), actual.getInv());
assertEquals(expItem2.getProd(), actual.getProd());
assertEquals(expItem2.getQuantity(), actual.getQuantity());
}
Example TestCodeDuplication embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
Note that the first test ends with a series of three assertions and the second test repeats the series of three assertions twice, once for each line item. This is clearly a bad case of Test Code Duplication.
Refactoring Notes
Refactoring zealots can probably see that the solution is to do an Extract Method[Fowler] refactoring on these tests. By pulling out all the common calls to Assertion Methods we will be left with only the differences in each test. The extracted method is our Custom Assertion. We may also need to introduce an Expected Object to hold all the values that were being passed to the individual Assertion Methods on a single object to be passed to the Custom Assertion.
Example: Custom Assertion
In this test, we use a Custom Assertion to verify the LineItem matches the expected LineItem(s). For one reason or another, we have chosen to implement test-specific equality rather than using a standard Equality Assertion.
public void testInvoice_addOneLineItem_quantity1_() {
Invoice inv = createAnonInvoice();
LineItem expItem = new LineItem(inv, product, QUANTITY);
// Exercise
inv.addItemQuantity(product, QUANTITY);
// Verify
List lineItems = inv.getLineItems();
assertEquals("number of items", lineItems.size(), 1);
// Verify only item
LineItem actual = (LineItem)lineItems.get(0);
assertLineItemsEqual("LineItem", expItem, actual);
}
public void testAddItemQuantity_sameProduct_() {
Invoice inv = createAnonInvoice();
LineItem expItem1 = new LineItem(inv, product, QUANTITY1);
LineItem expItem2 = new LineItem(inv, product, QUANTITY2);
// Exercise
inv.addItemQuantity(product, QUANTITY1);
inv.addItemQuantity(product, QUANTITY2);
// Verify
List lineItems = inv.getLineItems();
assertEquals("number of items", lineItems.size(), 2);
// Verify first item
LineItem actual = (LineItem)lineItems.get(0);
assertLineItemsEqual("Item 1",expItem1,actual);
// Verify second item
actual = (LineItem)lineItems.get(1);
assertLineItemsEqual("Item 2",expItem2, actual);
}
Example CustomAssertionUsage embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
The tests have become significantly smaller and more intent revealing. We have also chosen to pass a string indicating which item we are looking at as a argument to the Custom Assertion to avoid playing Assertion Roulette when a test fails.
This simplified test was made possible by having the following Custom Assertion available to us:
static void assertLineItemsEqual( String msg, LineItem exp, LineItem act) {
assertEquals(msg+" Inv", exp.getInv(),act.getInv());
assertEquals(msg+" Prod", exp.getProd(), act.getProd());
assertEquals(msg+" Quan", exp.getQuantity(), act.getQuantity());
}
Example CustomAssertionMethodCamug embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
Note that it compares the same attributes of the object as we were comparing inline in the previous version of the test so the semantics of the test haven't changed. We are also concatenating the name of the attribute being compared with the message parameter to get a unique failure message to avoid playing Assertion Roulette when a test fails.
Example: Domain Assertion
In this next version of the test we have further elevated the level of the assertions to better communicate the expected outcome of the test scenarios:
public void testAddOneLineItem_quantity1() {
Invoice inv = createAnonInvoice();
LineItem expItem = new LineItem(inv, product, QUANTITY);
// Exercise
inv.addItemQuantity(product, QUANTITY);
// Verify
assertInvoiceContainsOnlyThisLineItem(inv, expItem);
}
public void testRemoveLineItemsForProduct_oneOfTwo_() {
Invoice inv = createAnonInvoice();
inv.addItemQuantity(product, QUANTITY);
inv.addItemQuantity(anotherProduct, QUANTITY);
LineItem expectdItem = new LineItem(inv,product,QUANTITY);
// Exercise
inv.removeLineItemForProduct(anotherProduct);
// Verify
assertInvoiceContainsOnlyThisLineItem(inv, expectdItem);
}
Example VerificationMethodUsage embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
This simplified version of the test was made possible by extracting the following Domain Assertion method:
void assertInvoiceContainsOnlyThisLineItem( Invoice inv,
LineItem expItem) {
List lineItems = inv.getLineItems();
assertEquals("number of items", lineItems.size(), 1);
LineItem actual = (LineItem)lineItems.get(0);
assertLineItemsEqual("",expItem, actual);
}
Example VerificationMethod embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
Example: Exercising Verification Method
If the exercise SUT phase of several tests is also the same, we can incorporate it into our reusable Custom Assertion as well. Because this changes the semantics of the Custom Assertion from being just a function free of side-effects to an operation that changes the state of the SUT, we usually give it a more distinctive name starting with "verify".
This version of the test merely sets up the test fixture before calling a Exercising Verification Method that incorporates both the exercise SUT and verify outcome phases of the test. It is most easily recognized by the lack of an "exercise" phase in the test that is calling it.
public void testAddOneLineItem_quantity2() {
Invoice inv = createAnonInvoice();
LineItem expItem = new LineItem(inv, product, QUANTITY);
// Exercise & Verify
verifyOneLineItemCanBeAdded(inv, product, QUANTITY, expItem);
}
Example ExercisingVerificationMethodUsage embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
The Custom Assertion for this example looks like this:
public void verifyOneLineItemCanBeAdded( Invoice inv, Product product,
int QUANTITY, LineItem expItem) {
// Exercise
inv.addItemQuantity(product, QUANTITY);
// Verify
assertInvoiceContainsOnlyThisLineItem(inv, expItem);
}
Example ExercisingVerificationMethodDefn embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
Note that it calls the "pure" Custom Assertion although it could just as easily have included all the assertion logic if we didn't have the other Custom Assertion to call.
Example: Custom Assertion Test
This Custom Assertion isn't particularly complicated so we may feel comfortable without having any automated tests for it. But if there is anything complex about it, it may be worth writing some tests like these:
public void testassertLineItemsEqual_equivalent() {
Invoice inv = createAnonInvoice();
LineItem item1 = new LineItem(inv, product, QUANTITY1);
LineItem item2 = new LineItem(inv, product, QUANTITY1);
// exercise/verify:
assertLineItemsEqual("This should not fail",item1, item2);
}
public void testassertLineItemsEqual_differentInvoice() {
Invoice inv1 = createAnonInvoice();
Invoice inv2 = createAnonInvoice();
LineItem item1 = new LineItem(inv1, product, QUANTITY1);
LineItem item2 = new LineItem(inv2, product, QUANTITY1);
// exercise/verify:
try {
assertLineItemsEqual("Msg",item1, item2);
} catch (AssertionFailedError e) {
assertEquals("e.getMsg", "Invoice-expected: <123> but was <124>",
e.getMessage());
return;
}
fail("Should have thrown exception");
}
public void testassertLineItemsEqual_differentQuantity() {
Invoice inv = createAnonInvoice();
LineItem item1 = new LineItem(inv, product, QUANTITY1);
LineItem item2 = new LineItem(inv, product, QUANTITY2);
// exercise/verify:
try {
assertLineItemsEqual("Msg",item1, item2);
} catch (AssertionFailedError e) {
return;
}
fail("Should have thrown exception");
}
Example CustomAssertionTest embedded from java/com/clrstream/camug/example/test/InvoiceTest.java
I have shown a few of the Custom Assertion Tests needed for this Custom Assertion. Note that there is one "equivalent" and several "different" tests (one for each attribute who's difference should cause the test to fail.) I've had to use the second form of the Expected Exception Test template for the cases where we expect the assertion to fail. This is because fail throws the same exception as our assertion method. In one of the "different" tests, I've included sample logic for asserting on the exception message. (Although I've abridged it to save space, it should give you an idea of where to assert on the message.)
Copyright © 2003-2008 Gerard Meszaros all rights reserved