Preface
The book has now been published and the content of this chapter has likely changed substanstially.The Value of Self-Testing Code
In Chapter 4 of [Ref], Martin Fowler writes:
If you look at how most programmers spend their time, you'll find that writing code is actually a small fraction. Some time is spent figuring out what ought to be going on, some time is spent designing, but most time is spent debugging. I'm sure every reader can remember long hours of debugging, often long into the night. Every programmer can tell a story of a bug that took a whole day (or more) to find. Fixing the bug is usually pretty quick, but finding it is a nightmare. And then when you do fix a bug, there's always a chance that anther one will appear and that you might not even notiice it till much later. Then you spend ages finding that bug.
Some software is very difficult to test manually. In these cases we are often forced into writing test programs. I recall a project I was working on in 1996. My task was to build an event framework that would let client software register for an event and be notified when some other software raised that event (the Observer[GOF] pattern.) I could not think of a way to test this without writing some sample client software. I had about twenty different scenarios I need to try so I coded up each scenario with the requiste number of observers, events and event raisers. At first, I logged what was occuring in the console and scanned it manually. Scanning it became very tedious very quickly and being quite lazy, I looked for an easier way. For each test I populated a Dictionary indexed by the expected event and the expected receiver of it with the the name of the receiver as the value. When a particular receiver was notified of the event, they looked in the Dictionary for the entry indexed by themselves and the event they just received. If it existed, they removed themselves. If it didn't, they added the entry with an error message saying it was an unexpected event notification. After running all the tests, the test program merely looked in the Dictionary and printed out its contents if it was not empty. This made running all my tests nearly zero in cost. They either passed quietly or spewed a list of test failures. Unbeknown to myself, I had discovered the concept of a Mock Object (page X) and a Test Automation Framework (page X) out of necessity!
My First XP Project
In late 1999, I attended the OOPSLA conference and I picked up a copy of Kent Beck's new book on "Extreme Programming" [XPE]. I was used to doing iterative and incremental development and already believed in the value of automated unit testing although I had not tried to apply it universally. I had a lot of respect for Kent since I had known him since the first PLoP(The Pattern Languages of Programs conference.) conference back in 1994 so I decided that it was worth trying to apply eXtreme Programming on a ClearStream Consulting project. Shortly after OOPSLA I was fortunate to come across a suitable project for trying it on, an add-on application that interacted with an existing database but which had no user interface. The client was open to developing software in a different way.
We started doing eXtreme Programming "by the book" using pretty much all the practices including pair-programming, collective ownership and test-driven development. Of course, we had a few challenges figuring out how to test some aspects of the behavior of the application but we were writing tests for most of the code. Then, as the project progressed, I started to notice a disturbing trend; it was taking longer and longer to implement seemingly similar tasks.
I explained the problem to the developers and asked them to record on each task card how much time had been spent writing new tests, modifying existing tests and writing the production code. Very quickly, a trend emerged. While writing new tests and writing the production code seemed to be staying more or less constant, the amount of time spent modifying existing tests was increasing and the developers' estimates were going up as a result. When a developer asked me to pair on a task and we spent 90% of the time modifying existing tests to accommodate a relatively minor change, I knew we had to change something, and soon!
We analyzed the kinds of compile errors and test failures we were getting as we introduced the new functionality and found that many of the tests were affected by the changes to methods of the system under test (SUT). This was no surprise, but what was surprising is that most of the impact was during the fixture setup part of the test and that the changes were not impacting the core logic of the tests.
This was an important discovery because it showed us that we had the knowledge about how to create the objects of the SUT scattered across most of the tests. In other words, the tests knew too much about non-essential parts of the behavior of the SUT. I say "non-essential" because most of the affected tests did not care about how the objects in the fixture were created; they were interested in ensuring that they were in the correct state. Upon further examination, we found that many of the tests were creating identical or nearly identical objects in their test fixtures.
The obvious solution to this problem was to factor out this logic into a small set of Test Utility Methods (page X). There were several variations:
- When we had a bunch of tests that need identical objects, we simply created a method that returned that kind of object ready to use. We now call these Creation Methods (page X).
- Some tests needed to specify different values for some attribute of the object. So we passed that attribute as a parameter to the Parameterized Creation Method (see Creation Method).
- Some tests wanted to create a malformed object to ensure that the SUT would reject it. Defining a separate Parameterized Creation Method for each attribute cluttered the signature of our Test Helper (page X) so we created a valid object and then replaced the value of the One Bad Attribute (see Derived Value on page X).
We had discovered what would become(Technically, they are not truly patterns until they have been discovered by three independent project teams. ) our first test automation patterns.
Later, when tests started failing because the database did not like the fact that we were trying to insert another object with the same key that had a unique constraint, we added code to generate the unique key programmatically. We called this variant an Anonymous Creation Method (see Creation Method) to indicate the presence of this added behavior.
Identifying the problem, what we now call a Fragile Test (page X), was an important event on this project and the subsequent definition of the solution patterns saved this project from possible failure. Without this disc fa overy we would, at best, have abandoned the automated unit tests that we had already built. At worst, the tests would have reduced productivity so much that we would have been unable to deliver on our commitments. As it turned out, we were able to deliver what we had promised and with very good quality. Yes, the testers (The testing function is sometimes refered to as "Quality Assurance" but this usage is, strictly speaking, incorrect.) people still found bugs in our code because we were definitely missing some tests. Introducing the changes needed to fix those bugs, once we had figured out what the missing tests needed to look like, were relatively straight forward. We were hooked. Automated unit testing and test-driven development really did work and we have been using them consistently ever since.
As we applied the practices and patterns on subsequent projects, we have run into new problems and challenges. In each case, we have "peeled the onion" to find the root cause and have come up with ways to address them. As these techniques have matured, we have added them to our repertoire of techniques for automated unit testing.
We first described some of these patterns in a paper presented at XP2001. In discussion with other participants at that and subsequent conference we discovered that many of them were using the same or similar techniques. That elevated them from "practice" to "pattern" (a recurring solution to a recurring problem in a context). The first paper on test smells[RTC] was presented at the same conference building on the concept of code smells first described in [Ref].
My Motivation
I am a great believer in the value of automated unit testing. I practiced software development without it for the better part of two decades and I know that my professional life is much better with it than without it. I truly believe that the xUnit framework and the automated tests it enables is one of the truly great advances in software development. I find it very frustrating when I see companies trying to adopt automated unit testing but being unsuccessful because of a lack of key information and skills.
As a software development consultant with ClearStream Consulting, I get to see a lot of projects. Sometimes I get called in early in a project to help the client make sure they "do things right". More often than not I only get called in when things are already off the rails. As a result, I get to see a lot of "worst practices" that result in the test smells. If I am lucky and they called early enough, I can help the client recover from the mistakes. If not, they muddle through less than satisfied with how TDD and automated unit testing worked and the word goes out that automated unit testing is a waste of time.
In hindsight, most of these mistakes and best practices are easily avoidable given the right knowledge at the right time. But how do you get the knowledge without making the mistakes for yourself? At the risk of sounding self-serving, hiring someone who has the knowledge is the most time-efficient way of learning any new practice or technology. Taking a course or reading a book is a much less expensive (but less effective) alternative. I am hoping that by writing down a lot of these mistakes and how to avoid them, I can save you a lot of grief on your project whether it is fully agile or just more agile than it has been in the past. Gerry Wienberg's law of Raspberry Jam [SoC](The "Law of Raspberry Jam" states: "The wider you spread it, the thinner it gets.") not withstanding.
Who This Book Is For
I have written this book primarily for software developers (programmers, designers and architects) who want to write better tests and for the managers and coaches who need to understand what the developers are doing and why they (the developers) need to be cut enough slack so they can learn to do it even better! The focus is on developer tests and customer tests automated using xUnit but some of the higher level patterns also apply to tests automated using other technologies than xUnit. Rick Mugridge and Ward Cunningham have written an excellent book on Fit [FitB] and they advocate many of the same practices.
Developers will likely want to read the book cover to cover but they should focus on skimming the reference chapters rather than trying to read them word by word. The emphasis should be on getting an overall idea of which patterns exist and how they work so that they can come back to a particular pattern when a need arises. The first few elements (up to and include the "When to Use It" section) of each pattern should provide this overview.
Managers and coaches can focus on reading the narrative sections only and maybe read the Test Smells reference chapter. They can also read the Strategy patterns as these are decisions they need to understand and provide support to the developers as they work their way through them. As a minimum, managers should read the Goals of Test Automation chapter.
About the Cover Photo
One of the first thought I had when Martin Fowler asked if he could "steal me for his series" (right after "I'd be honored") was "What bridge would I put on the cover?" I thought about the ability of testing to avoid catastrophic failures of software and how that related to bridges. Several famous bridge failures immediately came to mind including the "Galloping Gertie" (the Tacoma Narrows bridge), the "Iron Workers Memorial Bridge" in Vancouver (named for the iron workers who died when a part of it collapsed during construction) and several others. After further reflection, it just did not seem right to claim that testing might have prevented these failures so I chose a bridge with a more personal connection. The picture on the cover is of the New River Gorge bridge in West Virginia. It spans the New River which cuts a deep gorge through the area. I first passed over and subsequently paddled under this bridge on a whitewater kayaking trip in the late 1980's. The style of the bridge is also relevant; the complex arch structure underneath the bridge is largely hidden from those who use it to get to the other side of the gorge. The road deck is completely level and four lanes wide resulting in a very smooth passage. In fact, at night it is quite possible to be completely oblivious to the fact that one is thousands of feet above the valley floor. A good test automation infrastructure has the same effect; writing tests is easy because most of the complexity is hidden beneath the road bed.
Colophon
This book was written using XML which I published to HTML for previewing on my web site. I edited the XML using Eclipse and the XML Buddy plug-in. The HTML was generated using a Ruby program that I first obtained from Martin Fowler and which I then evolved quite extensively as I evolved my custom markup language. Code samples were written, compiled and executed in (mostly) Eclipse and were inserted into the HTML automatically by XML tag handlers (one of the main reasons for using Ruby instead of XSLT.) This gave me the ability to "publish early, publish often" to the web site. I could also generate a single Word or PDF document for reviewers from the source but this required some manual steps.
Acknowledgments
While this book is largely a solo writing effort, many people have contributed in their own ways. Apologies in advance to anyone I may have missed. I tried to write in the first person but it many cases it just did not feel right so you will see "we" creeping in a fair bit.
People who know me well may wonder where I found all the time it takes to write a book like this. They know that when I am not working I am off doing various (some would say "extreme") outdoor sports. Sports such as backcounty (extreme) skiing, whitewater (extreme) kayaking and mountain (extreme) biking. Personally, I do not agree with the "extreme" adjective they might apply to my activities any more than I agree with it for highly iterative and incremental (extreme) programming but the question of time is a valid one. I must give special thanks to my friend Heather Armitage with whom I do most of the above activities. She has driven many long hours on the way to or from these adventures with me hunched over my laptop computer in the passenger seat working on this book. Also, thanks go to Alf Skrastins who loves to drive all his friends to back-country skiing venues west of Calgary in his Previa. Without their help, this book would have taken much longer to write!
As usual, I'd like to thank all my reviewers, both official and unofficial. Robert C ("Uncle Bob") Martin reviewed an early draft. The official reviewers of the first "official" draft were Lisa Crispin and Rick Mugridge. Lisa Crispin, Jeremy Miller, Alistair Duguid, Michael Hedgpeth and Andrew Stopford reviewed the second draft.
Thanks to my "shepherds" from the various PLoP conferences who provided feedback on drafts of these patterns; they provided expert comments on my experiments with the pattern form: Michael Stahl, Danny Dig and especially Joe Yoeder. I would also like to thank the members of the PLoP workshop group on Pattern Languages at PLoP 2004 and especially Eugene Wallingford, Ralph Johnson and Joseph Bergen. Brian Foote and the SAG group at UIUC posted several gigabytes of MP3's of the review sessions in which they discussed the early drafts of the book. Their comments caused me to rewrite from scratch at least one of the narrative chapters.
Many people e-mailed me comments about the material posted on my website at http://xunitpatterns.com or posted comments on the Yahoo! group; they provided very timely feedback on the sometimes very draft material I had posted there. These included: Javid Jamae, Philip Nelson,Tomasz Gajewski, John Hurst, Sven Gorts, Bradley T. Landis, Cedric Beust, Joseph Pelrine, Sebastian Bergmann, Kevin Rutherford, Scott W. Ambler, J. B. Rainsberger, Oli Bye, Dale Emery, David Nunn, Alex Chaffee, Burkhardt Hufnagel, Johannes Brodwall, Bret Pettichord, John Hurst, Clint Shank, Sunil Joglekar, Rachel Davies, Nat Pryce, Paul Hodgetts, Owen Rogers, Amir Kolsky, Kevin Lawrence and Joe Schmetzer. Special thanks to Neal Norwitz, Markus Gaelli, Stephane Ducasse and Stefan Reichhart who provided copious feedback as unofficial reviewers.
Quite a few people sent me e-mails describing their favorite pattern or special feature from their member of the xUnit family. Most of these were variations on patterns I had already documented; I've included them as aliases or implementation variations as appropriate. A few were more esoteric patterns that I had to leave out for space reasons and for that I apologize.
Many of the ideas described in this book came from projects I worked on with my colleagues from ClearStream Consulting. We all pushed each other to find better ways of doing things back in the early days of eXtreme Programming when there really were not any resources available. It was this single-minded determination that led to many of the more useful techniques described here. Those colleagues are Jennitta Andrea, Ralph Bohnet, Dave Braat, Russel Bryant, Greg Cook, Geoff Hardy, Shaun Smith and Thomas (T2) Tannahill. Many of them also provided early reviews of various chapters. Greg also provided many of the code samples in the Database Patterns chapter while Ralph set up my CVS repository and automated build process for the web site. I would also like to thank my bosses at ClearStream who let me take time off from consulting engagements to work on the book and for permision to use the code-based exercises from our 2 day "Testing for Developers" course as the base for many of the code samples. Thanks, Denis Clelland and Luke McFarlane!
Several people encouraged me to keep working on the book when the going was hard. They were always willing to take a phone call to discuss some sticky issue I was grappling with. Foremost amongst these were Joshua Kerievsky and Martin Fowler.
I'd like to especially thank Shaun Smith for helping me get started on this book and for all the technical support he provided throughout the early part of writing it. He hosted my web site, created the first CSS style sheets, taught me Ruby, set up a wiki for discussing the patterns and even provided some of the early content before personal and work demands forced him to pull out of the writing side of the project. Whenever I say "we" when I talk about experiences, I am probably referring to Shaun and myself although other co-workers may also share the same opinion.
Copyright © 2003-2008 Gerard Meszaros all rights reserved