Characterization Tests

  3 mins read  

I like software testing.  I know that it’s kind of the in thing to talk about testing software, especially in the Agile programming community. My observation though, has been that a lot of shops talk a good game without having a lot to show for it in the way of valuable tests written on their software. The reason for this gap between good intentions and reality?  Usually it’s the fact that it’s a lot easier to write tests on a greenfield system than on one that’s already established, and a lot of organizations that got turned on to Agile testing philosophies got turned on to them late in the game.

There are a number of ways to address this problem, many of which I’ll be talking about in future posts, but for now, I want to talk about characterization tests, which provide a relatively quick and easy way to jump right in to testing and get valuable test coverage on things that might otherwise take much longer to test properly.

What Is A Characterization Test?

A characterization test is an automated software test that describes the actual, observable behavior of your application.  In other words, it validates something that your software already does. Characterization tests are designed to detect change, and therefore will defend an existing system from undesired modification.

They should be written at a relatively high level, and they should be written based upon the current state of your application.  The assumption is that whatever your system is doing now is what you want it to do in the future, (unless you’re explicitly and intentionally changing that behavior, in which case you would change the characterization test and ideally write additional tests for the new code).

The reason that this strategy is a good way to begin adding tests to a system without them is that it doesn’t require you to intimately understand, (or write tests on), how the system works or why a given behavior is as it is.  All you’re doing is making sure that, for better or worse, the behavior as it stands won’t change.  These tests will help you to maintain a working state for your application and will make it possible to come back later to add additional tests once you’re armed with a better understanding of the internals.

Aren’t These Just Regression Tests?

Well — yes and no.  They’re only regression tests if the current, observed behavior of your application is the correct behavior.  When you write a regression test, you do so knowing that you’re describing the system as it should work.  You write a regression test to prevent a system from reverting to a known broken state.  With a characterization test, you’re making a lot of assumptions about the working state of your application, and you do so with the understanding that you could be wrong and future discoveries will cause the test to have to change.   Regression tests detect breakage, and characterization tests detect change, _whether that change is actually breakage or not.  As a consequence, characterization tests tend to change more frequently than regression tests do, (until you’re certain that the behavior the characterization test is defending _is correct behavior, at which point it really is a regression test).

How To Write A Characterization Test

The actual process of writing a characterization test will vary depending upon your platform and tools, but the ‘wetware algorithm’ for doing it goes something like this:

  1. Identify some discrete behavior in your application.  Do you have a counter that always starts at 60?  Does your website force you to login when you visit the /admin URL? Those are things worth testing. 
  2. Analyze the pre-conditions and inputs that drive/cause the behavior you observe, note variations based on those pre-conditions or inputs.
  3. Write tests that assert that the system behaves exactly as it does given a set of known inputs and pre-conditions. Write a test for each set of known inputs or pre-conditions. This may make your tests seem tautological, but that’s the point, your tests assume that things as they are now are as they should be.

Not a very detailed post this time, but something to think about, I hope.  In the future I’ll talk about writing characterization tests for specific platforms, (Ruby on Rails and Cucumber are an excellent combination for this purpose, for instance).