Software Testability, Part 3: Accidental Untestability

The next several posts about testability will cover programming techniques that can enhance or diminish testability. To reveal a bug, a test must:

  • Reach the buggy code
  • Trigger the bug
  • Propagate the incorrect result to an observable interface
  • Incorrect result must be observed
  • The incorrect result must be recognized as a failure.

 The absence any of these conditions means a test will fail in its mission to reveal bugs. 

Complexity contributes to this. There are many technical definitions of this, but for my talk, I tried to find visual and aural metaphors for complexity.  Here’s a visualization:

Jackson Pollock, Autumn Rhythms. 1950, Enamel on Canvas.

Jeff Voas provided canonical example of how even simple code can be hard to test in his excellent analysis of testability.

int scale(int j) {
    j = j - 1;             // should be j = j + 1
    j = j/30000;
    return j;
}

Exhaustive testing of this trivial function is possible: there are 65,536 possible values for input j.

But how many of these tests would reveal the bug? Only six test cases will produce an incorrect result:  -30001, -30000, -1, 0, 29999, and  30000.  All of the other 65,530 will run with out any failure and the bug will not be revealed.

So even trivial programs can limit testability, for even trivial bugs.  Why is this? Just executing buggy code doesn’t always result in a observable failure (the technical term for this is “coincidental correctness”.) 

Do you remember the Y2K IT pandemic?  Systems that had worked just fine for decades were certain to fail when run after December 31, 1999. This is an example of a kind of bug that produces an incorrect result only for certain data values – in that case, when the truncated year field advanced from 99 to 00. Many systems interpreted the 00 as 1900 (not 2000) resulting in many kinds failures.

What can we do about complexity?  Essential complexity is the implementation-independent extent of functions and aspects. Simply put, more is more. But if the “more” is necessary to solve the application problem at hand, this can’t be avoided. 

Accidental complexity is often avoidable and usually the result of careless amateur development or selfish local optimizations.  In fact this is so common, it has its own anti-pattern:  The Big Ball of Mud. Of course, this diminishes testability and helps to hide many bugs.

Dancing HampstersOther implementation aspects often decrease testability. 

  • Non-deterministic Dependencies
  • Race conditions
  • Message latency
  • Threading
  • CRUD shared and unprotected data

Dancing Hampsters are a great visual metaphor for the behavior complex systems – cute but unstable.

But all is not lost. Part 4, will discuss how to improve testability.

 

Share

About Robert V. Binder

Business leader, serial entrepreneur, and software technologist with extensive systems engineering experience.
This entry was posted in Software Testing, Technology and tagged . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>