This post covers part four of my 2010 talk on testability. White box testability refers to specific programming practices and components that can improve or hinder testability.
Since Dykstra’s 1968 note “Go To Considered Harmful,” practices for producing clean, well-structured, readable, and maintainable software have been the subject of extensive discussion, research, argument, and use. The nearby image shows the result of a failure to apply these practices, as generated by Peter Ljubič’s analyzer. Imagine trying to devise tests to cover all these relationships.
I won’t try to recite generally accepted good programming practices here. All of them improve testability for code in any language running on any platform. Instead, a sketch follows of some specific programming techniques and built-in-test components that can improve testability.
TTCN is a language for defining test cases. One its key concepts is the Point of Control or Observation (PCO).
A PCO is an interface of the system under test. Usually, we don’t want an automated test suite directly coupled to PCOs. Instead, it is better use abstract PCOs so that parts of the SUT interface not necessary for testing or that would result in brittle dependencies are hidden inside another piece of software. This is usually a wrapper class whose public methods provide well-structured functions to control and observe the actual SUT interfaces. They may include some test helpers.
Abstract PCOs can facilitate testing multiple configurations. The test suite uses the same interface, regardless of which configuration is being tested.
Like all good software abstraction, a good PCO has two aspects: the conceptual and the physical. We want our abstractions to have a coherent focus (conceptual) and through the magic of indirection to decouple the physical SUT interface from the test cases that drive (control) it and evaluate (observe) test results.
What does a tester/test harness have to do activate SUT components and aspects? Everything you (or your test drivers) touch is a PCO. Also, don’t forget the SUT environment. With present-day technology stacks, this can be very complex.
A component or an interface is a result of the way the system was built. An aspect is a feature, function, or behavior of a system that is performed by two or more components. This It is usually much easier to drive components. However aspects more interesting, but typically not directly controllable.
State-based testing is needed when the SUT can respond differently to the same inputs, depending on what has already happened. For example, you must logon before you can logoff. A data item must be created before it can be changed or deleted. Once deleted, it cannot be changed, and so on.
State-based testing strategies aim to cover enough of the possible sequences to reveal bugs related to state-based behavior. However, the “state” that determines behavior might not be directly accessible in a class interface. Two kinds of functions help when this the case: get state and set state. It is useful to have a method for each state. For example, after deleting the last item in a collection, we’d expect that the method
is_empty() returns true.
Suppose a collection class can hold N items. To test its behavior when full or near-full, we’d have to first do N calls to
add(x). If N is large, this can be time-consuming and wasteful. This problem is obviated if we add a test method
force_full() to get to this state and one to
force_empty() so that we don’t have call
delete(x) N times to get back to the empty state.
State-based testing is dicussed extenstively in TOOSMPT.
Built-in test code imposes a performance and space cost, and may increase the attack surface. The No Code Left Behind pattern obviates these concerns for built-in test code.
Invariantfunction to each class
InvariantCheckto evaluate class invariant or other environment condition
InvariantCheckfunction is controlled with global settings
constand inline idiom leaves no object code in production – no bloat, performance cost, or attack surface
For details, see Jeff Hurst and Robert Willhoft, “The Use of Coherence Checking for Testing Object Oriented Code,” IBM International Conference on Object Technology, San Francisco, CA, June 10-15, 1995.
Although class hierarchies and related features (polymorphism) provide much of the power of object-oriented programming, they also create many opportunities for subtle errors. The Percolation pattern is a strategy that adds self-checking code to detect and prevent these kinds of bugs.
John Lakos’ Large Scale C++ Design provides an excellent discussion.
Next up – Black Box Testability.