Robert V. Binder

Testability Part 4: White Box Strategies

March 8, 2012  |  Blog, Software Testing, Testability

call-treeThis post covers part four of my 2010 talk on testability. White box testability refers to specific programming practices and components that can improve or hinder testability.

Since Dykstra’s 1968 note “Go To Considered Harmful,” practices for producing clean, well-structured, readable, and maintainable software have been the subject of extensive discussion, research, argument, and use. The nearby image shows the result of a failure to apply these practices, as generated by Peter Ljubič’s analyzer. Imagine trying to devise tests to cover all these relationships.

I won’t try to recite generally accepted good programming practices here. All of them improve testability for code in any language running on any platform. Instead, a sketch follows of some specific programming techniques and built-in-test components that can improve testability.

  • Abstraction for Points of Control and Observation (PCOs)
  • State test helpers
  • No Code Left Behind pattern
  • Percolation pattern
  • Avoid ugly dependencies

Abstracting Points of Control and Observation

TTCN is a language for defining test cases. One its key concepts is the Point of Control or Observation (PCO).

A PCO is an interface of the system under test. Usually, we don’t want an automated test suite directly coupled to PCOs. Instead, it is better use abstract PCOs so that parts of the SUT interface not necessary for testing or that would result in brittle dependencies are hidden inside another piece of software. This is usually a wrapper class whose public methods provide well-structured functions to control and observe the actual SUT interfaces.  They may include some test helpers.

Abstract PCOs can facilitate testing multiple configurations.  The test suite uses the same interface, regardless of which configuration is being tested.

Like all good software abstraction, a good PCO has two aspects: the conceptual and the physical.  We want our abstractions to have a coherent focus (conceptual) and through the magic of indirection to decouple the physical SUT interface from the test cases that drive (control) it  and evaluate (observe) test results.

Control Scope

What does a tester/test harness have to do activate SUT components and aspects?  Everything you (or your test drivers) touch is a PCO.   Also, don’t forget the SUT environment. With present-day technology stacks, this can be very complex.

A component or an interface is a result of the way the system was built.   An aspect is a feature, function, or behavior of a system that is performed by two or more components.  This It is usually much easier to drive components.  However aspects more interesting, but typically not directly controllable.

Observation Scope

  • What does a tester/test harness have to do inspect SUT state or traces to evaluate a test?
  • Traces easier, but often not sufficient or noisy
  • Embedded state observers effective, but expensive or polluting
  • Aspects often critical, but typically not directly observable

State Test Helpers

State-based testing is needed when the SUT can respond differently to the same inputs, depending on what has already happened. For example,  you must logon before you can logoff. A data item must be created before it can be changed or deleted. Once deleted, it cannot be changed, and so on.

State-based testing strategies aim to cover enough of the possible sequences to reveal bugs related to state-based behavior. However, the “state” that determines behavior might not be directly accessible in a class interface. Two kinds of functions help when this the case: get state and set state.  It is useful to have a method for each state.  For example, after deleting the last item in a collection, we’d expect that the method is_empty() returns true.

Suppose a collection class can hold N items. To test its behavior when full or near-full, we’d have to first do N calls to add(x). If N is large, this can be time-consuming and wasteful. This problem is obviated if we add a test method force_full() to get to this state and one to force_empty() so that we don’t have call delete(x) N times to get back to the empty state.

State-based testing is dicussed extenstively in TOOSMPT.

No Code Left Behind

Built-in test code imposes a performance and space cost, and may increase the attack surface. The No Code Left Behind pattern obviates these concerns for built-in test code.

Forces

  • How to have extensive BIT without code bloat?
  • How to have consistent, controllable logging?
  • How to control frequency of evaluation
  • How to set options for error handling and detection
  • How to define once and reuse globally?

Solution

  • Add Invariant function to each class
  • Invariant calls InvariantCheck to evaluate class invariant or other environment condition
  • InvariantCheck function is controlled with global settings
  • Allows selective run time evaluation to minimize performance cost in sensitive code
  • const and inline idiom leaves no object code in production – no bloat, performance cost, or attack surface

For details, see Jeff Hurst and Robert Willhoft, “The Use of Coherence Checking for Testing Object Oriented Code,” IBM International Conference on Object Technology, San Francisco, CA, June 10-15, 1995.

Percolation Pattern

Although class hierarchies and related features (polymorphism) provide much of the power of object-oriented programming, they also create many opportunities for subtle errors.  The Percolation pattern is a strategy that adds self-checking code to detect and prevent these kinds of bugs.

  • Enforces Liskov Substitutability.
  • Implement with No Code Left Behind.
  • Built in to Eiffel and later versions of Java.
  • May be implemented with programmer added-code in C++ and other languages.

See the Percolation Pattern for details.

Avoid Ugly Dependencies

  • No cyclic dependencies
  • Don’t allow build dependencies to leak over structural and functional boundaries (levelization)
  • Partition classes and packages to minimize interface complexity

John Lakos’ Large Scale C++ Design provides an excellent discussion.

Next up – Black Box Testability.

 



Leave a Reply

Comment moderation is enabled, no need to resubmit any comments posted.