Robert V. Binder

How to Ice the Testing BackBlob

Poster, "The Blob"I recently presented a talk about the testing “BackBlob” and how to use model-based testing to deal with that. 

The Blob is a science-fiction B movie released in 1958. A mass of mobile goo (the Blob) arrives from outer space, grows exponentially, and consumes people and buildings in a terrified California town.

If you’re not familiar with Agile jargon, “Back Log” refers to work that remains to be done. “Back Blob” is a not too subtle play on this concept.

The “given-when-then” (GTW) narratives used in Behavior Driven development (BDD) and Acceptance Test Driven Development (ATDD) are single slices of an astronomically large input and state space.  Tools like JBehave and SpecFlow help to structure this text, associate it with hand-crafted test cases, and then track testing status.  However, they do not address the fundamental problem of adequate testing – getting a meaningful exploration of a system under test. 

Worse, I think they lead to a false sense of completeness, both as specifications and as tests. Although producing GWTs is a big step up from undocumented tribal knowledge, it is nowhere near a complete specification and even further from an adequate test suite.

But, even if a BDD collection was complete, it has sustainability limitations: to retain value as a system evolves, it must be maintained and then re-run. Test automaton helps with the re-run part, but significant work is still needed for maintenance, in addition to the work needed to develop the next incremental feature set. This is true of any hard-coded collection of non-parameterized requirements and test cases. Over time, as a BDD/test code base grows, it either crowds out development or, typically, is simply ignored.  Either is high-risk. That’s how Agile teams get eaten by the testing BackBlob.

With model-based testing, one maintains test models – not test cases – then generates executable test code.  As existing features change and new features are added, a test model is updated accordingly. Previously generated tests and test code can be discarded with no maintenance; a fresh test suite is automatically regenerated.  The size/complexity of test models grows much more slowly than the actual test suites/cases. That’s why updating a test model is much less effort than attempting to maintain a growing collection of test instances. Because models are simpler, the chance of update errors and omissions is reduced. Most importantly, model maintenance provides early validation that can reveal bugs and adverse feature-interactions that regression tests based on stale requirements will let escape.

The talk includes a demo of SpecExplorer.  I don’t discuss its theory, how to model, or its exotic features, but I do show the basic steps needed to do MBT with SpecExplorer.

One of the developer/testers in the audience tweeted about the talk, with a hash tag #MoreModelsLessTests using a catch phrase I tossed out.  This almost says it all, except that with models, we can automatically produce many, many more tests, but drive the cost of maintaining the tests to very near zero.

A video of the talk is on YouTube http://ow.ly/qMkb4.

The slides are on http://www.slideshare.net/robertvbinder/taking-bddtothenextlevel

The YouTube video couldn’t include the video clips from “The Blob” (1958) that I used to illustrate the concepts and have a little fun, so here they are

In the movie, the Blob is finally beaten back (spoiler alert) by freezing it – that’s what I think model-based testing can do for the testing backblob.



4 Comments


  1. Bob, nice presentation. Three comments:

    1) I have been using the following template for functional requirements for quite a few years now: If trigger(s) while precondition(s), then the system shall perform required responses (behavior) and establish required postconditions. For quality requirements, I have been using a similar template: If trigger(s) while precondition(s), then the system shall ensure quallity criterion(a) with threshold(s). Naturally, neither of these formats works for data and interface requirements as well as architecture, design, implementation, and configuration constraints. From this, it is clear that your GWT template works for requirements as well as test models, is incomplete, and also limited in scope to functional (behavioral) requirements.
    2) I agree that the approach does help with automatable tests, but many tests (e.g., penetration tests, usability tests) are not automatable.
    3) I really like model driven testing (MDT) for the reasons you give, but MDT is actually only a natural part of model driven development which also includes model driven requirements, architecture, design, and implementation.

    P.S Appreciate review of my latest book. It will be in the publisher’s warehouse on the 20th.

    • Hi Don

      Thanks for these observations.

      To be clear, I did not develop the GWT template and I am not advocating for it. It comes out of the Agile community where it has widespread use. It has some resemblance to design-by-contract, but its statements can be any kind of natural language so they are prone to ambiguity, contradiction, fragmentation, partiality, etc. I have yet to see any BDD “specification” that provides a useful representation of the allowable and unallowable ordering of GWT scenarios. Further, GWT scenarios typically define *single* instances of an input and output – in effect *one* test case. Its proponents call this “specification by example.” This perpetuates the fantasy that one can achieve a general specification though a few examples – a naïve notion advocated ad nauseam by the adherents of test driven development (TDD) and adhered to with great relief by lazy programmers everywhere. It seems they’ve learned nothing from the first 50 years of software engineering.

      As you note, the GWT template has no vocabulary for “non-functional” aspects, although I suppose one could back into that with some creative phrasing. Of course, many important system characteristics are not amenable to automated verification – all the more reason to automate what you can so that humans can spend their time doing what they’re well-suited to do.

      The useful intersection of model-based testing and model-driven development is quite interesting. I think they are complementary, but MBT can be used to good effect when there are no model-based antecedents.

  2. To the graph you presented in your Cerner video describing Agile backlog(backblob) growth…

    You may face the argument that if the ‘definition of Done’ is managed sprint-to-sprint — such a backlog could be avoided… i.e., using Rally as a test management system…

    What comeback would you have to this argument…?

    • Hi Rick – great question.

      The BackBlob refers to the increasing work needed to maintain and execute regression test suites.

      If this work is not included in the scope of “Done,” that doesn’t change the fact that test suite maintenance and regression testing has been skipped or shorted.

      There is no free lunch. A team has to allocate its resources to new development/testing and to test maintenance and regression testing. Assuming a fixed budget (headcount) for each sprint, the effort needed to do this will consume a larger proportion of that budget with each successive iteration. If test maintenance and regression testing is skipped or shorted, the risk of regression and feature-interaction bugs increases with each subsequent release. Typically, new feature development is preferred over test maintenance, so even if the overall budget is increasing, testing typically receives a lesser share.

      I don’t see how any test management tool can change this basic relationship, other than marginally reducing the cost of maintenance through better artifact management.

Trackbacks

  1. How Technical Debt turns into Technical Bankruptcy

Leave a Reply

Comment moderation is enabled, no need to resubmit any comments posted.