Panel Discussion on Open Source Testing Tools

Three chimps - hear no evil, see no evil, speak no evilA few days ago, I participated in a panel discussion on open source tools for testing at the QUEST conference with the two founders of Selenium: Jason Huggins of Sauce Labs and Simon Stewart of Google.

Before the panel started we chatted a bit with the moderator. We couldn’t come up with any contentious issues, so we told the moderator to be prepared with questions. As it turned out, we had different views and thereby avoided the dreaded panel of drinking birds. ;-)

I’d decided to talk about my inventory of open source model-based testing tools and experience in building a software product using about a dozen open source systems. I started with an overview listing a few mature open source MBT tools users might find useful (the slides are here.)  I concluded with the lesson I learned after three years of using a dozen open source tools and embedding the open source Tcl codebase: they’re free, but they’re not cheap.

Jason followed with a quick overview of Selenium, relating how it all started here in Chicago when he needed a better way to enter time reports. Selenium is a system that  automates interaction with a browser.  So, it can be used to test web content and features in any browser that supports Javascript, which nearly all do.

Jason briefly described how the Selenium maintainers are adding support for mobile platform browsers (importantly Android and iOS).

Audience questions then drove the discussion.  The following isn’t verbatim — just my best recollection.  Please post a comment with corrections/additions, if you were there.

 

Q: Do testers need to be more technical to use open source testing tools?

Simon: Simple manual testing is increasingly done by the lowest-cost provider, so if that’s all you (QA professionals and testers) can do, you probably won’t be doing it for long.  However, if you can learn how to develop Selenium scripts with Javascript programming, you’ll be providing higher value that isn’t as easily outsourced.

Jason: You should structure your test suites with action keywords (e.g., “Delete item from cart”) rather than the individual keystrokes and clicks used for an action.  This is also a good idea because there are often several ways to do an action.

Bob: It depends on the tool. There are 493 open source testing products listed at http://www.opensourcetesting.org/.  Most require at least some basic programming skill to use.  Also, some tools are not well documented, so you have to learn by experimentation.

Simon:  Many tools, including Selenium have very good documentation.

Bob: That’s certainly true for the popular tools. Also, there are often active and useful user forums where beginners and experts can get help.

 

Q: What are the advantages, myths and shortcomings of open source?

Simon:  Because you have access to the codebase, you can have higher confidence that it doesn’t have security problems, for example. We inspect every line of open source code we use at Google.

Bob: Are you saying you’ve inspected every line of development environment tools like Visual Studio or Eclipse  — each over 10 million lines of code?

Simon: We’ve inspected every line of open source code that is integrated with Google production code.

Simon: Besides being able to inspect every line of an open source codebase, you can change it if it doesn’t do what you want.

Bob: While that’s true in theory, most open source users are not willing or able to do that.  Modifying any complex tool means you have to understand its codebase pretty well. And, unless you submit changes as contributions, you’re entirely on your own — potentially a worse situation than being dependent on a commercial provider for fixes.  So, if you’re not willing to invest mindshare in learning a codebase so that you can change it without breaking it, the flexibility of an open source codebase is not much different than a commercial system.

Simon: Having active and open development participation is vital for the open source model to work – it is important to give back.  But, even with a large user base, only a small fraction of open source users are typically active codebase contributors/maintainers.

Bob: Open source systems with a large and active community (like Selenium) are often very useful and reliable. Of course, some commercial products/libraries are very buggy, and provide abysmal support.

 

Q: How does Selenium compare with commercial testing tools like QTP and Quality Center?

Jason/Simon: Selenium is all you need.

Bob: That’s true if you only test browser-based apps. Of course, this excludes the very large universe of “native” apps that use GUIs like Dot Net, Java, or the native GUI frameworks on Android, iOS, Windows Mobile.

Simon:  There are about four times as many job ads calling for Selenium experience than QTP.

Jason: Selenium is usually used as part of a QA tool chain, including other open source and commercial tools.

 

Q: Can you use open source for regulated products (e.g., FDA)?

Jason: Selenium wasn’t designed for safety-critical applications, so you shouldn’t rely on Selenium for safety critical applications.

Bob: For software in FDA regulated products, using an open source codebase or tool isn’t much different than using software with a non-open license.  There are two possible uses: as a development environment tool, which would require validation per 21 C.F.R. §820.70(i) as does any other automated tool used to produce FDA regulated products.  If you integrate an open source codebase as a component part of a delivered system, it would be subject to the same V&V that your Quality Management System requires for any other integrated component, produced in-house or by a third party.

 

Q: Is Selenium useful for Agile development?

All: Yes.

 

Q: Any final thoughts?

Bob: Learning model-based testing actually would provide much greater job security than learning to program test scripts, because programming can be outsourced just as easily as manual testing. Model-based testing requires analytical thinking and app understanding on a par with a system architect.  For many reasons, this kind of high-value skill is both hard to outsource and advantageous to keep in-house.

Members of the QA community can get a lot more respect using model-based testing than learning the details of Selenium, QTP, or any approach that relies mostly on programming skills.

For an interesting video that shows how MBT can generate a Selenium test suite to test the Sugar CRM system take a look at http://www.youtube.com/watch?v=LVGB88VXmf4

 

Posted in Blog, FDA, Software Products, Software Testing, Technology | Tagged , , , , , , , , , , , | Leave a comment

Technical Equity or Technical Debt: Stay Fit or Get Flabby

T Rex chasing runnerTechnical debt refers to aspects of a codebase are incomplete, deficient, obsolete, or buggy. This can occur for many reasons: insufficient time, uncertainty, omissions, poor workmanship, or poor management. This is termed “debt” because it will take additional time and money to correct, update, or revise.

Technical equity refers to aspects of a codebase that facilitate its development, testing, deployment, and maintenance. I call these aspects “equity” because they reduce the time and cost for all kinds of work on a codebase.  It is not the absence of technical debt: a codebase can easily have both.

The effects of software development practices on a codebase are like the effects of lifestyle on your health and longevity.

  • Accumulating technical equity is like getting regular exercise and following a good diet. Everyday life is better – you’re alert, relaxed, and sleep well. You have reserves of energy for regular work and short-term stress. Serious illness is prevented or minimized.
  • Accumulating technical debt is like leading a sedentary life fueled by junk food. Everyday life is diminished – you feel lethargic.  Any exertion outside of your routine leaves you sweaty and gasping for breath. Serious illness generally happens sooner and is worse when it does.

The following practices build technical equity. They are proven, platform-agnostic, and process-agnostic.  The links about the practices are just a taste — all have extensive liturature and many tools.

These practices prevent technical debt and establish a technical equity foundation that pays dividends over the life of your codebase.

Posted in Blog, Software Products, Technology | Tagged , , , , | Leave a comment

Open Source Tools for Model-Based Testing

A neon OPEN sign glowing red in the window of a restaurantI discussed the following inventory of open source and free model-based testing tools in a recent QUEST panel session.   

After checking each tool, it is clear there’s wide variation in maturity,  stability, and provisioning.  I see five groups.

 

  • Binaries: an installable available without source code.
  • Fielded: a codebase that has an active user community using the tool for non-research work, at arms-length with the developers.
  • Club: a codebase that has a small user community using it for non-research work.
  • Beta: a codebase that has been recently released and has limited use.
  • Development: a codebase that is incomplete or used primarily by its developers.

Nearly all of it has been taken from the provider’s web sites – YMMV. The “Stack” column indicates the source code language and/or run-time environment(s); “*nix” indicates a tool written to run on most Linux/Unix variants. See end for change list.

Binaries

ToolDescriptionStackLicense
JUMBL 5.0 The Java Usage Model Builder Library (JUMBL) supports automated, model-based, statistical testing of systems. Includes command-line and GUI interfaces. Java. Provided as Jar file (byte code). Windows and *nix.University of Tennesse
Spec Explorer 3.5A Visual Studio add-in with support for modeling software behavior, graphical visualization of model explorations, and model checking. Users write model programs in C# and define usage scenarios. The model programs can use a built-in combinational (e.g. pairwise) data generation feature. Generates ready to run test code from model programs. NModel was a prototype.C#, Visual Studio. Windows. Visual Studio License
Torx 3.10.5k Supports conformance testing of reactive software; checks the correct behaviour of a real implementation during its execution based on its formal specification. Can execute a Torx specification as SUT. *nix, maybe Windows+CygwinApache 2.0
Uppaal TRON 1.5 An integrated tool environment for modeling, validation and verification of real-time systems modeled as networks of timed automata, extended with data types (bounded integers, arrays, etc.) *nix, maybe Windows+CygwinUppaal license. Non-commercial use only.

Fielded Codebases

ToolDescriptionStackLicense
NModel 1.0.21029.0A model-based testing and analysis framework. User writes model programs in C#. NModel generates tests from your model program in advance or on-the-fly as the test run executes. Users may add custom test generation strategies in C# to achieve any coverage criteria. SUT not limited to Windows.C#. Requires .NET, but not Visual Studio. Microsoft Open Source License
TopCased 5.1Model-driven development and testing environment for critical hw/sw embedded systems; Uses formal methods, SysML, UML, OCL. Java/EclipseEclipse Public

Club Codebases

ToolDescriptionStackLicense
Graphwalker 2.5.16Generates offline and online test sequences from state machines modeled with GraphML -- no UML to learn. User selects any of seven built-in coverage (stopping) criteria for test generation. Tool may be integrated with a Java test harness or called with SOAP as web service from the developer's server.JavaMIT
IFX Omega 2.0A compiler/simulator/model-checker for a rich subset of UML 2.2, SysML 1.1, based on the VERIMAG IF model checker. May be used to validate and verify asynchronous timed system models with complex dynamics. Uses rigorous semantics suited for critical real-time embedded systems. Java/EclipseNot Specified
ModelJunit 2.0Adds model-based testing to Junit. Models are written as Java classes. Uses the annotations feature of Java 5.0. Automatically generates Junit test suites from these models.Java, JDK 1.6 or higher.GPL

Beta Codebases

ToolDescriptionStackLicense
ATS4 AppModel Supports app specification, model based testing, and test script generation. Simple interface to manage complex models. Developed at Nokia for handheld testing.JavaBSD
AutoFocus 0.3.0Supports modeling and analyzing the structure and behavior of distributed, reactive, and timed computer-based systems.Java/EclipseApache 2.0
Fastest 1.3.6Generates tests from models expressed in the Z notation, following the Test Template Framework proposed by Phil Stocks and David Carrington. Supports unit testing.JavaNone specified
GUITAR 1.0 Automatically extracts structure and widget APIs from Java Foundation Classes (JFC) and web GUIs. Generates an Event-Flow Graph (test model), generates test cases from the model, evaluates tests with built-in oracle. JavaNone specified
JTorx 1.10.0Generates tests to check that the ioco testing relation holds between a labelled transition system expressed in Aldebaran or GraphML and an IUT. Derived from Torx. JavaBSD
MISTA 1.0Uses lightweight high-level Petri nets as a visual modeling notation.Test models can be animated and verified.Generates executable test code fom a test model in Java, C, C++, C#, VB, or HTML and/or test engine JUnit, NUnit, Selenium IDE1, or Robot Framework. Several test generation strategies are supported. JavaNone specified
TEMA 3.2.1A prototype tool and methodology for online test generation for smartphone GUI testing. Based on the GUI keywords/action word testing strategy. JavaMIT

Development Codebases

ToolDescriptionStackLicense
EvoMTA model based software test utility using genetic algorithms to generate test oracles.JavaNone specified
NModel Remote Stepper 1.0 Extensions to NModel enabling model-based testing of systems implemented in Java or other languages. C#, JavaApache 2.0
OSMO Tester 2.1.1Open Source Modelling Objects (OSMO) includes tools and libraries to model software behaviour with extended state machines in Java for online/offline test generation and non-deterministic online models. JavaLGPL
Parteg 1.3.3 Partition Test Generator: generates tests meeting control flow-based coverage criteria (e.g., MC/DC) and boundary-based coverage criteria. Java EclipseEclipse Public

 

Revision History

DateRevision
31 May 2012Added Fastest.
25 April 2012Formatting, clarifications.
20 April 2012Added GUITAR.
15 April 2012Initial release.

The WordPress plugin WebSimon generated the above tables — easy to use, looks great.

 

 

Posted in Blog, Software Products, Software Testing, Technology | Tagged , | 5 Comments

Technical Equity

A pile of nice shiny gold barsTechnical Equity is the value that accrues when a software system is well-formed.  Instead of burdening you with unnecessary excess cost, your codebase works for you. Technical equity pays dividends: you avoid wasted effort and the consequences of buggy releases, and gain the advantage of releasing sooner and/or with more features, for the same cost.  It is the opposite of technical debt.  It is manifested in better than average productivity and lower than average bug escapes.  Instead of code that bleeds you dry, you create an asset that produces value.

The concept of technical debt has certainly gotten a lot of mindshare.  But even if your codebase is debt-free, that’s not enough. We should seek to maximize technical equity.

While it takes focus and skill, it often takes less work (cost) to produce technical equity than to go into technical debt.  The value-building practices are nothing new or secret.

  • Build the right “it:” know the user’s problem and engineer requirements accordingly
  • Design a coherent and rationalized system architecture to solve the real problem
  • Consistently produce well-structured code: high cohesion, low coupling, absence of code smells, minimal complexity
  • High code readability and documentation of intent
  • Clean out dumb bugs with routine use of static analyzers
  • Consistently produce robust code that handles errors, bad input, and exceptions correctly
  • Consistently, frequently, and automatically test code to achieve both high black box and white box coverage
  • Scope increments, iterations, or sprints to manage risk

The secrets of technical equity are not secrets — they’ve been advocated for over thirty years by thousands of well-known and anonymous software engineers and their management.  They do not require exotic or arcane knowledge. They do not require a huge and expensive investment in tools or training (some is necessary.)  But they do take clarity of purpose and sustained, disciplined focus.

So instead of trying to avoid technical debt, try to build technical equity.  Accentuate the positive.

Model-based testing is a significant new tool for creating technical equity — more about that in my next post.

 

Posted in Blog, Business, Software Products, Technology | Tagged , , , , , , , , | Leave a comment

How Technical Debt turns into Technical Bankruptcy

Black Hole Technical Bankruptcy occurs when technical debt overwhelms a software system. I’ve previously blogged about a case study – how the accumulation of poor development practices resulted in the business failure of highly successful Enterprise IT software company.

The technical debt metaphor provides a nice handle for a software development  trade-off that’s been present in every project I’ve ever done- as a manager, architect, programmer, or tester. Ward Cunningham has done us all a favor in creating a memorable tag line. And with this popularity, everyone is of course against it. But how many really understand what drives technical debt? And that is it a symptom — not a root cause?

Fred Brooks Effort MatrixAs Fred Brooks learned in the 1960s, the true difference in effort between developing a program (which is how most developers see what they’re doing) and programming systems product is 10 times (Brooks said 9x – I’m rounding up a little.) The drivers are the additional complexity of system of programs and features for sustaining a product.

This is true regardless of scope: number of function/feature points, use cases, user stories, etc.  Even this is under-stated: the number of interfaces increases exponentially with the number of components (programs.)  That is,  a system of 1000 components isn’t necessarily 10X to system of 100 components  — it is often 20x or 30x. The context of usage matters in ways that are hard to appreciate unless you’ve had to live with the long-term effects of sustaining a system of programs for a user community (a product) over many releases.

Yet hope springs eternal in software development. Developers and sponsors of software projects often don’t understand this disparity, so expectations and commitments are made assuming programs instead of a systems product. Developers, managers, and sponsors all underestimate work. Then Bad Things happen. Even if a project is de-scoped and time-boxed, necessary work is skipped – that’s where technical debt starts. Then schedules slip, bugs escape, and revenues don’t come.

This explains why about 3 out of 10 IT projects are canceled.  Technical debt is symptom of underlying problems — not a cause.

If the project isn’t cancelled after an initial confrontation with reality, attitudes often harden and people begin to game each other. This usually results in more technical debt piled on top of the first tranche. That’s how technical debt gets compounded. Left unchecked, this leads to technical bankruptcy.

This vicious cycle can be broken. Preventative principles were articulated in the 1970s and have been rediscovered about every ten years since (just as each generation rediscovers sex, as friend and mentor Boris Beizer often observed.) Hundreds of software engineering and software process strategies have been worked out and applied with success to many thousands of systems. They all share certain strategies:

Moreover, problems worth solving tend to be big and complex, so some kind of step-wise, iterative, incremental, or agile process is necessary.

Model-based testing is a relative newcomer to the mix. It can and has prevented much technical debt and therefore substantially reduces the risk of technical bankruptcy. I explain how in Model-based Testing builds Technical Equity.

 

Posted in Business, Software Products, Software Testing, Technology | Tagged , , , , , | Leave a comment

Britannica Brat

I’ve been reflecting on the recent announcement that Encyclopedia Britannica (EB) will no longer publish in print. Subscription to its web site is now the only offered media.

encyclopedia britannica full set

My full set of the 15th edition (1974) rests on the lower two shelves of a bookcase in the room where I’m writing this — 4 feet 7 inches first to last, leather bound. EB has had more of an influence on me than just generating complaints about all those **** heavy books on moving day.

My stepfather was a Principal Editor of the radically new 15th edition. This was the last chapter of a long-standing relationship he had with the University of Chicago, Mortimer Adler, and EB. It started in Chicago’s Hyde Park neighborhood, then to Lake Forest, San Francisco, and then back to Chicago. It spanned my life from kindergarten to college.

The 15th edition was a very big deal, because it realized Adler’s project to create and instantiate a systematic framework for all knowledge. The plan had three levels as indicated by the cover colors in the nearby image: an overview and correlation of big ideas (single volume), short (750 words or less) survey articles (ten volumes), then in-depth articles on selected subjects (nineteen volumes). These articles were written and revised by acknowledged experts in their fields and extensively reviewed and checked before publication. And long before there was hypertext, all of this was carefully cross-referenced. The physical and intellectual heft of this set of tomes is impressive.

Given the popularity of Wikipedia and crowd-sourcing in general, the recent announcement made me wonder how much traction EB’s expert and authoritative content would have in cyberspace. Today, I was relieved to read Gordon Krovits’ discussion of EB’s new media and the rise of Wikipedia. EB is still selling well: “Despite Wikipedia, more people pay to access the encyclopedia website annually than paid for the print edition in any year.”

For an unrelated project, I recently reviewed some preliminary research developed to provide content  for a history of technology as a ten episode TV series. To boot up a timeline database of events, about eighty articles from Wikipedia were parsed. Along with some other sources, this yielded about 2,700 significant events (X was invented in 1934, when …)

As part of my review, I checked many of referenced Wikipedia articles. They seemed generally accurate as far as they went. I sampled several and they mostly held up to scrutiny. However, what jumped out after pulling back from the details was not what was there, but what was not. Many key events and subjects were simply missing from this conceptual mashup. Trivialities had the same standing as game-changers. I didn’t see this as a big problem for the series, as the constraints of a ten-hour video format means that most details will be have to be rolled up. Nevertheless, it drove home a significant limitation of Wikipedia.

Wikipedia cannot be relied on as a comprehensive source – there are often significant gaps, owing to the typically parochial interests of its contributors. In contrast, EB’s articles are expected to be comprehensive and accurate, regardless of the author’s interests. As Krovits reports, “In 2008, [EB] company president Jorge Cauz told the New Yorker, “Wikipedia is to Britannica as ‘American Idol’ is to the Juilliard School.” (This quote appears in the Wikipedia entry on Mr. Cauz.)”

There may not be (m)any more Britannica brats, but I certainly hope there will always be a Britannica.

 

 

 

Posted in Blog, Business, Technology | Tagged , , , , | Leave a comment

The Spreadsheet Experiment

Having spent manmad scientist cartoony years on the bleeding edge of test automation and model-based testing, I recently tried an experiment to see to what extent Excel could support a test plan for manual testing of mobile apps.

In a earlier post, I explained some of the thinking behind my new course “How to Test Mobile Apps.” Most mobile app devs don’t have any testing tools and do only limited manual testing. With this audience in mind, I used Excel to develop a complete test plan template, following IEEE 829. Excel is not limited to MS Office — you can use Open Office or Google Docs to develop and share spreadsheets. The course case study includes a complete example test plan in Excel, which I used to refine the approach.

The test plan follows my test model: tours, use cases, variants, actions, and variables. Worksheets (tabs) are basic grouping structure for use case test instances, with rows as test actions. It includes tours, which are sequential overlays on the use case test instances.

Following the incremental approach I teach in the course, adding new test instances is easy — you just make a new copy of the baseline sheet and tweak as necessary.

Then the completed plan file is copied, producing a file for each test run. The tester enters verdicts, etc. into the test run file. This had the unexpected good side effect of providing an exact snapshot of the test plan and the test run — subsequent updates to the plan didn’t wipe out the plan used at the time.

In contrast to a good test suite management tool, the burden of consistency, naming, and indexing is entirely on the user. Excel isn’t designed to support a large number of sheets with inter-sheet dependencies — you need a database and UI for that. And, using a spreadsheet (or other kind of flat file) is bad idea for more sophisticated testing: maintenance gets messy, no integration with requirements, bug tracking, or project management; it can’t scale; shared access/update is not supported (if files are housed in Sharepoint, you get some support for shared, multi-user access.)

However, for testers who are willing and able to follow the structure and maintain it, this is a practical, complete, and no-cost solution.

The template is available with the course:  http://www.udemy.com/how-to-test-mobile-apps/

Posted in Blog, Software Testing, Technology | Tagged , , , , , , | Leave a comment

Testability Part 4: White Box Strategies

Spaghetti code visualizationThis post covers part four of my 2010 talk on testability. White box testability refers to specific programming practices and components that can improve or hinder testability.

Since Dykstra’s 1968 note “Go To Considered Harmful,” practices for producing clean, well-structured, readable, and maintainable software have been the subject of extensive discussion, research, argument, and use. The nearby image shows the result of a failure to apply these practices, as generated by Peter Ljubič’s analyzer. Imagine trying to devise tests to cover all these relationships.

I won’t try to recite generally accepted good programming practices here. All of them improve testability for code in any language running on any platform. Instead, a sketch follows of some specific programming techniques and built-in-test components that can improve testability.

  • Abstraction for Points of Control and Observation (PCOs)
  • State test helpers
  • No Code Left Behind pattern
  • Percolation pattern
  • Avoid ugly dependencies

Abstracting Points of Control and Observation

TTCN is a language for defining test cases. One its key concepts is the Point of Control or Observation (PCO).

A PCO is an interface of the system under test. Usually, we don’t want an automated test suite directly coupled to PCOs. Instead, it is better use abstract PCOs so that parts of the SUT interface not necessary for testing or that would result in brittle dependencies are hidden inside another piece of software. This is usually a wrapper class whose public methods provide well-structured functions to control and observe the actual SUT interfaces.  They may include some test helpers.

Abstract PCOs can facilitate testing multiple configurations.  The test suite uses the same interface, regardless of which configuration is being tested.

Like all good software abstraction, a good PCO has two aspects: the conceptual and the physical.  We want our abstractions to have a coherent focus (conceptual) and through the magic of indirection to decouple the physical SUT interface from the test cases that drive (control) it  and evaluate (observe) test results.

Control Scope

What does a tester/test harness have to do activate SUT components and aspects?  Everything you (or your test drivers) touch is a PCO.   Also, don’t forget the SUT environment. With present-day technology stacks, this can be very complex.

A component or an interface is a result of the way the system was built.   An aspect is a feature, function, or behavior of a system that is performed by two or more components.  This It is usually much easier to drive components.  However aspects more interesting, but typically not directly controllable.

Observation Scope

  • What does a tester/test harness have to do inspect SUT state or traces to evaluate a test?
  • Traces easier, but often not sufficient or noisy
  • Embedded state observers effective, but expensive or polluting
  • Aspects often critical, but typically not directly observable

State Test Helpers

State-based testing is needed when the SUT can respond differently to the same inputs, depending on what has already happened. For example,  you must logon before you can logoff. A data item must be created before it can be changed or deleted. Once deleted, it cannot be changed, and so on.

State-based testing strategies aim to cover enough of the possible sequences to reveal bugs related to state-based behavior. However, the “state” that determines behavior might not be directly accessible in a class interface. Two kinds of functions help when this the case: get state and set state.  It is useful to have a method for each state.  For example, after deleting the last item in a collection, we’d expect that the method is_empty() returns true.

Suppose a collection class can hold N items. To test its behavior when full or near-full, we’d have to first do N calls to add(x). If N is large, this can be time-consuming and wasteful. This problem is obviated if we add a test method force_full() to get to this state and one to force_empty() so that we don’t have call delete(x) N times to get back to the empty state.

State-based testing is dicussed extenstively in TOOSMPT.

No Code Left Behind

Built-in test code imposes a performance and space cost, and may increase the attack surface. The No Code Left Behind pattern obviates these concerns for built-in test code.

Forces

  • How to have extensive BIT without code bloat?
  • How to have consistent, controllable logging?
  • How to control frequency of evaluation
  • How to set options for error handling and detection
  • How to define once and reuse globally?

Solution

  • Add Invariant function to each class
  • Invariant calls InvariantCheck to evaluate class invariant or other environment condition
  • InvariantCheck function is controlled with global settings
  • Allows selective run time evaluation to minimize performance cost in sensitive code
  • const and inline idiom leaves no object code in production – no bloat, performance cost, or attack surface

For details, see Jeff Hurst and Robert Willhoft, “The Use of Coherence Checking for Testing Object Oriented Code,” IBM International Conference on Object Technology, San Francisco, CA, June 10-15, 1995.

Percolation Pattern

Although class hierarchies and related features (polymorphism) provide much of the power of object-oriented programming, they also create many opportunities for subtle errors.  The Percolation pattern is a strategy that adds self-checking code to detect and prevent these kinds of bugs.

  • Enforces Liskov Substitutability.
  • Implement with No Code Left Behind.
  • Built in to Eiffel and later versions of Java.
  • May be implemented with programmer added-code in C++ and other languages.

See the Percolation Pattern for details.

Avoid Ugly Dependencies

  • No cyclic dependencies
  • Don’t allow build dependencies to leak over structural and functional boundaries (levelization)
  • Partition classes and packages to minimize interface complexity

John Lakos’ Large Scale C++ Design provides an excellent discussion.

Next up – Black Box Testability.

 

Posted in Blog, Software Testing, Technology | Tagged , , , , , , , , , , , , | Leave a comment

What I Learned Building a Software Product with Tcl

Toy Plastic HammerThrough Google Circles, I happened to see David Welton’s very interesting reflection on the Tcl programming language (posted in 2010):

http://journal.dedasys.com/2010/03/30/where-tcl-and-tk-went-wrong/

About ten years ago, I chose to develop a commercial automated software testing tool with Tcl and Tk. This post explains that decision and its consequences.

Despite testing tools that tout ”visual programming” interfaces, the primary user interaction with a software testing tool is through its test scripting (programming) language. If this isn’t a fully featured programming language, the user cannot create effective, scalable, and maintainable test suites. Many built-in testing functions are also needed. There are essentially two ways to do this: as a framework/library added to an existing language or as elements of a domain-specific language (DSL.)

Being a bootstraping startup, we couldn’t afford to develop yet another test scripting language (a DSL) and thought that would be ridiculous anyway, given the ease of simply adding a framework to an existing language. Also, testing DSLs are well-known lock-ins and will be resisted by any savvy customer — which our target customers certainly were.

The platforms of our target customers were a very mixed bag, so being able to run and/or interface with multiple platforms was important.

We wanted a language that could produce binaries for small embedded processors as well as other kinds, support distributed testing, was easy to learn, and could be easily extended to get into the physical layer of the system under test (this is necessary for test control and observation.)

So which base language to use? Knowing the intense tribalism of languages and platforms, I wanted something that was neutral. If I used Java, I could kiss the Microsoft world goodbye. If I used C#, I could kiss everything else goodbye, and so on.

Tcl was already in use for many kinds of test automation. Its open source development cadre appeared to be stable and competent. It had been incorporated to support scripting in other successful products. It certainly met all of the technical criteria. Tcl also had a very large and attractive library of platform-abstracted features. It could achieve multi-platform coverage out of the box. Cisco had something like 15 MLOC of Tcl running in commercial release in its IOS, so it was not an unproven lab rat.  Lots of good books and training were available and there was an active UseNet discussion group. The BSD license meant we did not have to be married to the doctrinaire impositions of GPL and its assorted poison pills for commercial licensing.

As part of my technical due diligence, I talked to a lot of Tcl users to find out what it was like to live with.  I learned that Tcl quickly and often resulted in a big ball of mud. Why? The perception that “scripts” were throw-aways for which good design was over-engineering, its LISP-ish flexibility, and primitive modularity were the common factors. However, as our app would support test suite management and implement the test object pattern (like Junit), I didn’t see that as a problem. We experimented with Incr Tcl  and found it provided good support for the basic elements of object-oriented programming, following the familiar C++ object idiom. With Incr Tcl, our tool could support test objects, so I concluded we could achieve a scalable and maintainable testing platform with Tcl.

So, we chose Tcl. To make our product platform-agnostic, we then went all-in and developed the product front end in Tk and Tcl. I personally did a lot of Tk and Tcl programming.

Over about four years of this, things went from bad to worse.

The syntactic and conceptual worm holes that give Tcl much of its power are confusing, distracting, and error-prone if you’re not a devotee. As Tcl was a means to an end, not a mission, this was a constant source of vexation for us and our customers. But, you don’t get any sympathy about this from the Tcl community, who take it as a badge of honor (they aren’t much different than other language tribes in this respect.)  The sophisticated support now routinely available in development environments like Eclipse or Visual Studio isn’t available for Tcl – when I first saw automated refactoring and commenting in Visual Studio after years of using a 1990s text editor for Tcl coding, I was depressed.

None the less, I believe the Incr Tcl test object framework I design and programmed remains the best test automation framework ever released. I explained why I think this (and still do) in my 2006 Google Test Automation conference talk: Test Objects: They Just Work.

But, as the core technology of a commercial tool, Tcl was the wrong choice.

In any complex app, sooner or later, you need reliable, efficient, and controllable support for concurrent processing. Ideally, this is provided by your platform and seamlessly integrated into your programming language. It has taken the Java world and Microsoft the better part of fifteen years to get this right. The Tcl support for this was okay for toy applications, but didn’t scale for us. The only alternative was to instantiate multiple interpreter processes, for which we then had to develop an IPC protocol. We did that, but it was ugly and inefficient.

As we pushed the Tcl interpreter to its limits (often related to concurrency), it started to break in ways that cost us sales and customers. Although we were paying a third party for support and made it abundantly clear to them these bugs were an existential problem, all we got were partial answers and some rude innuendo about our intellectual limitations. Our problems simply weren’t of interest to the maintainers. Finally, we were told to fix it ourselves, because after all, the Tcl interpreter is open source. We were maxed out on product development, had no such expertise, and branching the core tree would have been insanity. We were simply stuck and had to devise work-arounds for customers.

The Tcl talent pool is very small. When I tried to hire Tcl devs, instead of getting inundated with resumes, I ended up seeing the same several dozen names. The back end of our system was implemented in C++ and C. We tried to cross-train, but that didn’t work. As a small startup team, this was the source of many, many headaches. Had the product taken off, it would have been a show-stopper.

Finally, the requirement for customers to learn yet another programming language and one they routinely perceived as weird didn’t ever help. Customers didn’t care that it was platform-agnostic: they wanted to use what they knew.

I’ve taken many lessons from all of this. In the main: for a system of any criticality and complexity, a niche language like Tcl is a very bad choice.  I’ll never forget what one VC said when I mentioned that we were using it: “Tcl? Sounds like a bad acid trip.” I nearly boiled over at the time, but in retrospect, he was spot-on.

 

Posted in Blog, Business, Software Products, Technology | Tagged , , , , , , , , , , , , , , | Leave a comment

The Genesis of How to Test Mobile Apps

Quality Checked Handheld

A few months ago, the millionth mobile app was released.

This is is an amazing milestone. It shows how important the mobile space has become and how rapidly it’s evolving.

 

I wondered, have any of these apps been tested? My guess: probably only a few.  So, I know what will happen — once users get their hands on the apps, bugs will start jumping out.  Here’s a great example on You Tube.

I then wondered, how could I help, and fast? What if I could provide a short and easily accessible course that would teach mobile app devs how to do a good job of testing their apps?

The course would have to present a testing approach meeting some basic goals.

  • Any mobile app developer could get up to speed quickly
  • No special tools — fingers and eyeballs only (manual testing)
  • Must target mobile app bugs — not just rehash principles
  • Must be platform-agnostic

Okay, that’s doable. Keep it simple and clean. No tester mumbo-jumbo. Focus on the most likely bug sources in mobile apps.

But, I also know with certainty that effective testing depends on good test models, design, and analysis. Achieving that can involve some esoteric stuff. Could I reduce all this to a practical recipe without loosing the bug-zapping power of advanced techniques? And could I provide all this in a kernel that could scale to complex apps and automated testing?

I didn’t see any reason why not.

It took me a little longer than the long weekend I’d originally planned, but I think How to Test Mobile Apps meets all these design goals. It is hosted on the Udemy.com online education platform, so anyone can take it any where, any time.

http://www.udemy.com/how-to-test-mobile-apps/

The course includes a test plan template based on the IEEE 829 standard for test documentation and specialized for moble apps.

Each “lecture” (Udemy’s content unit) has an exercise.  In these, the user creates a test plan for their own app, step-by-step. By the last lab, they’ll have a complete ready-to-use test plan for their mobile app.

I kept it strictly platform-agnostic, so that the resulting test plan can be re-run to support cross-platform development, and multiple deployment configurations.

Here are the lectures:

  1. Introduction
  2. Why We Test
  3. Model-Driven Testing
  4. Use Cases
  5. Actions and Variables
  6. Use Case Variants
  7. Test Cases
  8. Expected Results
  9. Negative Testing
  10. The Basic Tour
  11. Testing Device Events
  12. More Tours
  13. Running Your Tests
  14. Scaling Up

The course has a Chicago influence.  I use the Groupon Mobile App as a case study providing a partial test plan (with progressive additions for each lecture) to show how to apply the techniques.  There’s also an example that uses the iconic Chicago Hot Dog as an example of how to test excluded combinations — no ketchup.

 

 

Posted in Blog, Software Testing, Technology | Tagged , , , , , , , , | Leave a comment