Robert V. Binder

Panel Discussion on Open Source Testing Tools

Three chimps - hear no evil, see no evil, speak no evilA few days ago, I participated in a panel discussion on open source tools for testing at the QUEST conference with the two founders of Selenium: Jason Huggins of Sauce Labs and Simon Stewart of Google.

Before the panel started we chatted a bit with the moderator. We couldn’t come up with any contentious issues, so we told the moderator to be prepared with questions. As it turned out, we had different views and thereby avoided the dreaded panel of drinking birds. 😉

I’d decided to talk about my inventory of open source model-based testing tools and experience in building a software product using about a dozen open source systems. I started with an overview listing a few mature open source MBT tools users might find useful (the slides are here.)  I concluded with the lesson I learned after three years of using a dozen open source tools and embedding the open source Tcl codebasethey’re free, but they’re not cheap.

Jason followed with a quick overview of Selenium, relating how it all started here in Chicago when he needed a better way to enter time reports. Selenium is a system that  automates interaction with a browser.  So, it can be used to test web content and features in any browser that supports Javascript, which nearly all do.

Jason briefly described how the Selenium maintainers are adding support for mobile platform browsers (importantly Android and iOS).

Audience questions then drove the discussion.  The following isn’t verbatim — just my best recollection.  Please post a comment with corrections/additions, if you were there.


Q: Do testers need to be more technical to use open source testing tools?

Simon: Simple manual testing is increasingly done by the lowest-cost provider, so if that’s all you (QA professionals and testers) can do, you probably won’t be doing it for long.  However, if you can learn how to develop Selenium scripts with Javascript programming, you’ll be providing higher value that isn’t as easily outsourced.

Jason: You should structure your test suites with action keywords (e.g., “Delete item from cart”) rather than the individual keystrokes and clicks used for an action.  This is also a good idea because there are often several ways to do an action.

Bob: It depends on the tool. There are 493 open source testing products listed at  Most require at least some basic programming skill to use.  Also, some tools are not well documented, so you have to learn by experimentation.

Simon:  Many tools, including Selenium have very good documentation.

Bob: That’s certainly true for the popular tools. Also, there are often active and useful user forums where beginners and experts can get help.


Q: What are the advantages, myths and shortcomings of open source?

Simon:  Because you have access to the codebase, you can have higher confidence that it doesn’t have security problems, for example. We inspect every line of open source code we use at Google.

Bob: Are you saying you’ve inspected every line of development environment tools like Visual Studio or Eclipse  — each over 10 million lines of code?

Simon: We’ve inspected every line of open source code that is integrated with Google production code.

Simon: Besides being able to inspect every line of an open source codebase, you can change it if it doesn’t do what you want.

Bob: While that’s true in theory, most open source users are not willing or able to do that.  Modifying any complex tool means you have to understand its codebase pretty well. And, unless you submit changes as contributions, you’re entirely on your own — potentially a worse situation than being dependent on a commercial provider for fixes.  So, if you’re not willing to invest mindshare in learning a codebase so that you can change it without breaking it, the flexibility of an open source codebase is not much different than a commercial system.

Simon: Having active and open development participation is vital for the open source model to work – it is important to give back.  But, even with a large user base, only a small fraction of open source users are typically active codebase contributors/maintainers.

Bob: Open source systems with a large and active community (like Selenium) are often very useful and reliable. Of course, some commercial products/libraries are very buggy, and provide abysmal support.


Q: How does Selenium compare with commercial testing tools like QTP and Quality Center?

Jason/Simon: Selenium is all you need.

Bob: That’s true if you only test browser-based apps. Of course, this excludes the very large universe of “native” apps that use GUIs like Dot Net, Java, or the native GUI frameworks on Android, iOS, Windows Mobile.

Simon:  There are about four times as many job ads calling for Selenium experience than QTP.

Jason: Selenium is usually used as part of a QA tool chain, including other open source and commercial tools.


Q: Can you use open source for regulated products (e.g., FDA)?

Jason: Selenium wasn’t designed for safety-critical applications, so you shouldn’t rely on Selenium for safety critical applications.

Bob: For software in FDA regulated products, using an open source codebase or tool isn’t much different than using software with a non-open license.  There are two possible uses: as a development environment tool, which would require validation per 21 C.F.R. §820.70(i) as does any other automated tool used to produce FDA regulated products.  If you integrate an open source codebase as a component part of a delivered system, it would be subject to the same V&V that your Quality Management System requires for any other integrated component, produced in-house or by a third party.


Q: Is Selenium useful for Agile development?

All: Yes.


Q: Any final thoughts?

Bob: Learning model-based testing actually would provide much greater job security than learning to program test scripts, because programming can be outsourced just as easily as manual testing. Model-based testing requires analytical thinking and app understanding on a par with a system architect.  For many reasons, this kind of high-value skill is both hard to outsource and advantageous to keep in-house.

Members of the QA community can get a lot more respect using model-based testing than learning the details of Selenium, QTP, or any approach that relies mostly on programming skills.

For an interesting video that shows how MBT can generate a Selenium test suite to test the Sugar CRM system take a look at


Leave a Reply

Comment moderation is enabled, no need to resubmit any comments posted.