I attended a great talk today about testing mobile applications, given by Lee Barnes of Utopia Solutions. It recounted the Rubik’s cube permutations that affect mobile app quality and reliability: multiple stacks, multiple handheld devices/form factors, constrained battery life, constrained memory/processor, variability of wireless connectivity, different behavior in dedicated and multi-tasking handheld OS, sensitivity to non-application events (power sleep, incoming call/text, drained battery, use the camera, dropped call, lost connection, etc.) And then there is the sheer proliferation of devices, apps, networks, stacks, etc.
In 2000, a Gartner analyst termed this the “mobile testing nightmare.”
Lee observed that some of his clients are quite surprised to discover that mobile app testing isn’t simpler than testing desktop apps.
He included an overview of some current test automation tools for mobile apps. The take-away? Available tools leave a lot to be desired for automating functional and regression testing of mobile applications. Why?
Lee’s recommendation is to develop smart manual test strategies, and use available test automation only where it provides a clear advantage.
These were exactly the same problems I set out to solve when I founded mVerify ten years ago. Although we made a lot of progress, we didn’t get past most of the above barriers.
Ten years ago, I founded mVerify to provide a solution to the mobile testing nightmare. We had one main competitor, who received over 20 times as much funding as we did from leading VCs. I didn’t raise any VC investment – we raised some Angel money and mostly bootstrapped. Despite that, our products were essentially the same. Our competitor had a few features we didn’t (and we had some they didn’t). Neither of us had a complete solution and neither firm exists today.
However, I did learn what it will take to make the mobile testing nightmare go away. I wish present-day tool providers well as they grapple with the same challenges I did, because an effective solution is still very much needed. But, unless some kind of disruption in this space makes it possible to pay for the substantial development needed, I don’t expect to see much more than the limited present-day technology.
Until that happens, I think Lee Barnes has it about right. A well-crafted test plan conducted by skilled testers, even if it mostly relies on fingers and eyeballs, is still the best (and unfortunately only) way to test mobile apps.