Interview with Harry Robinson
Interview by Michael Mlynarski at 1st ETSI MBT User Conference, Berlin, October 2011
Harry Robinson is a Principal Software Design Engineer in Test (SDET) for Microsoft’s Bing team, with over twenty years of software development and testing experience at AT&T Bell Labs, HP, Microsoft, and Google, as well as time spent in the startup trenches. He currently works with Bing teams on developing effective test strategies across the product.
MBT Community (MBT): Hello Harry! Great to meet you here at the ETSI MBT User Conference in Berlin. Let’s start with the first question: To give our readers a first impression, can you shortly introduce what is your relation to MBT?
Harry Robinson (HR): I started doing model-based testing in 1993 when I worked for AT&T Bell Labs. My first model was written in Korn Shell on a UNIX system. The system under test was a trouble tracking system and all tests had to be written in shell scripts, because this was the language that the test team used.
Creating models to do the testing occurred to me because my previous development projects involved communication protocols, and it felt natural to see test execution as a conversation between the test program and the system under test.
MBT was an effective way to test the software, but it caused problems with the existing test metrics. For instance, having a model generate tests meant I didn’t have to craft individual test cases, and test case counts were an important metric for Bell Labs at the time. In the end, MBT allowed us to deliver high quality software that went through its final system test without any bugs being found. But, even the fact that no bugs were found caused a new set of problems. We had never before run a system test pass where we didn’t find any bugs. Did we not find bugs because the tests were weak, or because the software was robust?
A few years later, I went to work for Hewlett-Packard where I applied many model-based testing ideas to internationalization testing. We reasoned that functionality in one language could serve as a test oracle for that functionality in a different language. We were therefore able to use different language versions to oracle each other.
In 1998, Microsoft’s Director of Test invited me to Redmond to give a talk on this internationalization work. Through our conversations, it turned out that he was interested in introducing MBT at Microsoft. A few months later, he made me an offer to switch from HP to Microsoft, specifically to help launching model-based testing efforts. And that is what I did from 1998 to 2005. After 2005, I went to Google and to a startup for a while. Finally, I came back to Microsoft to be a test architect for Bing, where I now support MBT and other new test automation initiatives.
MBT: Let‘s talk about the new initiatives. Are they also related to MBT?
HR: Yes, they are. For me, the notion of models as state machines is too restrictive, especially for search engines where sequences are not as important as data coverage. We call the work we’re doing “heuristic-based automation”. We use heuristic oracles to automate large-scale functional testing. The approach still falls under the umbrella of MBT – we determine, from a black-box testing perspective, what we consider to be reasonable system behaviour. We don‘t necessarily predict what the behaviour will be, but we use heuristic validation rules to check it after the execution.
MBT: Since you are a kind of MBT veteran. What were the biggest problems you faced in using MBT in the industry?
HR: The biggest problems have always been getting MBT to fit into existing processes. For instance, if your team likes to count test cases then traditional test automation is a good fit. But with test generation, test case counts don’t matter.
Another problem is that MBT lets you start testing very early in the development cycle, which means that you can provide tests early and find bugs before the code is officially turned over to the test team. Well, if there is no clean handover to test (e.g. code complete), then the bug metrics are going to be very strangely skewed. When the software does enter test, it has already undergone lots of testing; therefore, you don‘t end up with a burndown as much as you would end up with a sort of “background hum” of bugs.
A particularly odd problem is that many people who end up in test teams have not had the opportunity to study software testing. They think of test automation as a programming activity. But as soon as you start to do MBT you realize you have to know more about testing – issues around test generation and test oracles become very important.
MBT: That‘s a good point! Some people look at testing as a programming activity. As we have seen today, SpecExplorer (test generator provided by Microsoft) provides a more low-level kind of MBT, where models are programmed in a specific programming language. What is your opinion on that topic? Do you prefer to create models the SpecExplorer way or more in a graphical manner?
HR: I‘m not actually a fan of graphical models. Graphical models tend to be what Bob Binder calls “cartoons”. They have limited value because they are usually not executable and can’t generate tests automatically. As for using any particular tool, I have created models in Korn Shell, in C, Visual Test, Python, and C#. It doesn’t really matter what language or tool you use. What you need is a programming language that allows you to do more than simple capture/replay.
The programming that testers do is a different kind of programming from traditional development. Testers don‘t care about performance in a test, not nearly as much as in development. Therefore the test code, test algorithms, and data structures can be much simpler. The difference in focus is actually beneficial; testers are less susceptible to common mode failure since they are implementing a different algorithm than the developers.
In fact, sometimes testers can actually work a problem backwards: start with a known result and work backward to derive a test input that should produce that pre-determined result. For example if you start with a sorted list, randomize it, and feed it to your sort routine, you already know what the sorted output should look like. It should match the sorted list you had at the beginning!
MBT: Do you provide trainings about MBT at Microsoft? What advice can you give our community readers on how to start with MBT?
HR: I do train and coach at Microsoft and occasionally at conferences and workshops as well. When teaching, I prefer to take a problem the testers are working on and that they are familiar with. It’s especially useful if this problem is hard to solve with their current test automation approach. I then work with them to analyze the problem with MBT. We take a step back and think about the mental model they used to create their existing automation. Now, if we take those mental models and generalize them, we have the beginnings of a model that can generate tests and is not restricted to individual sequences.
I advise people to stay away from tools as long as they can. Every tool has its own way of looking at a problem, and the tool will frame your own thinking about the problem. If your tool favors one type of solution then you unconsciously begin to think that that is the only way to do it, even if that solution does not fit your problem.
MBT: Based on your experience, can you play an oracle for us and predict how MBT will evolve in the next years? Will it be the future of software testing? And is it more a standalone technique or should we always combine it with other test solutions?
HR: Yes, I do think that the future of testing has a large model-based component. I don‘t think that it is the only solution. There are activities that machines do better than people do, so we should let the machines do that part. Humans should do the parts humans are better at. So, there will always be a part for people to play.
The testing I’m currently doing actually needs people to be involved. Computers can execute heuristic oracles to tell you “This behavior of the system seems suspicious. We should really have a human look at it“. I often think of these heuristics as analogous to spam filters for e-mail. An email spam filter doesn’t delete suspicious e-mails but leaves them in a folder for a human to look at.
I think what will go away in the future are capture/replay solutions – they haven’t ever worked; they’ve always been fragile. Vendors in the past pushed these solutions on teams without understanding what those test teams needed.
The future will also see an increase in software that tests itself as it executes, especially as we move to the cloud. Self-verification, models and filtering will become embedded in products as a matter of course, so that the product itself can recognize when it encounters a quality issue.
In short, the future will have far fewer testers. But the testers who remain will know quite a bit about testing!
MBT: Last thing we would like to ask you are your future plans? Maybe for the next 10 years in the area of MBT? 🙂
HR: Wow! My hope for the next 10 years is to continue to do what I’ve been doing – work on interesting test problems. That’s what testing is really about for me. I like to coach and work with teams, because it’s very rewarding to help solve problems that seem unsolvable. Sometimes a challenging problem becomes easy to solve just by changing how we think about the testing.
MBT: Ok, that‘s it Harry! Thank you very much for this interesting interview. We wish you many interesting MBT projects in the future!
HR: Thanks and good bye.