Parallelization to speed up MBT
In contrast to classical manual testing, MBT proposes generating test cases automatically from formal models. But the number of generated test cases can become huge, especially if the specification is large, the test case generation algorithm too simple (e.g. based mainly on randomization), or some coverage criterion needs to be achieved. Consequently, MBT needs to handle two challenges:
- elaborate test case generation can consume a lot of time, especially on complex specifications;
- test execution can consume a lot of time, especially if the SUT is slow.
One solution to increase the speed of test case generation as well as of test case execution is parallelization, more precisely: distribution. Further benefits of distributed test execution are higher fault-tolerance and being able to test on multiple platforms.
We know of the following three tools from the domain of MBT that focus on parallelizing test case generation. They all use Hadoop for parallelization, and thus use a client server model at least to some extend:
- Parallel QuickCheck uses property-based random testing via the tool QuickCheck. Properties can be specified either in Haskell or in VDM-SL. Besides test case generation, the tool also performs parallel test execution.
- Yeti on the cloud uses the automated random testing tool Yeti, which is an on-the-fly MBT tool that tests for runtime exceptions and failures or uses contracts as test oracles. Yeti additionally tries to reduce impracticably large error traces. Besides test case generation, the tool also performs parallel test execution.
- In PGA, a parallel genetic algorithm is used to parallelize the individual fitness evaluation in each iteration. After sufficiently many iterations, a JUnit test suite with high branch coverage is derived. Therefore, this tool does not parallelize test execution on its own.
All tools mentioned so fare use a client server model for distribution, hence parallelization induces a single point of failure, complexity, overhead and contention. Thus there has been research on decentralizing the distribution: GridUnit partitions the cluster, Almeida at al uses a distributed tree.
Parallelizing both test case generation and test execution enables full decentralization: a peer-to-peer model instead of a client server model can be used since each instance can autarkly generate test cases. We do not know of such a tool and are currently conducting research on this with an own prototype.
Do YOU know of such a tool? Or, more generally, of any other testing tool that makes use of parallelization? What is your opinion of parallelization for MBT?
/\/\3rr`/ (|-|r1$7/\/\45 & 4 |-|4xx`/ N3w `/34r!