Model-Based Testing Community | Connecting the MBT world…

Dec/13

25

Parallelization to speed up MBT

In contrast to classical manual testing, MBT proposes generating test cases automatically from formal models. But the number of generated test cases can become huge, especially if the specification is large, the test case generation algorithm too simple (e.g. based mainly on randomization), or some coverage criterion needs to be achieved. Consequently, MBT needs to handle two challenges:

  • elaborate test case generation can consume a lot of time, especially on complex specifications;
  • test execution can consume a lot of time, especially if the SUT is slow.

One solution to increase the speed of test case generation as well as of test case execution is parallelization, more precisely: distribution. Further benefits of distributed test execution are higher fault-tolerance and being able to test on multiple platforms.

We know of the following three tools from the domain of MBT that focus on parallelizing test case generation. They all use Hadoop for parallelization, and thus use a client server model at least to some extend:

  • Parallel QuickCheck uses property-based random testing via the tool QuickCheck. Properties can be specified either in Haskell or in VDM-SL. Besides test case generation, the tool also performs parallel test execution.
  • Yeti on the cloud uses the automated random testing tool Yeti, which is an on-the-fly MBT tool that tests for runtime exceptions and failures or uses contracts as test oracles. Yeti additionally tries to reduce impracticably large error traces. Besides test case generation, the tool also performs parallel test execution.
  • In PGA, a parallel genetic algorithm is used to parallelize the individual fitness evaluation in each iteration. After sufficiently many iterations, a JUnit test suite with high branch coverage is derived. Therefore, this tool does not parallelize test execution on its own.

There are multiple tools that parallelize test execution, but not test case generation. They are not from the domain of MBT. Examples are HadoopUnit and Selenium (Grid).

All tools mentioned so fare use a client server model for distribution, hence parallelization induces a single point of failure, complexity, overhead and contention. Thus there has been research on decentralizing the distribution: GridUnit partitions the cluster, Almeida at al uses a distributed tree.

Parallelizing both test case generation and test execution enables full decentralization: a peer-to-peer model instead of a client server model can be used since each instance can autarkly generate test cases. We do not know of such a tool and are currently conducting research on this with an own prototype.

Do YOU know of such a tool? Or, more generally, of any other testing tool that makes use of parallelization? What is your opinion of parallelization for MBT?

/\/\3rr`/ (|-|r1$7/\/\45 & 4 |-|4xx`/ N3w `/34r!
David

No tags

2 comments

  • Antti Huima · January 25, 2014 at 2:55 pm

    I know that Conformiq Designer has fully parallelized MBT generation based on master/slave architecture since 2010 or so, using CORBA as middleware.

    Reply

  • Kimmo Nupponen · February 27, 2014 at 10:20 am

    Indeed because large real world test generation problems are computationally very complex and can take long to process on a single PC, Conformiq’s test generation algorithm has been carefully crafted for fully distributed parallel test generation, allowing users automatically to split test generation tasks across multiple cores and PCs in, for example, a public or private cloud / cluster environment or just a large server. This technology enables users to create computational clusters for rapid test generation that can be shared across the users. However even without access to a cloud or a cluster environment, users can still gain the benefits of parallel test generation with Conformiq Designer because even on a single PC setup it automatically maximizes the use of computation resources available locally on the workstation to speed up the test generation process by distributing computation work over all the processor cores available on the local host.

    I actually just wrote a blog post about this which you can check out at http://www.conformiq.com/2014/02/distributed-test-generation/

    Reply

Leave a Reply

<<

>>

Theme Design by devolux.nh2.me
© by MBT Community 2011