Statistical Testing Using Automated Search - Crest

POULDING AND CLARK: EFFICIENT SOFTWARE VERIFICATION: STATISTICAL TESTING USING AUTOMATED SEARCH 767more complex than “toy” problems. It is possible to find suchsolutions in a practical time, using computing power typical ofa desktop PC.Hypothesis 2. Test sets generated by statistical testing usingautomated search detect more faults than sets of the same sizederived using random testing with a uniform distribution(uniform random testing).Hypothesis 3. Test sets generated by statistical testing usingautomated search that exercise each coverage element multipletimes detect more faults than test sets typical of traditionalstructural testing that exercise each element once (deterministicstructural testing).Hypothesis 4. Test sets generated from probability distributionsthat demonstrate high lower bounds for the coverage elementprobabilities detect more faults than test sets of the same sizegenerated from distributions with lesser lower bounds. In otherwords, it is beneficial to search for distributions with nearoptimallower bounds.Hypothesis 5. Test sets generated from probability distributionsfound by automated search with a diversity constraint aremore effective at detecting faults than those derived withoutsuch a constraint.The term diversity is used here to describe probabilitydistributions that sample many different input vectors fromthe subdomains associated with each coverage element. Forexample, the following distribution for simpleFunc has thesame optimal lower bound as that defined by (5):80:25; if a ¼ 3;b¼ 9;>< 0:25; if a ¼ 3;b¼ 19;fða; bÞ ¼ 0:25; if a ¼ 28;b¼ 2; ð6Þ0:25; if a ¼ 28;b¼ 12;>:0; otherwise:However, a test set generated from this distributionwould contain only four different input vectors, and it isreasonable to expect that this lack of diversity would lead toa reduced ability to detect faults in the SUT compared to themore diverse distribution defined by (5).The notion of diversity is implicit in existing work onstatistical testing, but Hypothesis 5 proposes that it isbeneficial to make this an explicit objective when usingautomated search. The hypothesis was formulated as aresult of experimental work on the first four hypotheses.Some distributions found using automated search that hadnear-optimal lower bounds nevertheless demonstrated poorfault-detecting ability. A manual inspection of the test casesgenerated by these distributions suggested a lack ofdiversity might be the cause.5 IMPLEMENTATIONIn order to test the experimental hypotheses, an automatedsearch method was implemented. The following aspects ofthe implementation are described in this section:. the representation of a probability distribution overthe software’s input domain,. how the fitness of a candidate solution is evaluated,Fig. 3. Example BN for an SUT with four input arguments.. the search method used to locate the fittest solutions,. brief details of the software engineering.5.1 RepresentationThe components of the input vector are representedinternally by real numbers from the unit interval. Wedenote the internal components as x i , where 0 x i < 1.This normalization enables the internal representation to beindependent of the actual data types and domains of theinput arguments. The mapping from the normalized value,x i , to a realized input argument is straightforward forinteger, floating-point, and Boolean data types. For example,if x 1 is the internal value of the argument a ofsimpleFunc, then a sensible mapping is:a ¼ 1 þb50x 1 c:ð7ÞWe believe this mechanism can be extended to other datatypes by defining a mapping from the unit interval,0 x i < 1, to the domain of the data type. For nonnumericand compound data types—such as strings, pointers, orobject data types—the construction of a suitable mappingwill be nontrivial and may need to incorporate contextspecificinformation such as object constructor methods orvalid pointer values. One possibility is to include thegeneration of suitable mappings as part of the search itself:The search method would locate both a probabilitydistribution over the unit interval and a mapping from theunit interval to the domain of the data type. However, forthe empirical work described in this paper, we consideronly scalar numeric data types where the mapping can bepredefined, as in the example above.In general, there will be dependencies between thecomponents of the input vector. The representation of theprobability distribution must therefore be capable ofdescribing these dependencies as a joint multivariateprobability distribution. It would be insufficient to representonly the marginal distributions of each input vectorcomponent: This would enforce independence between thecomponents and prevent the derivation of the most optimalprobability distributions for statistical testing.For this reason, a Bayesian network (BN) is used todescribe the probability distribution. A BN consists of adirected acyclic graph in which each node represents acomponent of the input vector, and a directed edgeindicates that the child node’s probability distribution isconditionally dependent on the parents node’s value.Fig. 3 is an example of a BN describing a probabilitydistribution over the input domain of an SUT with fourarguments. The directed edges indicate that the distributionof argument x 1 is conditional on the value taken by its

768 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 36, NO. 6, NOVEMBER/DECEMBER 2010parent, x 3 . Similarly, x 4 is conditional on x 3 , while x 2 isconditional on the values taken by both its parents, x 3 and x 4 .A BN typically describes a probability distribution over adiscrete set of values. Since the internal componentrepresentations, x i , take real—and therefore nondiscrete—values, the unit interval for each component is discretizedby partitioning it into contiguous intervals or “bins.” Thenumber of bins and the bin boundaries are variable and arenot the same for each node. As a result of this discretization,the nodes in the BN define conditional probability distributionsin terms of the bins rather than the actual values.As an example, consider again the function simpleFunc.The two arguments, a and b, are represented internallyusing real-valued components x 1 and x 2 .Assume the BN for a distribution indicates that theprobability distribution of x 2 is conditional on x 1 . Assumealso that x 1 is discretized by two bins, say ½0; 0:12Þ; ½0:12; 1Þ,and x 2 by three bins, ½0; 0:17Þ; ½0:17; 0:88Þ; ½0:88; 1Þ. At node x 1 ,a single probability distribution defines the chances ofselecting bin 1 or bin 2. However, at node x 2 , two conditionalprobability distributions are defined: The first specifies thechance of selecting bin 1, 2, or 3 for x 2 if x 1 takes a value frombin 1 of node 1; the second specifies a different distributionthat is used if x 1 takes a value from bin 2.A random vector is sampled from the BN by traversingthe graph from parent nodes to child nodes. At each childnode, one bin is randomly selected according to theconditional probability distribution defined by the binschosen at its parent nodes. To convert bin choices tocomponent values, a value is sampled from the bin’sinterval according to a uniform probability distribution.(This is an arbitrary choice, and we suspect other distributionsmay be more effective.)5.2 Fitness EvaluationThe fitness is a measure of how close a candidateprobability distribution is to satisfying both a coverageconstraint—the statistical testing adequacy criterion—and adiversity constraint. These constraints are expressed usingtarget values: t cov for the lower bound of the coverageelement probabilities and t div for the diversity.To calculate the fitness, K input vectors are sampledfrom the probability distribution. An instrumented versionof the SUT is executed using the sampled input vectors. Theinstrumentation identifies which coverage elements areexercised during each execution.To assess the coverage, the indicator variable e c;i is setto 1 if the coverage element c 2 C is exercised one or moretimes when the SUT is executed using the ith input vectorfrom the sample, and set to 0 otherwise. An estimate of thelower bound of the coverage element probabilities is thencalculated as:g cov ¼ 1 K minc2CX Ki¼1e c;i!: ð8ÞTo assess the diversity, the same sample of input vectorsis considered. For each coverage element c, h c denotes thenumber of times that the most frequent input vector thatexercises the element occurs in the sample. For example, ifbranch c 2 of simpleFunc is exercised by input vectorsfð1; 19Þ; ð3; 20Þ; ð1; 18Þ; ð1; 20Þ; ð3; 20Þg, then h c2 is 2 sincethe most frequent input vector, ð3; 20Þ, occurs twice. Thediversity is estimated as follows:!1 X h cg div ¼ 1 PjCj Kc2C i¼1 e : ð9Þc;iThe term h c = P Ki¼1 e c;i estimates the diversity for a singlecoverageelement, with the denominator adjusting for thenumber of times the element was exercised by the sample ofinput vectors. If a single input vector tends to dominate—i.e.,the sample of input vectors that exercises thecoverage element is not diverse—then this term will beclose to 1. The average of this value is taken over all of thecoverage elements, and then subtracted from 1 so that morediverse probability distributions have a value of g div that isnumerically higher. This is for consistency with g cov , whichalso takes higher values for distributions with bettercoverage.An overall fitness is formed by a weighted sum of thedistance of the coverage and diversity values from theirrespective targets:f ¼ w cov maxðt cov g cov ; 0Þð10Þþ w div maxðt div g div ; 0Þ;where w cov and w div are weightings that adjust the relativecontributions of each constraint in guiding the search’strajectory. The weightings are parameters to the searchalgorithm: As for the other algorithm parameters, suitablevalues must be chosen that ensure an effective algorithm, bymeans of preliminary experimentation, for example.The search method attempts to minimize the fitness withthe objective of finding a distribution with a fitness of 0.This value indicates that both constraints have beensatisfied.5.3 Search MethodStochastic hill climbing is used as the search method. Themethod is similar to the random mutation hill climbingalgorithm described by Forrest and Mitchell in [31].Preliminary experimentation with a number of local searchmethods indicated that stochastic hill climbing is sufficientlyeffective, in terms of performance and quality, infinding probability distribution for statistical testing, withthe benefit of relative simplicity in terms of both implementationand the number of search parameters.We explicitly made no attempt to find the most effectivesearch method for the SUTs under investigation nor to findthe optimal search parameter settings. Since one of theexperimental objectives is to assess the practicality of thetechnique, extensive optimization of the search method andparameters for each SUT would not have been realistic.The initial candidate solution is a BN, where all variablesare independent and the marginal distribution of eachvariable is the uniform distribution over its domain. Thischoice of a constant, rather randomly-chosen, initial distributionfacilitated the experimentation by controlling onesource of stochastic variance.At each iteration of the algorithm, a small sample, size ,of the neighbors to the candidate solution is chosen at

POULDING AND CLARK: EFFICIENT SOFTWARE VERIFICATION: STATISTICAL TESTING USING AUTOMATED SEARCH 769random. Neighbors are defined as BNs formed by makingone, and only one, of the following mutations:. adding an edge between two nodes (as long as thegraph does not become cyclic),. removing an edge,. multiplying or dividing the length of a bin intervalby a fixed value,. splitting a bin into two equally sized bins,. joining two adjacent bins,. multiplying or dividing the conditional probabilityof a bin by a fixed value.These six mutations modify all of the nonconstantproperties of the representation: the directed edges betweennodes, the number and length of bins, and the probabilitiesassigned to bins can all be mutated by these operators, butthe number of nodes—which is determined by the numberof input arguments to the SUT—remains constant. Themutations are also reversible: Any single mutation has acorresponding mutation that reverses it. Nevertheless, thechoice of these particular mutations is arbitrary, and it ispossible that other mutation operators would result in amore effective algorithm.All of the neighbors are equally likely to be sampled,apart from those formed by mutations that increase thenumber of bins in a node or increase the number of edges inthe network graph. These neighbors have a smaller chanceof being sampled than the others, the difference beingcontrolled by a parameter

770 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 36, NO. 6, NOVEMBER/DECEMBER 2010Finding near-optimal probability distributions for thisSUT is particularly difficult. For many coverage elements,there are only a few input vectors that exercise theelement—e.g., code reached only when the player has awinning position—and the vectors belong to disconnectedregions in the input domain.6.1.2 nsichneuThis function consists of automatically generated C codethat implements a Petri net. It forms part of a widely usedsuite of benchmark programs used in worst-case executiontime research [34]. The complexity of this example arisesfrom the large number (more than 250) of if statementsand the data-flow between them. The function wasmodified to restore the number of iterations of the mainloop to the original (larger) value in order to reduce theamount of unreachable code.Not only does this function have a large number ofcoverage elements, the code itself takes significantly longerto execute than the other two SUTs, making fitnessevaluation particularly time consuming.6.2 Assessing Fault-Detecting AbilityThe fault-detecting ability of a test set was assessed by itsability to detect mutant versions of the SUT.The mutants were generated using the Proteum/IM tool[35]. Sets of mutants were created by making single-pointmutations, using each of the large number of mutationoperators built in to the tool. Since mutants were being usedonly to compare test sets, there was no need to identify andremove equivalent mutants. For nsichneu, a random selectionconsisting of 10 percent of all possible mutants were used sothat the mutant set was of a practical size; for the other twoSUTs, all of the possible mutants were used (see Table 1).The mutation score—the proportion of mutants “killed”by a test set—is used as the measure of its ability to detectfaults. A mutant was considered as “killed” if for one ormore test cases, a mutant produced a different output or theprocess was terminated by a different operating signal,from the unmutated SUT. The testing process was configuredso that mutant executables were terminated by theoperating system if they used more than 1 s of CPU time.This limit is significantly longer than the CPU time requiredby the unmutated versions of the SUTs, and so identifiesand terminates any mutant that entered an infinite loop.Test sets were generated by random sampling of inputvectors from the probability distribution without replacement:If an input vector was already in the test set, it was rejectedand a further input vector was sampled. We believe this isconsistent with how test sets would be generated in practice:Using more than one test case with the same input vectorusually offers little benefit in terms of the number of faultsdetected. (Although not described here, initial experimentationshowed that using a with replacement mechanismproduced results that were qualitatively no different.)6.3 Experiment AThirty-two runs of the search algorithm were performedand the proportion of searches that found distributionssatisfying the coverage constraint—the statistical testingTABLE 2Search Parametersadequacy criterion—were measured. A diversity constraintwas not applied in this experiment.The searches were run on a server class machine runninga customized version of Slackware Linux. Each search was asingle-threaded process and used the equivalent of one coreof a CPU running at 3 GHz with a 4 MB cache. The CPUuser times taken by both successful (those finding a suitabledistribution) and unsuccessful searches were recorded. TheCPU user time is approximately equivalent to the wall-clocktime when running on an otherwise unloaded server.The parameters used for the search runs are listed inTable 2. Since the objective of the experiment is to showpracticality of the technique, little time was spent in tuningthe parameters and, where possible, the same parameterswere used for all three SUTs. The only difference betweenthe 32 runs were the seeds provided to the pseudorandomnumber generator (PRNG).Using multiple runs in this way allows us to estimate theexpected proportion of successful searches and the expectedtime taken across all possible seeds to the PRNG. The choiceof 32 runs is a compromise between the accuracy of theseestimates and the resources (computing power and time)that were available for experimentation. During the analysisof the results in Section 7.2, we provide confidence intervalsfor these estimates as a measure of the accuracy that wasachieved.The coverage target—equivalent to the lower bound of thecoverage element probabilities—is near the optimal valuesfor simpleFunc (0.25) and bestMove (0.1667). These optimalvalues are straightforward to determine by a manualexamination of the control flow graph of both these SUTs.The optimal lower bound itself is not used as the coveragetarget since it is difficult to obtain this exact fitness, evenwhen the distribution itself is optimal. It requires a randomsample where each input vector occurs with a frequency thatis exactly proportional to its probability in the distribution,and this is unlikely for the relatively small, finite samplesused to evaluate the fitness.The optimal lower bound for nsichneu is difficult todetermine from its control flow graph owing to thecomplexity of the data dependencies. Instead, we use a

POULDING AND CLARK: EFFICIENT SOFTWARE VERIFICATION: STATISTICAL TESTING USING AUTOMATED SEARCH 771coverage target that is close to the best value found duringpreliminary experimentation.This experiment is designed to provide evidence forHypothesis 1, that it is practical to use automated search toderive probability distributions for statistical testing.6.4 Experiment BExperiment B measures the fault-detecting ability of testsets generated by 10 of the probability distributions foundin Experiment A. For each of the 10 distributions, 10 test setswere generated by random sampling without replacement,using a different PRNG seed in each case. Each time a testcase was added to the test set, the mutation score wasassessed using the method described in Section 6.2.For comparison with random testing, the same experimentwas performed using a uniform distribution in placeof that derived by automated search. One hundred test setswere generated from the uniform distribution so that thetwo samples had the same size.This experiment is designed to provide data to demonstrateHypotheses 2 and 3 that, when using automatedsearch, the superior efficiency of statistical testing comparedto uniform random and deterministic structuraltesting is maintained.6.5 Experiment CExperiment C compares the fault-detecting ability ofprobability distributions that have different lower bounds.It is designed to provide evidence for Hypothesis 4, thatprobability distributions with higher lower bounds generatemore efficient test sets.For each SUT, a sequence of probability distributionswas taken at points along the trajectory of one search run. Ingeneral, such a sequence consists of distributions withincreasing probability lower bounds. The search run usedthe same parameters as in Experiment A. Twenty test setswere generated from each distribution and their mutationscores assessed over a range of test sizes.6.6 Experiment DExperiment D compares the fault-detecting ability of testsets generated from distributions found by automatedsearch with and without a diversity constraint. It providesdata to support Hypothesis 5, that the use of a diversityconstraint results in more efficient test sets.A sample of 10 distributions was found using theparameters specified in Table 2. The parameters have anonzero value for w div and so add a diversity constraint tothe search objectives. The coverage target parameters (t cov )for bestMove and nsichneu were less than the near-optimalvalues of Experiment A in order to demonstrate that a lackof diversity has a significant effect even when usingdistributions with only moderate lower bounds.For comparison, a further sample of distributions wasfound using the same parameters, but with w div set to zeroin order to disable the diversity constraint. More than10 such distributions were found, and a subset of size 10was selected in a principled manner so that the distributionsof the coverage fitnesses across the two samples wereas similar as possible. This was designed to minimize theeffect of probability lower bounds on this experiment.Ten test sets were generated from each distribution in thetwo samples, and their mutation scores assessed over a rangeof test sizes.TABLE 3Search Times (Experiment A)7 RESULTS AND ANALYSIS7.1 Statistical AnalysisIn this section, results are summarized using the mean.Although the median is potentially a more robust statisticfor the skewed distributions that the results could exhibit, itwas found to be misleading when, on occasion, the dataformed similarly sized clusters around two (or more)widely separated values. In this case, the median returneda value from one of the clusters, while the mean gave amore meaningful statistic located between the two clusters.Confidence intervals quoted for the mean values, and theerror bars shown in the graphs, are at the 95 percentconfidence level. They are calculated using bootstrapresampling, specifically the bias corrected and acceleratedpercentile method [36].Nonparametric statistical tests are used to analyze thedata. Since parametric statistical tests can be invalidatedby small deviations from the assumptions that the testsmake [37], the use of nonparametric tests ensures thevalidity of the analysis, while avoiding the need toperform additional analysis to verify that the data conformto the test assumptions.To compare samples, the nonparametric Mann-Whitney-Wilcoxon or rank-sum test is applied [38]. The nullhypothesis for the rank-sum test is that the samples arefrom the same distribution; the alternate hypothesis is thatthe distributions are different. We apply the test at a5 percent significance level.Given a sufficiently large sample, hypothesis tests such asthe rank-sum test can demonstrate statistically significantresults even when the underlying differences are extremelysmall. Therefore, we use an additional test to show that theeffect size—in this case, a difference in the ability to detectfaults—is large enough to be meaningful, given the variabilityin the results. The nonparametric Vargha-DelaneyA-test [39] is used here since its value can be calculated fromthe statistic used by the rank-sum test. We use the guidelinespresented in [39] that an A-statistic of greater than 0.64 (orless than 0.36) is indicative of a “medium” effect size andgreater than 0.71 (or less than 0.29), of a “large” effect size.7.2 Experiment AThe results of Experiment A are summarized 2 in Table 3.In this table, þ is the proportion of search algorithmruns that found a suitable probability distribution (i.e., with2. The unsummarized data for all four experiments is available from:http://www.cs.york.ac.uk/~smp/supplemental.

772 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 36, NO. 6, NOVEMBER/DECEMBER 2010Fig. 4. Fault-detecting ability of statistical testing compared to uniform random testing (Experiment B). (a) simpleFunc, (b) bestMove, (c) nsichneu.a probability lower bound greater than or equal to the targetvalue, t cov given in Table 2). T þ is the mean time taken bythese successful runs (measured as CPU user time), and Tthe mean time taken by unsuccessful runs.We may consider each run of the algorithm to beequivalent to a flip of a coin where the probability of thecoin landing with “heads up” or of the algorithm finding asuitable distribution is þ . Trials such as these, where thereare two possible outcomes and they occur with fixedprobabilities, are termed Bernoulli trials [40]. In practice, afteran unsuccessful search, we would run another search with anew, different seed to the PRNG; in other words, we wouldperform a series of Bernoulli trials until one is successful. Thenumber of failures that occur before a successful outcome in aseries of Bernoulli trials follows a geometric distribution [40],with the mean number of failures being given by ð1 þ Þ= þ[41]. Therefore, we may calculate the mean time taken until asuccessful search occurs (including the time taken for thesuccessful search itself) as:C þ ¼ 1 þT þ T þ : ð11Þ þThe calculated values for C þ are given in the Table 3 and area prediction of how long, on average, it would take to find asuitable probability distribution.It is a conservative prediction in that it assumes theavailability of only one CPU core on which searches canbe run. Software engineers are likely to have access tosignificantly more computing resources than this, even ontheir desktop PC, which would allow searches to beperformed in parallel.We argue that the predicted time until success isindicative of practicality for all three SUTs. The longesttime—the value of C þ for nsichneu—is approximately10 hours, rising to 22 hours at extreme end of its confidenceinterval. These results provide strong evidence for Hypothesis1, that automated search can be a practical method ofderiving probability distributions for statistical testing.7.3 Experiment BThe results of Experiment B are summarized in Fig. 4.The upper row of graphs compares the mutation score oftest sets derived from statistical and uniform randomtesting over a range of test sizes. The mutation scores wereassessed at each integer test size in the range, but for clarity,error bars are shown at regular intervals only.The lower row of graphs show the p-value (left-hand axisusing a log scale) for the rank-sum test applied at each testsize, and the effect size A-statistic (right-hand axis). Dottedlines indicate a 5 percent p-value and the lower boundaries ofthe medium effect size regions. p-values below the 5 percentline indicate that the differences in the mutation scores arestatistically significant. A-statistic values above the 0.64 line,or below the 0.36 line, indicate a medium or large effect size.The graphs demonstrate statistically significant differencesin the mutation scores of test sets derived by statisticaland uniform random testing, with the former having the

POULDING AND CLARK: EFFICIENT SOFTWARE VERIFICATION: STATISTICAL TESTING USING AUTOMATED SEARCH 773greater fault-detecting ability for most test sizes. The effectsize is medium or large for many test sizes.These results support Hypothesis 2 that test setsgenerated using statistical testing are more efficient thantest sets generated by uniform random testing: For the sametest size, the former test sets typically have greater faultdetectingability.The exception is for larger test sets (size > 100) applied tonsichneu. For these larger test sets, uniform random testingis significantly more effective at detecting faults. We suspectthat this is due to a lack of diversity in the probabilitydistribution, and this motivates the use of a diversityconstraint during search.The horizontal dotted lines in the upper row of graphs inFig. 4 are the mean mutation score for each statistical test setat the size at which it first exercises all of the coverageelements in the SUT at least once. (The vertical dotted linesare the mean of these sizes.) At these sizes, the test setssatisfy the adequacy criterion of deterministic structuraltesting. However, due to the stochastic nature in which theyare generated, they will usually contain more test cases thana minimally sized test set. The additional test cases can onlyimprove the mutation score of the test set; therefore, theirmutation scores—the horizontal dotted line—represent anupper bound for the mutation scores achievable bydeterministic structural testing.For each SUT, the mutation scores obtained by statisticaltesting exceed this upper bound at sufficiently large testsizes. This is evidence in support of Hypothesis 3, that testsets derived using statistical testing where each coverageelement is exercised multiple times detect more faults thantest sets typical of deterministic structural testing thatexercise each element only once.7.4 Cost-Benefit AnalysisThe results of Experiments A and B taken together enable asimple cost-benefit analysis that compares statistical testingwith uniform random testing. This can be performed in anumber of ways: for example, by considering the differencein fault detecting ability of test sets of the same size for thetwo methods or the difference in the cost of testing for testsets of different size that have same fault detecting ability.We take the second of these two approaches in order toavoid speculation on the costs of later rectifying faults thatare not discovered during testing, as these costs are highlydependent on the context.If we choose a fault-detecting ability equivalent to amutation score of 0.55 for bestMove, the results of ExperimentB show that this occurs at a test size of 4 for statisticaltesting and at a test size of 61 for uniform random testing.(These sizes are obtained from the data used to plot thegraphs of Fig. 4b.) Statistical testing incurs a cost in findinga suitable probability distribution, and from Table 3, thissearch takes, on average, 129 minutes. If executing,and—often more significantly—checking the results of thetests against a specification or an oracle takes longer than129 minutes for the additional 57 test cases required byrandom testing, then statistical testing will be more efficientin terms of time. We argue that when checking test resultsinvolves a manual comparison against a specification, thenthe superior time efficiency of statistical testing is likely tobe realized in this case. We also note that the search for aprobability distribution is automated, requiring little or nomanual effort, and so speculate that statistical testing islikely to be superior in terms of monetary cost (even insituations where it is not superior in terms of time) if theexecution and checking of the additional 57 test cases is apredominantly manual, and therefore expensive, process.We may repeat this analysis for nsichneu, again choosing afault detecting ability equivalent to a mutation score of 0.55.In this case, random testing requires 14 more test cases thanstatistical testing: The test sizes are 66 and 52, respectively.The search for a probability distribution suitable for statisticaltesting takes 592 minutes. The superiority of statistical testingin terms of time is not as convincing for nsichneu: Randomtesting is quicker overall if executing and checking the extra14 test cases takes no longer than 10 hours. However, weagain note that the search for the probability distribution isautomated, and so, considering monetary cost, statisticaltesting may be cheaper than random testing if checking theresults of the extra 14 test cases is a manual process.The simple analyses of this section are dependent on themutation score we choose. Nevertheless, for bestMove, thelarge difference in the fault-detecting ability of the twomethods shown by Fig. 4b suggests that the generalconclusion remains the same for most choices of mutationscore. For nsichneu, however, any benefit of statistical testingis lost for mutation scores above approximately 0.6. As can beseen in Fig. 4c, mutation scores above this level are achievedby random testing with test sizes that are smaller than thoserequired by statistical testing. This underlines the importanceof investigating whether the use of a diversity constraintimproves the fault-detecting ability of statistical testing.7.5 Experiment CThe results of Experiment C are summarized in Fig. 5. Eachgraph shows the mutation scores for a sequence ofprobability distributions taken from the trajectory of onesearch run. The probability lower bound for each distribution(evaluated accurately using a large sample size) isplotted on the x-axis, and the mutation score (with errorbar) on the y-axis. The lines connect mean mutation scorescalculated at the same test size. Note that the left-hand pointin each graph is the uniform probability distribution that isused to initialize the search: This distribution is equivalentto that used for uniform random testing.At small values of the lower bound—at the left of thegraphs—mutation scores get better as the probability lowerbound increases. However, for larger values of the lowerbound at the right of Fig. 5c, the mutation score begins todecrease. There is also some evidence for this effect at thehighest lower bound values in Fig. 5b.We suspect that is because diversity can be lost as thesearch proceeds—the probability distribution is “overfitted”to the coverage constraint—and so, despite a highlower bound, the distribution generates test sets that arerelatively poor at detecting faults. This again motivates theuse of a diversity constraint in order to avoid this loss ofefficiency as the search proceeds.The results of Experiment C provide some evidence insupport of Hypothesis 4 that distributions with higher lowerbounds generate test sets with a greater ability to detectfaults. However, this is not true in all cases: Searching fornear-optimal lower bounds can be counter-productive forsome SUTs, and we suspect this is a result of losing diversity.

774 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 36, NO. 6, NOVEMBER/DECEMBER 2010Fig. 5. Fault-detecting ability of probability distributions at a sequence of points along a search trajectory (Experiment C). (a) simpleFunc,(b) bestMove, (c) nsichneu.7.6 Experiment DThe results of Experiment D are summarized in Fig. 6. Theupper row of graphs compares the mutation score of testsets derived from probability distributions found, usingautomated search with and without a diversity constraint,and the lower row shows the rank-sum p-value and effectsize A-statistic, as for Experiment B.It can be seen that probability distributions found using adiversity constraint are better at detecting mutants. However,the p-values and A-statistics indicate that the effect isonly significant for some test sizes. For simpleFunc andbestMove, the improvement is greatest for larger tests sizes.For nsichneu, the converse is true: The improvementFig. 6. Fault-detecting ability of probability distributions found by search with and without a diversity constraint (Experiment D). (a) simpleFunc,(b) bestMove, (c) nsichneu.

POULDING AND CLARK: EFFICIENT SOFTWARE VERIFICATION: STATISTICAL TESTING USING AUTOMATED SEARCH 775brought about by the diversity constraint is greatest atsmaller test sizes, but the effect size is generally small.The additional dotted line in Fig. 6c plots the mutationscores for uniform random testing taken from Experiment B.It can be seen that the improved efficiency enabled by thediversity constraint is insufficient to close the gap withuniform random testing at the largest test sizes.These results provide partial evidence for Hypothesis 5that using a diversity constraint during the search improvesthe efficiency of statistical testing. However, the nature of theimprovement depends on both the SUT and the test size.8 CONCLUSIONS AND FUTURE WORKThe experimental results support a number of the hypothesesproposed in Section 4. We have shown that usingautomated search to derive near-optimal probability distributionsfor statistical testing is not only viable, butpractical for different types of SUT (Hypothesis 1). We havedemonstrated that statistical testing using automated searchcontinues to show superior fault-detecting ability comparedto uniform random and deterministic structural testing(Hypotheses 2 and 3). There was also some evidence thatsearching for distributions with the highest probability lowerbounds resulted in the most efficient test sets (Hypothesis 4).However, uniform random testing was more efficientthan statistical testing at large test sizes for one of the SUTs,and distributions with near-optimal lower bounds showeddiminished fault-detecting ability. We hypothesize thatboth effects result from a lack of diversity in the generatedtest data. Although the addition of a diversity constraint asa search objective did improve test set efficiency (Hypothesis5), the results indicate that the metric used is probablytoo crude to retain all of the important forms of diversity inthe generated data.Further work is therefore indicated on different diversitymetrics, with the goal of identifying a measure thatcorrelates well with the ability of the test set to detectfaults. In addition, we plan to expand the notion of diversityto nondiscrete input domains.One of the challenges of researching software engineeringtechniques is the vast range of software to which atechnique may be applied. We have experimented on threeprograms that we feel have contrasting characteristics.However, in the absence of a detailed understanding of thesoftware characteristics that affect our proposed technique,it is inappropriate to extrapolate more widely. In particular,the cardinality of the input domain for the SUTs wasrelatively small as a result of using integer data types. It willbe important to demonstrate that the technique remainseffective and practical for larger input domains.This suggests further work to investigate the softwarecharacteristics that affect the efficacy of our technique. Asfor other SBSE applications, a barrier to take up by softwareengineers is the lack of guidance on the types of problemsfor which use of search is effective, and how to configurethe search algorithm based on the characteristics of theproblem. For the experimental work in this paper, we tookthe conservative approach of using a simple searchalgorithm and used similar algorithm parameters acrossall three SUTs. However, greater scalability would bepossible if more effective and faster algorithms areidentified—particularly those that make full use of availablecomputing resources through parallel processing—and ifthe optimal algorithm parameters could be set a prioribased on relevant characteristics of the SUT.An alternative to tuning the algorithm in advance based onthe characteristics of the SUT would be to adapt parametervalues during the algorithm run itself. Eiben et al. refer to thisapproach as parameter control in their recent survey of bothparameter tuning and parameter control [42]. Given the vastrange of software that may be tested (noted above) and thedifficulty in identifying SUT characteristics that affectalgorithm performance, parameter control may be a viabletechnique if it avoids the need to identify such characteristics.Currently, the fitness metric for the coverage constraintuses a count of the elements exercised by a sample of inputvectors. However, if an element is exercised by no inputvectors in the sample, little guidance is provided to thesearch. In this case, the incorporation of metrics used by othersearch-based test data generation techniques—such as theapproach level and branch distance discussed in Section 3—into the fitness function might also enable the technique toscale to larger and more complicated SUTs.Another path for improvement in algorithm performanceis the use of optimization methods that efficientlyaccommodate—or even make use of—the noise in thefitness function. (We speculate that some noise in the fitnessmay actually be beneficial to the hill climbing algorithmused for this paper by occasionally permitting moves to lessfit distributions in order to escape nonglobal local optima.)A number of methods are suggested by existing work onnoisy fitness functions for simulated annealing and evolutionaryalgorithms [43], [44], [45], [46].Experiment D used two competing objectives—thecoverage and diversity constraints—to find the mostefficient probability distribution. It might therefore beconstructive to apply optimization techniques that areexplicitly multiobjective, rather than using a single fitnessfunction that combines the constraints, as we did in thispaper. This use of efficient multi-objective optimizationalgorithms is an approach taken by many recent SBSEapplications [7], [10], [13], [47].The wider applicability of the search technique proposedin this paper requires an extension of the representation toother input data types. The current use of real numbers in theinternal representation of probability distributions, and ofbinning to control the size of the representation, promises arelatively straightforward extension to floating point arguments.However, the incorporation of nonscalar data types,such as strings, objects, and pointers, will be a significantchallenge, and is a current topic of research for other searchbasedtest data generation techniques (e.g., [48]).Finally, there is scope to search for distributionssatisfying broader adequacy criteria. The coverage elementscould be specified by other testing objectives, suchas coverage of the software’s functionality. The criterion onthe coverage probability distribution could be expressed interms of properties other than the lower bound, such as anincreased probability of exercising parts of the softwarethat have previously shown a propensity for faults, or thatare particularly critical to the correct or safe operation ofthe program.

776 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 36, NO. 6, NOVEMBER/DECEMBER 2010ACKNOWLEDGMENTSThe authors would like to thank Jose Carlos Maldonado andAuri M. R. Vincenzi for their assistance with the Proteum/IMmutation testing tool, Marie-Claude Gaudel for her discussionof statistical testing, and Hélène Waeselynck forclarifying the techniques used in her work with PascaleThévenod-Fosse and which inspired the research describedin this paper. This work is funded by Engineering andPhysical Sciences Research Council grant EP/D050618/1,SEBASE: Software Engineering by Automated SEarch.REFERENCES[1] P. Thévenod-Fosse and H. Waeselynck, “An Investigation ofStatistical Software Testing,” J. Software Testing, Verification andReliability, vol. 1, no. 2 pp. 5-26, 1991.[2] P. Thévenod-Fosse, “Software Validation by Means of StatisticalTesting: Retrospect and Future Direction,” Dependable Computingfor Critical Applications, A. Avizienis and J. C. Laprie, eds., pp. 23-50, Springer, 1991.[3] P. Thévenod-Fosse, H. Waeselynck, and Y. Crouzet, “SoftwareStatistical Testing,” Technical Report 95178, Laboratoire d’Analyseet d’Architecture des Systèmes du CNRS (LAAS), 1995.[4] P. Thévenod-Fosse and H. Waeselynck, “Statemate Applied toStatistical Testing,” Proc. Int’l Symp. Software Testing and Analysis,pp. 99-109, 1993.[5] M. Harman and B.F. Jones, “Search-Based Software Engineering,”Information and Software Technology, vol. 43, pp. 833-839,2001.[6] J. Clark, J.J. Dolado, M. Harman, R. Hierons, B. Jones, M. Lumkin,B. Mitchell, S. Mancoridis, K. Rees, M. Roper, and M. Shepperd,“Reformulating Software Engineering as a Search Problem,” IEEProc. Software, vol. 150, no. 3, pp. 161-175, 2003.[7] Y. Zhang, M. Harman, and S.A. Mansouri, “The Multi-ObjectiveNext Release Problem,” Proc. Genetic and Evolutionary ComputationConf., pp. 1129-1137, 2007.[8] E. Alba and F. Chicano, “Software Project Management withGAs,” Information Sciences, vol. 177, no. 11, pp. 2380-2401, 2007.[9] P. Emberson and I. Bate, “Minimising Task Migrations andPriority Changes in Mode Transitions,” Proc. 13th IEEE Real-Timeand Embedded Technology and Applications Symp., pp. 158-167, 2007.[10] M. Harman and L. Tratt, “Pareto Optimal Search Based Refactoringat the Design Level,” Proc. Genetic and Evolutionary ComputationConf., pp. 1106-1113, 2007.[11] C. Hao, J.A. Clark, and J.L. Jacob, “Synthesising Efficient andEffective Security Protocols,” Proc. Workshop Automated Reasoningfor Security Protocol Analysis, pp. 25-40, 2004.[12] P. McMinn, “Search-Based Software Test Data Generation: ASurvey,” Software Testing, Verification and Reliability, vol. 14, no. 2,pp. 105-156, 2004.[13] D. White, J.A. Clark, J. Jacob, and S. Poulding, “EvolvingSoftware in the Presence of Resource Constraints,” Proc.Genetic and Evolutionary Computation Conf., pp. 1775-1782, 2008.[14] M. Harman, “The Current State and Future of Search BasedSoftware Engineering,” Proc. 29th Int’l Conf. Software Eng. Future ofSoftware Eng., pp. 342-357, 2007.[15] J. Duran and S. Ntafos, “An Evaluation of Random Testing,” IEEETrans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.[16] D. Hamlet, “When Only Random Testing Will Do,” Proc. First Int’lWorkshop Random Testing, pp. 1-9, 2006.[17] H. Zhu, P.A.V. Hall, and J.H.R. May, “Software Unit TestCoverage and Adequacy,” ACM Computing Surveys, vol. 29,no. 4, pp. 366-427, 1997.[18] M.J. Harrold, R. Gupta, and M.L. Soffa, “A Methodology forControlling the Size of a Test Suite,” ACM Trans. Software Eng. andMethodology, vol. 2, no. 3, pp. 270-285, 1993.[19] J.A. Jones and M.J. Harrold, “Test-Suite Reduction and Prioritizationfor Modified Condition/Decision Coverage,” IEEE Trans.Software Eng., vol. 29, no. 3, pp. 195-209, Mar. 2003.[20] S. Ntafos, “On Random and Partition Testing,” ACM SIGSOFTSoftware Eng. Notes, vol. 23, no. 2, pp. 42-48, 1998.[21] DO-178B, Software Considerations in Airborne Systems and EquipmentCertification, Radio Technical Commission for Aeronautics, 1992.[22] Defence Standard 00-55(PART2)/Issue 2, Requirements for SafetyRelated Software in Defence Equipment, Part 2: Guidance, Ministryof Defence, 1997.[23] P. Thévenod-Fosse and H. Waeselynck, “Towards a StatisticalApproach to Testing Object-Oriented Programs,” Proc. IEEE Ann.Int’l Symp. Fault Tolerant Computing, pp. 99-108, 1997.[24] A. Denise, M.-C. Gaudel, and S.-D. Gouraud, “A Generic Methodfor Statistical Testing,” Technical Report 1386, Laboratoire deRecherche en Informatique (LRI), CNRS—Univ. de Paris Sud,Apr. 2004.[25] N. Baskiotis, M. Sebag, M.-C. Gaudel, and S. Gouraud, “AMachine Learning Approach for Statistical Software Testing,”Proc. Int’l Joint Conf. Artificial Intelligence, pp. 2274-2279, 2007.[26] R.P. Pargas, M.J. Harrold, and R.R. Peck, “Test-Data GenerationUsing Genetic Algorithms,” Software Testing, Verification andReliability, vol. 9, pp. 263-282, 1999.[27] B. Korel, “Automatic Software Test Data Generation,” IEEE Trans.Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.[28] N. Tracey, J. Clark, and K. Mander, “Automated Program FlawFinding Using Simulated Annealing,” ACM SIGSOFT SoftwareEng. Notes, vol. 23, no. 2, pp. 73-81, 1998.[29] N.J. Tracey, “A Search-Based Automated Test-Data GenerationFramework for Safety-Critical Software,” PhD dissertation, Dept.of Computer Science, Univ. of York, 2000.[30] J. Wegener, A. Baresel, and H. Sthamer, “Evolutionary TestEnvironment for Automatic Structural Testing,” Information andSoftware Technology, vol. 43, pp. 841-854, 2001.[31] S. Forrest and M. Mitchell“Relative Building-Block Fitness and theBuilding Block Hypothesis,” Foundations of Genetic Algorithms 2,D.L. Whitley, ed., pp. 109-126, Morgan Kaufmann, 1993.[32] M. Galassi et al., GNU Scientific Library Reference Manual, seconded. Network Theory, 2006.[33] “JDK 1.4 Demo Applets.” http://java.sun.com/applets/jdk/1.4/index.html, 2010.[34] “WCET Analysis Project.” http://www.mrtc.mdh.se/projects/wcet/benchmarks.html, 2010.[35] M.E. Delamaro and J.C. Maldonado, “Proteum/IM 2.0: AnIntegrated Mutation Testing Environment,” Mutation Testing forthe New Century, pp. 91-101, Kluwer Academic Publishers, 2001.[36] T.J. DiCiccio and B. Efron, “Bootstrap Confidence Intervals,”Statistical Science, vol. 11, no. 3, pp. 189-212, 1996.[37] N.L. Leech and A.J. Onwuegbuzie, “A Call for Greater Use ofNonparametric Statistics,” technical report, US Dept. of Education,Educational Resources Information Center, 2002.[38] F. Wilcoxon, “Individual Comparisons by Ranking Methods,”Biometrics Bull., vol. 1, no. 6, pp. 80-83, 1945.[39] A. Vargha and H. Delaney, “A Critique and Improvement of theCL Common Language Effect Size Statistics of McGraw andWong,” J. Educational and Behavioral Statistics, vol. 25, no. 2,pp. 101-132, 2000.[40] E.W. Weisstein, “‘Bernoulli Distribution.’ From MathWorld—AWolfram Web Resource.” http://mathworld.wolfram.com/BernoulliDistribution.html, 2010.[41] E.W. Weisstein, “‘Geometric Distribution.’ From MathWorld—AWolfram Web Resource.” http://mathworld.wolfram.com/GeometricDistribution.html, 2010.[42] A.E. Eiben, Z. Michalewicz, M. Schoenauer, and J.E. Smith,“Parameter Control in Evolutionary Algorithms,” ParameterSetting in Evolutionary Algorithms, F.G. Lobo, C.F. Lima, andZ. Michalewicz, eds., vol. 54, pp. 19-46, Springer, 2007.[43] W.J. Gutjahr and G.C. Pflug, “Simulated Annealing for Noisy CostFunctions,” J. Global Optimization, vol. 8, no. 1, pp. 1-13, 1996.[44] S. Rana, D.L. Whitley, and R. Cogswell, “Searching in the Presenceof Noise,” Parallel Problem Solving from Nature, pp. 198-207,Springer, 1996.[45] H.-G. Beyer, “Evolutionary Algorithms in Noisy Environments:Theoretical Issues and Guidelines for Practice,” Computer Methodsin Applied Mechanics and Eng., vol. 186, pp. 239-267, 2000.[46] S. Markon, D.V. Arnold, T. Bäck, T. Beielstein, and H.-G. Beyer,“Thresholding—A Selection Operator for Noisy ES,” Proc. 2001IEEE Congress on Evolutionary Computation, pp. 465-472, 2001.[47] S. Yoo and M. Harman, “Pareto Efficient Multi-Objective TestCase Selection,” Proc. 2007 Int’l Symp. Software Testing and Analysis,pp. 140-150, 2007.[48] K. Lakhotia, M. Harman, and P. McMinn, “Handling DynamicData Structures in Search Based Testing,” Proc. Genetic andEvolutionary Computation Conf., pp. 1759-1766, 2008.

POULDING AND CLARK: EFFICIENT SOFTWARE VERIFICATION: STATISTICAL TESTING USING AUTOMATED SEARCH 777Simon Poulding received the BSc degree inmathematics and the MSc degree in mathematicswith modern applications from the Universityof York in 1989 and 2006, respectively.From 1989 to 2005, he was a software engineerin the financial and healthcare industries, specializingin enterprise management systems anddatabase applications. He is currently a lecturerin the Department of Computer Science, Universityof York. His research interests includethe scalability and robustness of search-based software engineeringalgorithms and the use of reliable and efficient empirical methods incomputer science.John A. Clark received the MA degree inmathematics and the MSc degree in appliedstatistics from the University of Oxford in 1985and 1986, respectively, and the PhD degree incomputer science from the University of York in2002. After industrial experience in securityresearch and development, he joined the Departmentof Computer Science, University ofYork, as a CSE lecturer in safety-critical systems.He is currently a professor of criticalsystems at York, where he leads a team of researchers working onsoftware engineering and security. He has a particular interest inoptimization-based approaches to applications from both of these fields.. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.