SEKE 2012 Proceedings - Knowledge Systems Institute

the following inclusion relation among the criteria: ACCU,
APU, AE, AES include the ANW criterion; the APU criterion
includes the ACU criterion and the ANW includes the
ACU criterion. These results suggest a test strategy to apply
these testing criteria: 1) generate test cases adequate to
ACCU criterion; 2) in the sequence, select new test cases to
cover the APU criterion and 3) since there is not relationship
between the other criteria pairs, it can be used information
about the cost and effectiveness to generate new test cases,
assuring that relevant aspects of the application under test
will be properly tested.
4.1 Threats to Validity
The external validity of the experiment refers to the ability
to generalize the results. Since the programs used are
small, they may not be representative of the population.
However these benchmarks are largely used to compare
other testing techniques for multithreaded programs and
they include programs that contain faults commonly found
in these applications. The generalization can be achieved
if the experiment is replicated considering other domain of
concurrent applications.
The construct validity threats concern the relationship
between theory and observation. In our case, this threat is
related to knowledge about the used programs and inserted
faults, where this knowledge can influence the application
of the testing criterion. To avoid this threat, the order of
application of the criteria was different for each program,
preventing that the knowledge of the programs and inserted
faults would influence in the generation of test cases.
5 Conclusion
This paper presents an experimental study to evaluate a
family of structural testing criteria for multithreaded programs,
where the objective was to find evidence about the
cost, effectiveness and strength of these testing criteria, aiming
to define a testing strategy to apply them.
The results of hypothesis analysis indicate that the testing
criteria present different cost and effectivess. The comparison
among the sequential criteria: AN, AE, ACU and
APU, and the respective concurrent criteria: ANP and
ANW, AES, ACCU and ACPU, resulted that the sequential
criteria sequential are the most expensive in terms of
application cost. In respect to effectiveness, the sequencial
testing criteria obtained better results, where only the AES
criterion had effectiveness similar to the AE criterion. The
results related to the strength indicated the complementary
relationship among the testing criteria.
According to Wohlin et al. [9], an experiment will never
provide the final answer to a question, hence, it is important
to facilitate its replication. In this sense, we are packing the
material used and generated in this experiment to provide a
lab package, which it is available for public access 1 . We believe
that is important to evaluate these aspects considering
other testing criteria for multithreaded programs, but using
the same benchmarks to allow the comparison between the
results obtained.
Acknowledgment
The authors would like to thank FAPESP, Brazilian
funding agency, for the financial support (process number:
2010/04042-1).
References
[1] O. Edelstein, E. Farchi, E. Goldin, Y. Nir, G. Ratsaby, and
S. Ur. Framework for testing multi-threaded java programs.
Concurr. Comput.: Pract. Exper., 15:485–499, 2003.
[2] P. V. Koppol, R. H. Carver, and K. C. Tai. Incremental integration
testing of concurrent programs. IEEE Trans. Softw.
Eng., 28:607–623, June 2002.
[3] N. Li, U. Praphamontripong, and J. Offutt. An experimental
comparison of four unit test criteria: Mutation, edge-pair,
all-uses and prime path coverage. In ICSTW09 - International
Conference on Software Testing, Verification and Validation
Workshops, pages 220 –229, april 2009.
[4] N. Rungta and E. G. Mercer. Clash of the titans: tools
and techniques for hunting bugs in concurrent programs. In
Proc. of PADTAD ’09, pages 9:1–9:10, New York, NY, USA,
2009. ACM.
[5] F. S. Sarmanho, P. S. Souza, S. R. Souza, and A. S.
Simão. Structural testing for semaphore-based multithread
programs. In Proc. of ICCS ’08, pages 337–346, Berlin,
Heidelberg, 2008.
[6] S. R. S. Souza, S. R. Vergilio, P. S. L. Souza, A. S. Simão,
and A. C. Hausen. Structural testing criteria for messagepassing
parallel programs. Concurr. Comput. : Pract. Exper.,
20:1893–1916, November 2008.
[7] Valgrind-Developers. Valgrind-3.6.1. http://valgrind.org.
Accessed: jun/2011.
[8] E. J. Weyuker. The cost of data flow testing: An empirical
study. IEEE Trans. Softw. Eng., 16:121–128, February 1990.
[9] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell,
and A. Wesslén. Experimentation in software engineering:
An Introduction. Kluwer Academic Publishers, 2000.
[10] W. E. Wong, Y. Lei, and X. Ma. Effective generation of test
sequences for structural testing of concurrent programs. In
ICECCS, pages 539–548, 2005.
[11] C. S. D. Yang, A. L. Souter, and L. L. Pollock. All-du-path
coverage for parallel programs. SIGSOFT Softw. Eng. Notes,
23:153–162, March 1998.
[12] Y. Yang, X. Chen, and G. Gopalakrishnan. Inspect: A runtime
model checker for multithreaded C programs. Technical
report, 2008.
1 http://stoa.usp.br/silvanamm/files
479