Domain Testing: Divide and Conquer - Testing Education

Analysis of Performance Test Results for Sowmya Padmanabhan's Thesis Data
Cem Kaner, J.D., Ph.D.
January 30, 2004
This memo summarizes my notes on the results of two experiments conducted by Sowmya
Padmanabhan in the summer of 2003. I am the research advisor for that work.
Three of us, James Bach, Pat McGee and I, have done this analysis. I have discussed analysis
criteria and results with both McGee and Bach. Their analyses have influenced my analysis to
some degree, and I have somewhat influenced their analysis. My primary influence on them was
in helping them develop their performance criterion. The evaluation criterion that I laid out for
them is this:
Compare the results from this Performance Test to the results that you would
expect from a tester who claimed to have a year's experience and who claimed to
be good at domain testing.
All three of us have substantial testing experience, and have interviewed, hired and supervised
testers. All of us have our own sense of what we would expect from a 1-year tester, and we also
differ in the details of how we would do a domain testing analysis. We also recognize that there
are different approaches to domain testing and that our goal was to determine whether this
person looked reasonably experienced and whether the work looked reasonably sound.
I am confident that our discussions influenced how we articulated our conclusions, but did not
influence our conclusions. Each of us was independently uncomfortable with the results.
I analyzed two experiments' results. Experiment 1's subjects had been mis-instructed on all-pairs
testing. Additionally, the wording of some of the Experiment 1 questions was confusing.
Accordingly, I am ignoring the Experiment 1 data on combination testing.
The students took this as an open book exams. They had lecture notes, checklists, and other
materials available. The lecture notes and handouts presented a series of concepts associated with
domain testing, and then boiled the concepts down into step-by-step procedures.
Overview
Student performance was remarkably consistent. They approached the task in the same order,
identified the same variables, analyzed and discussed them in essentially the same way,
presented the results in extremely similar tables, and made the same mistakes.
Overall, I think that the students learned the procedures well, but didn't learn domain testing
well.
The answers are both more and less than I would expect from a 1-year experienced tester.
• The tables are tidy and well-structured. Each variable is analyzed in terms of several
dimensions. The all-pairs analyses of the Experiment 2 students look OK.
• The students' descriptions of risks are uninsightful and largely redundant with the tests.