How Things Work - Doha Academy of Tertiary Studies

experiential Understanding 61
well workers would perform on other assessments, on other criteria. The
ranking is more important than directly measuring. That is the way the
criterial researcher often thinks: It’s better to measure a little well than
to measure a lot poorly.
Looking ahead, criterial researchers conceptualize outcome levels,
levels of confidence, levels of decision making. How high will the performance
have to be for a certain decision to be made? They do not usually
set such levels, but they would like their analysis to be that refined.
What they may do is to take the forthcoming performance and compare
it with a previous performance to indicate how the thing is working.
Or they may compare the performance of the studied group with the
performance of a control group. At other times, they leave it to experts
to decide, after the fact, what the meaning of the performance is—but
many of them dislike such subjectivity. Most criterial evaluators are happier
when they can be explicit in advance about the level of success,
such as using a level of statistical significance. An objective standard is
wanted for deciding how the thing works.
What criterial researchers are most proud of is their measurement.
They like to get numbers down on paper to show the performances of
participants and beneficiaries. They analyze the numbers, sometimes in
complex statistical ways, to show how things are working. They might
show, for example, that—after adjustments for differences in prior standing
and amount of assistance provided—the changes just made caused
production to rise significantly. Sometimes that will be seen as increasing
understanding of how the thing works.
But it takes more than that to conclude with some certainty that
other things work that way, or that policy should be changed for future
operations. For generalization, we need to study variations of the changes
in a variety of situations. Seeking generalizations is pretty much the way
of ordinary social science and policy study.
I started my career doing criterial research. When I did instructional
research in the early 1960s, I was a psychometrician and an educational
psychologist, and I only did criterial research. But I failed to
make that research answer enough of the practical questions, so over the
next 40 years, I slowly changed to being more of an ethnographer and
case researcher. And I recognized this as more experiential work and
called it “responsive evaluation.” But here in this book we are calling it
“qualitative research,” and we are seeking activity more than merit.