I__. - International Military Testing Association

Barbara Jezior
U.S. Army
Natick Research, Development, and Engineering Center
Natick, MA.
Interpreting Rating Scale Results:
What does a Mean Mean?
Larry Lesher
GEO-CENTERS, Inc.
Newton Centre, MA.
Richard Popper
Ocean Spray Cranberries Inc.
Lakeville-Middleboro, MA.
Charles Greene Vanessa Ince
U.S. Amly * U.S. Army
Natick Research, Development, ,and Engineering Center Natick Research, Development, and Engineering Center
Natick, MA. Natick, MA.
How well soldiers like items they use in garrison or the field is often measured on
Likert scales, and the mean ratings obtained from these scales are then used as
indicators of user acceptance. In examining data contributing to 176 mean ratings
of various Natickproducts we found that the means accurately predict the acceptor
set, i.e. the percentage of soldiers who rated a product on the positive end of a scale.
Knowing the percentage who find a product acceptable provides a more intuitive
and concrete basis for product development or improvement decisions. For
example, the product developer can operate from the knowledge that 66% find the
product acceptable, insteadof amean rating that deems the product “slightly good.”
Introduction
Natick is deeply involved in consumer acceptability
issues. We develop basic subsistence items for servicemen
- rations, protective clothing, shelters, and airdrop
equipment. These products support m annual procurement
of over 3 billion dollars, making consumer (soldier)
acceptance critic‘al. Items that are unacceptable could sit
in wnrehouses or never be used, and the soldier would be
lacking necessary equipment as well.
To obtain additional quantitative information on how
soldiers felt about our products, we started a large-scale
systematic survey program six years ago. Like many, we
operatedunder the assumption that one of thebest ways to
measure and describe how well the soldier liked the
products was to use the mean and other parameters derived
from verbal rating scales.
After analyzing over 7,000 questionnaires throughout
the six years and writing many reports for managers and
product project officers webegan toquestion this assumption.
We ourselves began to get curiousabout what means
were saying in respect to the measure of product accepta-
241
bility. For instance, while we felt that a mean of 5 on a 7point
scale shoul(l denote a relatively acceptable product,
we found that we usually had many more negative ratings
than expected. Over time we also began to feel, on an
intuitive level, that a mean of 6 on a 7-point scale indicated
a “very” good product but our verbal anchor was labelling
such a product as “moderately” good.
Moreover, in describing survey results to product
managers, we found that while the concept of an average
is rather commonly understood, the accompanying parameters
of standard deviations, skewness, etc., are not
understood outside the research communities, nor should
we expect them to be. The problem here is that a mean in
isolation, whichis what amanageris grappling with when
not understanding its accompanying parameters, can be
very misleading. A manager who makes product decisionswithout
somesemeofwhataratingdistributionisall
about may m‘ake the wrong decisions.
Another problem with means for many is that they
don’t provide a good intuitive feel for what relative
differences are in regard to measuring products, any
statistically significant differences notwithstanding. For
instance, if means differ by one scale point. some don’t