Curbstoning and beyond Confronting data fabrication in survey research

420 S. Koczela et al. / Curbstoning and beyond: Confronting data fabrication in survey research
data collection tends to be via paper and pencil, which
makes tight controls over administrative and paradata
considerably more difficult and the resulting data less
reliable.
7. Motivation in fabrication prevention
Statistical agencies would like to prevent as much
fabrication as possible, a clearly preferable outcome to
catching it after the fact. Here again, the motivation for
fabricating data can be useful to examine, particularly
the difference between intrinsic versus extrinsic motivation
for field staff. Although the following discussion
focuses primarily on interviewer motivation, the
issues clearly apply to others such as supervisors, data
entry, or project staff.
Motivation has been widely studied in social psychology
(e.g. [16]) and there are clear distinctions between
intrinsic and extrinsic motivations for behavior.
Extrinsic motivation is a key focus of De Haas
and Winker’s work on interviewer fabrication (2014).
Maximizing money income and reducing risk of being
caught would each be examples of external rewards, or
extrinsic motivation.
To illustrate money as an extrinsic motivation, imagine
from the interviewer’s point of view: “because of
how much they are paying me, I’m going to collect
the most accurate data possible.” This is extrinsic motivation
in the form of a paycheck. If interviewers responded
clearly to this as motivation, it would lead to
the question of adjusting interviewer pay to avoid fabrication.
The pay scale that would be required to provide
sufficient extrinsic motivation on its own would
likely be exorbitant.
Another extrinsic motivator is the knowledge that
someone is checking up on the interviewer’s work,
which increases the perceived risk of being caught
cheating. If interviewers do not think that anyone is
checking, they may not be motivated to put in the effort
to collect accurate data, given the low perceived risk
of simply fabricating the data rather than collecting it.
Fabrication motivation is reduced when interviewers
know that quality control procedures are in place and
that people are monitoring their work. Fear of getting
caught is the extrinsic motivation. In this case, interviewers
are saying to themselves, “because I know they
are checking my work, I’m going to collect the most
accurate data possible.”
However, the ability of many survey organizations
to monitor interviewers is too weak to rely solely or
primarily on monitoring as a motivation. Quality control
monitoring or verification of field work is quite expensive
and thus not likely to be conducted for most of
an interviewer’s caseload. A key reason for the importance
of Winker’s work and other statistical methods of
detecting likely fabricators is that we cannot fully monitor
or evaluate 100 percent of field work. Thus, methods
for identifying which small slice of the field work
is most important to monitor is crucial. With only a
small percentage of interviews monitored, interviewers
may not give much weight to the fear of being caught.
To change this calculation, sponsors would have to improve
their statistical detection methods to better allocate
reinterviews, or find ways to confirm fabrication
without reinterviewing.
In addition to extrinsic motivation, survey organizations
can look at how to cultivate intrinsic motivation
[12]. Intrinsic motivation comes from within the
interviewers themselves and is internalized without external
punishments or rewards. This can be accomplished
in a number of ways. First, organizations have
to minimize an “us versus them” culture. Survey organizations
have to cultivate the idea that interviewers are
on the same side as headquarters personnel and survey
project leaders, and that project leaders and staff are
all on the same side as respondents. That is, the field
firm is not collecting data for their own benefit at the
expense of respondents, but instead to produce results
for the respondents’ benefit. As survey researchers, we
can encourage interviewers to think, “Because we are
all on the same team, I’m going to collect the most accurate
data possible.”
Additionally, survey organizations can emphasize
the value of the agency, encouraging interviewers to
think “Because I work for my particular organization,
I am going to collect the most accurate data possible.”
Or, “Because of the value of the work and because
these statistics are so important, I’m going to collect
the most accurate data possible.” These intrinsic motivators
would minimize the temptation for interviewers
to fabricate.
One way to foster intrinsic motivation is investment
in training and using training to increase employee engagement
at all levels. This requires communication
up and down the chain of responsibility, from survey
managers to field managers to interviewers and back.
Staff at all levels, not just survey program managers,
must know why the survey and the data are important.
If the interviewers do not feel that importance, if they
have not gotten that message, they are less motivated
to collect quality data. Here, we can refer the reader
to Deming’s famous 14 points (1982) on total quality
management.