Asking the Right Questions, A Guide to Critical Thinking, 8th Ed

How Good Is the Evidence 11? 123
9. Are there any biases or distortions in the surveys, questionnaires, ratings, o
other measurements that the researcher uses} We need to have confidence
that the researcher has measured accurately what she has wanted to
measure. The problem of biased surveys and questionnaires is so pervasive
in research that we discuss it in more detail in a later section.
Generalizing from the Research Sample
Speakers and writers usually use research reports to support generalizations,
that is, claims about events in general. For example, "the medication was
effective in treating cancer for the patients in the study" is not a generalization;
"the medication cures cancer" is. The ability to generalize from research
findings depends on the number, breadth, and randomness of events or people
the researchers study.
The process of selecting events or persons to study is called sampling.
Because researchers can never study all events or people about which
they want to generalize, they must choose some way to sample; and some ways
are preferable to others. You need to keep several important considerations in
mind when evaluating the research sample:
1. The sample must be large enough to justify the generalization or conclusion.
In most cases, the more events or people researchers observe,
the more dependable their conclusion. If we want to form a general
belief about how often college students receive help from others on term
papers, we are better off studying 100 college students than studying 10.
2. The sample must possess as much breadth, or diversity, as the types of
events about which conclusions are to be drawn. For example, if
researchers want to generalize about college students' drinking habits in
general, their evidence should be based on the sampling of a variety of
different kinds of college students in a variety of different kinds of college
settings. Students at a small private school in the Midwest may have
different drinking habits than students at a large public school on the
West Coast; thus, a study of students attending only one school would
lack breadth of sampling.
3. The more random the sample, the better. When researchers randomly
sample, they try to make sure that all events about which they want to