An Introduction to Critical Thinking and Creativity - always yours

EVALUATING SURVEYS AND SAMPLING STUDIES 147
A good way to minimize biased sampling is through random sampling, by
which each sample is selected randomly from the population. Given an adequate
sample size, this method is highly likely to result in a representative sample. But
even with random sampling, we should be careful of potential biases in the results
due to the fact that some selected individuals might not be reachable (for
example, in a telephone survey) or are not willing to participate.
Statistics and advertisements
Many advertisements cite statistical surveys. But we should be cautious
because we usually do not know how these surveys are conducted. For
example, toothpaste manufacturer Colgate once had a poster that said
"More than 80% of dentists recommend Colgate." This seems to say that
most dentists prefer Colgate to other brands. But it turns out that the survey
questions allowed the dentists to recommend more than one brand,
and in fact another competitor's brand was recommended just as often
as Colgate! No wonder the UK Advertising Standards Authority ruled in
2007 that the poster was misleading and it had to be withdrawn.
A similar case concerns a well-known cosmetics firm marketing a
cream that is supposed to rapidly reduce wrinkles. But the only evidence
provided is that "76% of 50 women agreed." But what this means is that
the evidence is based on just the personal opinion from a small sample
with no objective measurement of their skin condition. Furthermore
we are not told how these women were selected and whether there was
any double-blind study conducted. Without such information, the "evidence"
provided is pretty much useless. Unfortunately such advertisements
are quite typical, and as consumers we just have to use our own
judgment and avoid taking advertising claims too seriously.
17.1.4 What method is used to investigate the sample?
If a sample is investigated using a biased method, the statistical results can be
unreliable even if the sample is representative. There are various ways this might
come about:
• Social desirability: Suppose a teacher selects some students randomly and
asks whether they have cheated in exams. This survey will underestimate
the extent of cheating since students are unlikely to admit to cheating to
their teacher. We generally want to portray ourselves positively and are reluctant
to confess to undesirable attitudes or activities. This is especially
true when we are questioned directly or when we have doubts about the
confidentiality of the results.