Contents

CHAPTER 5: Survey Research 165the two administrations of the questionnaire, but a person’s relative position inthe distribution of scores should be similar at the two test times. The consistencyof this relative positioning is determined by computing a correlation coefficientusing the two scores on the questionnaire for each person in the sample. A desirablevalue for test–retest reliability coefficients is .80 or above, but the size of thecoefficient will depend on factors such as the number and types of items.A self-report measure with many items to measure a construct will be morereliable than a measure with few items. For example, we are likely to haveunreliable measures if we try to measure a baseball player’s hitting ability basedon a single time at bat or a person’s attitude toward the death penalty based on asingle question on a survey. The reliability of our measures will increase greatlyif we average the behavior in question across a large number of observations—many at-bats and many survey questions (Epstein, 1979). Of course, researchersmust walk a fine line between too few items and too many items. Too manyitems on a survey can cause respondents to become tired or careless about theirresponses.In general, measurements will also be more reliable when there is greatervariability on the factor being measured among the individuals being tested.Often the goal of measurement is to determine the extent to which individualsdiffer. A sample of individuals who vary a great deal from one another iseasier to differentiate reliably than are individuals who differ by only a smallamount. Consider this example. Suppose we wish to assess soccer players’ abilityto pass the ball effectively to other players. We will be able to differentiatemore reliably good players from poor players if we include in our sample awider range of players—for example, professionals, high school players, andpeewee players. It would be much harder to differentiate players reliably if wetested only professional players—they’d all be good! Thus, a test is often morereliable when administered to a diverse sample than when given to a restrictedsample of individuals.A third and final factor affecting reliability is related to the conditions underwhich the questionnaire is administered. Questionnaires will yield more reliablemeasurements when the testing situation is free of distractions and whenclear instructions are provided for completing the questionnaire. You mayremember times when your own test performance was hindered by noise orwhen you weren’t sure what a question was asking.The reliability of a survey measure is easier to determine and to achieve than thevalidity of a measure. The definition of validity is deceptively straightforward—a valid questionnaire measures what it is intended to measure. Have you everheard students complain that questions on a test didn’t seem to address thematerial covered in class? This is an issue of validity.At this point, we will focus on construct validity, which is just one of themany ways in which the validity of a measurement is assessed. The constructvalidity of a measure represents the extent to which it measures the theoreticalconstruct it is designed to measure. One approach to determining the constructvalidity of a test relies on two other kinds of validity: convergent validity anddiscriminant validity. These concepts can best be understood by consideringan example.