Contents

CHAPTER 5: Survey Research 157well-off financially” as a very important or essential objective, higher thanany other item on the survey (Pryor et al., 2009). The researchers cited theeconomic downturn as an important factor in students’ responses to the survey,including items reflecting the increasing financial difficulties associatedwith attending college.The successive independent samples design has limitations. Consider hypotheticalresults from a successive independent samples design. Suppose youhear it reported that in 1977, 35% of college students surveyed said they don’ttrust the U.S. government, 25% reported they have mixed feelings, and 40%reported they do trust the U.S. government. Then you hear it reported that in2007 results to the same survey question showed that 55% of students say theydon’t trust the government, 25% say they have mixed feelings, and 20% do havetrust. How can we interpret these results? To account for the attitude change inthe 2007 sample, can we conclude, for example, that 20% of the 1977 “do trust”group changed their minds and now don’t trust the government? No! And perhapsyou can see why.What we must remember is that the students surveyed in 1977 (in our hypotheticalsurvey) were not the same students surveyed in 2007. The extent towhich specific individuals change their views over time can be determined onlyby testing the same individuals on both occasions. We cannot determine in thesuccessive independent samples design who has changed their views or byhow much. You may have considered a similar problem of interpretation whenexamining the results of the Sax et al. (2003) survey presented in Figure 5.3.What accounts for the changes in students’ attitudes observed from 1966 to2003? We can’t say on the basis of these data. The purpose of the successiveindependent samples design is to describe changes over time in the distributionof population characteristics, not to describe changes in individual respondents.Accordingly, the successive independent samples design is not always helpfulin ferreting out reasons for observed changes like those shown in Figure 5.3.(As you will soon see, another survey design, the longitudinal design, is moreappropriate in these situations.)A second potential limitation of the successive independent samples designarises when the successive samples are not representative of the same population.Imagine that in our hypothetical survey of students’ attitude toward theU.S. government, the sample comprised students from small rural collegesin 1977 and students from large urban universities in 2007. The comparisonsof students’ attitudes toward the government over this time period would bemeaningless. That is, we wouldn’t be able to state that the student populationhad become less trusting over time because it’s possible that the degree of trustdiffers for rural and urban students, which could also account for the differencebetween 1977 and 2007 results. The rural and urban samples illustrate theproblem of noncomparable successive samples. Changes in the population across timecan be described accurately only when the successive independent samples representthe same population. Although sophisticated statistical procedures exist to helpunravel the problems associated with noncomparable successive samples, thebest solution is to avoid the problem by carefully selecting successive samplesthat represent the same population.