Conceptual Framework and Overview of Psychometric Properties

magnitude of the mode coefficients were highenough to be of practical importance towarrant attention. Finally, we applied poststratificationweights at the student-level forall survey items to minimize nonresponse biasrelated to sex and enrollment status.We analyzed the Web-only and Web-optionresults separately against paper as shown inTable 5 by Model 1 (Web-only) and Model 2(Web-option) against paper. We comparedWeb-only against Web-option in Model 3.For 39 of the 67 items, the unstandardizedcoefficients for Model 1 favored Web-onlyover paper. For Model 2, 40 of the 67 itemsshowed statistically significant effectsfavoring the Web option over paper. Incontrast, there are only 9 statisticallysignificant coefficients that are morefavorable for paper over Web in Models 1 and2 combined. Model 3 reveals that there arerelatively few statistically significantdifferences between the two Web-basedmodes.The effect sizes for most comparisons in bothModel 1 and Model 2 are not large --generally .15 or less, with a few exceptions.Interestingly, the largest effect sizes favoringWeb over paper were for the three computerrelateditems: “used e-mail to communicatewith an instructor” (EMAIL), “used anelectronic medium to discuss of complete anassignment” (ITACADEM), and self-reportedgains in “using computers and informationtechnology” GNCMPTS).These models take into account many studentand school characteristics. However, theresults for items related to computing andinformation technology might differ if a moredirect measure of computing technology atparticular campuses was available. That is,what appears to be a mode effect mightinstead be due to a preponderance of Webrespondents from highly Awired@ campusesthat are, in fact, exposed to a greater array ofcomputing and information technology.On balance, responses of college students toNSSE 2000 Web and paper surveys showsmall but consistent differences that favor theWeb. These findings, especially for itemsunrelated to computing and informationtechnology, generally dovetail with studies insingle postsecondary settings (Layne,DeCristoforo, & McGinty, 1999; Olsen,Wygant, & Brown, 1999; Tomsic, Hendel, &Matross, 2000). This said, it may bepremature to conclude that survey modeshapes college students= responses. First,while the responses slightly favor Web overpaper on a majority of items, the differencesare relatively small. Second, only itemsrelated to computing and informationtechnology exhibited some of the largesteffects favoring Web. Finally, for specificpopulations of students mode may havedifferent effects than those observed here.In auxiliary multivariate analyses, we foundlittle evidence for mode-age (net ofdifferential experiences and expectationsattributable to year in school) or mode-sexinteractions, suggesting that mode effects arenot shaped uniquely by either of thesecharacteristics.Additional information about the analysis ofmode effects is available in the NSSE 2000Norms report (Kuh, Hayek et al., 2001) andfrom Carini, Hayek, Kuh, Kennedy andOuimet (in press). A copy of the Carini et al.paper can is on the NSSE website. We willcontinue to analyze NSSE data in future yearsto learn more about any possible mode effects.Framework & Psychometric PropertiesPage 18 of 26