Conceptual Framework and Overview of Psychometric Properties

eliability coefficient for all students (N=569)across all items on The Report was arespectable .83. This indicates a fair degree ofstability in students= responses, consistentwith other psychometric tools measuringattitude and experiences (Crocker & Algina,1986). Some sections of the survey were morestable than others. For example, the reliabilitycoefficient for the 20 College Activities itemswas .77. The coefficient for the 10 OpinionsAbout Your School items was .70, for the 14Educational and Personal Growth items .69,for the five reading, writing, and nature ofexaminations items .66, and for the six timeusage items .63. The mental activities andprogram items were the least stable, withcoefficients of .58 and .57 respectively.In 2002, we conducted a similar test-retestanalysis with 1,226 respondents whocompleted the paper survey twice. For thisanalysis, we used the Pearson productmoment correlation to examine thereliability coefficients for the items used toconstruct our benchmarks. For the itemsrelated to three of the benchmarks(academic challenge, enriching educationalexperiences, and the academic challenge),the reliability coefficients were .74. Thestudent interaction with faculty membersitems and supportive campus environmentitems had reliability coefficients of .75 and.78 respectively.Summary. Taken together, these analysessuggest that the NSSE survey appears to bereliably measuring the constructs it wasdesigned to measure. Assuming thatrespondents were representative of theirrespective institutions, data aggregated at theinstitutional level on an annual basis shouldyield reliable results. The correlations are highbetween the questions common to both years.Some of the lower correlations (e.g., nature ofexams, rewriting papers, tutoring) may be afunction of slight changes in item wording andmodified response options for other items onthe later surveys (e.g., number of paperswritten). At the same time, compared with2000, 2001 and 2002 data reflect a somewhathigher level of student engagement on anumber of NSSE items, though the relativemagnitude of these differences is small.Checking for Mode of Administration EffectsUsing multiple modes of surveyadministration opens up the possibility ofintroducing a systematic bias in the resultsassociated with the method of data collection.That is, do the responses of students who useone mode (i.e., Web) differ in certain waysfrom those who use an alternative mode suchas paper? Further complicating this possibilityis that there are two paths by which studentscan use the Web to complete the NSSEsurvey: (1) students receive the paper surveyin the mail but have the option to complete itvia the Web (Web- option), or (2) studentsattend a Web-only school and must completethe survey on-line (Web-only).Using ordinary least squares (OLS) or logisticregressions we analyzed the data from NSSE2000 to determine if students who completedthe survey on the Web responded differentlythan those who responded via a traditionalpaper format. Specifically, we analyzedresponses from 56,545 students who hadcomplete data for survey mode and all controlvariables. The sample included 9,933students from Web-exclusive institutions andanother 10,013 students who received a papersurvey, but exercised the Web-option. Wecontrolled for a variety of student andinstitutional characteristics that may be linkedto both engagement and mode. The controlvariables included: class, enrollment status,housing, sex, age, race/ethnicity, major field,2000 Carnegie Classification, sector,undergraduate enrollment from IPEDS,admissions selectivity (from Barron’s, 1996),urbanicity from IPEDS, and academic supportexpenses per student from IPEDS. In additionto tests of statistical significance, wecomputed effect sizes to ascertain if theFramework & Psychometric PropertiesPage 17 of 26