Conceptual Framework and Overview of Psychometric Properties

included six additional questions and Form Bincluded six different additional questions.Students in the non-respondent sample wererandomly assigned a priori to one of twogroups. Those in Group 1 were interviewedusing Form A and those in Group 2 wereinterviewed using Form B. This procedureallowed us to ask a substantial number ofquestions from the survey without making theinterview too long to jeopardize reliability andvalidity.CSR staff randomly selected between 100 and200 students from each school (based on totalundergraduate enrollment) who were judgedto be non-respondents by mid-April 2001.That is, those classified as non-respondentshad been contacted several times and invitedto complete The College Student Survey buthad not done so. The goal was to interview atleast 25 non-respondents from each of the 21institutions for a total of 525.Data were collected using the University ofCalifornia Computer-Assisted SurveyMethods software (CASES). All interviewershad at least 20 hours of training ininterviewing techniques and an additionalhour of study-specific training using theNSSE Non-Respondent Interview protocol.Students with confirmed valid telephonenumbers were called at least a dozen times,unless the respondent refused or insufficienttime remained before the end of the study.Multivariate analysis of variance was used tocompare the two groups of respondents andnon-respondents from the respective schoolson 21 engagement and 3 demographic itemsfrom The College Student Report. Theanalyses were conducted separately for firstyearand senior students. The total numbers ofstudents with complete usable information forthis analysis were as follows: first-yearrespondents = 3,470 and non-respondents =291, and senior respondents = 3,391 and nonrespondents=199.Compared with first-year respondents, firstyearnon-respondents scored higher on ninecomparisons. First-year respondents scoredhigher on only three items (using e-mail tocontact an instructor, writing more papersfewer than 5 pages, and taking more classesthat emphasized memorization). There wereno differences on 7 of the 21 comparableitems. For seniors, non-respondents againappeared to be somewhat more engaged thanrespondents as they scored higher on six itemswhile senior respondents scored higher on thesame three items as their first-yearcounterparts (using e-mail to contact aninstructor, writing more papers fewer than 5pages long, taking more classes thatemphasized memorization). No differenceswere found on more than half (11) of theitems.Overall, it appears that undergraduate studentswho do not complete the NSSE survey wheninvited to do so may be slightly more engagedthan respondents. This is counter to whatmany observers believe, that non-respondentshave a less educationally productiveexperience and, as a result, do not respond tosurveys. The findings from the telephoneinterviews suggest that the opposite may betrue, that non-respondents are busier in manydimensions of their lives and don=t take timeto complete surveys.At the same time we must exercise duecaution in drawing firm conclusions fromthese results. Telephone interviews typicallyare associated with a favorable mode effect,meaning that those interviewed often respondsomewhat more positively to telephonesurveys than when answering the samequestions on a paper questionnaire (Dillman,Sangster, Tarnai & Rockwood, 1996). Thus, itappears that few meaningful differences existbetween respondents and non-respondents interms of their engagement in educationallyeffective practices.Framework & Psychometric PropertiesPage 13 of 26