Conceptual Framework and Overview of Psychometric Properties

The National Survey of Student Engagement:Conceptual Framework and Overview of Psychometric PropertiesGeorge D. KuhIndiana University Center for Postsecondary Research and PlanningWhat students do during college counts morein terms of desired outcomes than who theyare or even where they go to college. That is,the voluminous research on college studentdevelopment shows that the time and energystudents devote to educationally purposefulactivities is the single best predictor of theirlearning and personal development (Astin,1993; Pascarella & Terenzini, 1991; Pace,1980). The implication for estimatingcollegiate quality is clear. Those institutionsthat more fully engage their students in thevariety of activities that contribute to valuedoutcomes of college can claim to be of higherquality in comparison with similar types ofcolleges and universities.Certain institutional practices are known tolead to high levels of student engagement(Astin, 1991; Chickering & Reisser, 1993;Kuh, Schuh, Whitt & Associates, 1991;Pascarella & Terenzini, 1991). Perhaps thebest known set of engagement indicators isthe "Seven Principles for Good Practice inUndergraduate Education" (Chickering &Gamson, 1987). These principles includestudent-faculty contact, cooperation amongstudents, active learning, prompt feedback,time on task, high expectations, and respectfor diverse talents and ways of learning. Alsoimportant to student learning are institutionalenvironments that are perceived by students asinclusive and affirming and whereexpectations for performance are clearlycommunicated and set at reasonably highlevels (Education Commission of the States,1995; Kuh, 2001; Kuh et al., 1991; Pascarella,2001). All these factors and conditions arepositively related to student satisfaction andachievement on a variety of dimensions(Astin, 1984, 1985, 1993; Bruffee, 1993;Goodsell, Maher, & Tinto, 1992; Johnson,Johnson, & Smith, 1991; McKeachie,Pintrich, Lin, & Smith, 1986; Pascarella &Terenzini, 1991; Pike, 1993; Sorcinelli, 1991).Thus, educationally effective colleges anduniversities -- those that add value -- channelstudents' energies toward appropriateactivities and engage them at a high level inthese activities (Educational Commission ofthe States, 1995; The Study Group, 1984).Emphasizing good educational practice helpsfocus faculty, staff, students, and others on thetasks and activities that are associated withhigher yields in terms of desired studentoutcomes. Toward these ends, faculty andadministrators would do well to arrange thecurriculum and other aspects of the collegeexperience in accord with these goodpractices, thereby encouraging students to putforth more effort (e.g., write more papers, readmore books, meet more frequently withfaculty and peers, use information technologyappropriately) which will result in greatergains in such areas as critical thinking,problem solving, effective communication,and responsible citizenship.Framework & Psychometric PropertiesPage 1 of 26

Overview and Content of the NSSEProject and QuestionnaireThe National Survey of Student Engagement(NSSE) is specifically designed to assess theextent to which students are engaged inempirically derived good educationalpractices and what they gain from theircollege experience (Kuh, 2001). The maincontent of the NSSE instrument, The CollegeStudent Report, represents student behaviorsthat are highly correlated with many desirablelearning and personal development outcomesof college. Responding to the questionnairerequires that students reflect on what they areputting into and getting out of their collegeexperience. Thus, completing the survey itselfis consistent with effective educationalpractice.The results from the NSSE project have beenused to produce a set of national benchmarksof good educational practice that participatingschools are using to estimate the efficacy oftheir improvement efforts (Kuh, 2001). Forexample, administrators and faculty membersat dozens of schools are using their NSSEresults to discover patterns of student-facultyinteractions and the frequency of studentparticipation in other educational practicesthat they can influence directly and indirectlyto improve student learning. In addition, somestates are using NSSE data in theirperformance indicator systems and for otherpublic accountability functions.Structure of the InstrumentThe College Student Report asks students toreport the frequency with which they engagein dozens of activities that represent goodeducational practice, such as using theinstitution's human resources, curricularprograms, and other opportunities for learningand development that the college provides.Additional items assess the amount of readingand writing students did during the currentschool year, the number of hours per weekthey devoted to schoolwork, extracurricularactivities, employment, and family matters,and the nature of their examinations andcoursework. Seniors report whether theyparticipated in or took advantage of suchlearning opportunities as being a part of alearning community, working with a facultymember on a research project, internships,community service, and study abroad. Firstyearstudents indicate whether they have doneor plan to do these things. Students alsorecord their perceptions of features of thecollege environment that are associated withachievement, satisfaction, and persistenceincluding the extent to which the institutionoffers the support students need to succeedacademically and the quality of relationsbetween various groups on campus such asfaculty and students (Astin, 1993; Pascarella& Terenzini, 1991; Tinto, 1993). Then,students estimate their educational andpersonal growth since starting college in theareas of general knowledge, intellectual skills,written and oral communication skills,personal, social and ethical development, andvocational preparation. These estimates aremindful of a value-added approach tooutcomes assessment whereby students makejudgments about the progress or gains theyhave made (Pace, 1984). Direct measures ofstudent satisfaction are obtained from twoquestions: "How would you evaluate yourentire educational experience at thisinstitution?" "If you could start over again,would you go to the same institution you arenow attending?"Students also provide information about theirbackground, including age, gender, race orethnicity, living situation, educational status,and major field. Finally, up to 20 additionalquestions can be added to obtain informationspecific to an institutional consortium.Schools have the option of linking theirstudents' responses with their owninstitutional data base in order to examineother aspects of the undergraduate experienceFramework & Psychometric PropertiesPage 2 of 26

or to compare their students= performancewith data from other institutions on amutually-determined basis for purposes ofbenchmarking and institutional improvement.Validity, Reliability, and Credibilityof Self-Report DataAs with all surveys, the NSSE relies onself-reports. Using self-reports from studentsto assess the quality of undergraduateeducation is common practice. Someoutcomes of interest cannot be measured byachievement tests, such as attitudes and valuesor gains in social and practical competence.For many indicators of educational practice,such as how students use their time, studentreports are often the only meaningful sourceof data.The validity and credibility of self-reportshave been examined extensively (Baird, 1976;Berdie, 1971; Pace, 1985; Pike, 1995;Pohlmann & Beggs, 1974; Turner & Martin,1984). The accuracy of self-reports can beaffected by two general problems. The mostimportant factor (Wentland & Smith, 1993) isthe inability of respondents to provideaccurate information in response to aquestion. The second factor is unwillingnesson the part of respondents to provide whatthey know to be truthful information (Aaker,Kumar, & Day, 1998). In the former instance,students simply may not have enoughexperience with the institution to render aprecise judgment or they may not understandthe question. The second problem representsthe possibility that students intentionallyreport inaccurate information about theiractivities or backgrounds. Research showsthat people generally tend to respondaccurately when questions are about their pastbehavior with the exception of items thatexplore sensitive areas or put them in anawkward, potentially embarrassing position(Bradburn & Sudman, 1988).The validity of self-reported time use has alsobeen examined (Gershuny & Robinson, 1988).Estimates of time usage tend to be lessaccurate than diary entries. However, thisthreat to validity can be amelioratedsomewhat by asking respondents aboutrelatively recent activities (preferably sixmonths or less), providing a frame ofreference or landmark to use, such as theperiod of time to be considered (Converse &Presser, 1989). Such landmarks aid memoryrecall and reduce distortion by telescoping,the tendency for respondents to rememberevents as happening more recently than theyactually did (Singleton, Straits, & Straits,1993). Requesting multiple time estimatesalso makes it possible to control for outliers,those whose combined estimates of time areeither so high that the total number of hoursreported exceeds the number available for theset of activities or those that are unreasonablylow.Student self-reports are also subject to thehalo effect, the possibility that students mayslightly inflate certain aspects of theirbehavior or performance, such as grades, theamount that they gain from attending college,and the level of effort they put forth in certainactivities. To the extent this Ahalo effect@exists, it appears to be relatively constantacross different types of students and schools(Pike, 1999). This means that while theabsolute value of what students report maydiffer somewhat from what they actually do,the effect is consistent across schools andstudents so that the halo effect does not appearto advantage or disadvantage one institutionor student group compared with another.With this in mind, self-reports are likely to bevalid under five general conditions (Bradburn& Sudman, 1988; Brandt, 1958; Converse &Presser, 1989; DeNisi & Shaw, 1977;Hansford & Hattie, 1982; Laing, Swayer, &Noble 1989; Lowman & Williams, 1987;Pace, 1985; Pike, 1995). They are: (1) whenFramework & Psychometric PropertiesPage 3 of 26

the information requested is known to therespondents; (2) the questions are phrasedclearly and unambiguously; (3) the questionsrefer to recent activities; (4) the respondentsthink the questions merit a serious andthoughtful response; and (5) answering thequestions does not threaten, embarrass, orviolate the privacy of the respondent orencourage the respondent to respond insocially desirable ways. The College StudentReport was intentionally designed to satisfyall these conditions.The NSSE survey is administered during thespring academic term. The students randomlyselected to complete The Report are first-yearstudents and seniors who were enrolled theprevious term. Therefore, all those who aresent the survey have had enough experiencewith the institution to render an informedjudgment. The questions are about commonexperiences of students within the recent past.Memory recall with regard to time usage isenhanced by asking students about thefrequency of their participation in activitiesduring the current school year, a referenceperiod of six months or less. To eliminate thevariability in week-to-week fluctuations,students report the number of hours spent ineach of six activities during a typical week,which also allows an accuracy check on thetotal number of hours students report. Theformat of most of the response options is asimple rating scale, which helps students toaccurately recall and record the requestedinformation, thereby minimizing this as apossible source of error.Most of the items on The Report have beenused in other long-running, well-regardedcollege student research programs, such asUCLA's Cooperative Institutional ResearchProgram (Astin, 1993; Sax, Astin, Korn, &Mahoney, 1997) and Indiana University'sCollege Student Experiences QuestionnaireResearch Program (Kuh, Vesper, Connolly, &Pace, 1997; Pace, 1984, 1990). Responses tothe Educational and Personal Growth itemshave been shown to be generally consistentwith other evidence, such as results fromachievement tests (Brandt, 1958; Davis &Murrell, 1990; DeNisi & Shaw, 1977;Hansford & Hattie, 1982; Lowman &Williams, 1987; Pike, 1995; Pace, 1985).For example, Pike (1995) found that studentreports to gains items from the CSEQ, aninstrument conceptually similar to TheCollege Student Report, were highlycorrelated with relevant achievement testscores (also see Anaya, 1999). He concludedthat self-reports of progress could be used asproxies for achievement test results if therewas a high correspondence between thecontent of the criterion variable and proxyindicator.In summary, a good deal of evidence showsthat students are accurate, credible reportersof their activities and how much they havebenefited from their college experience,provided that items are clearly worded andstudents have the information required toaccurately answer the questions. In addition,students typically respond carefully and inmany cases with personal interest to thecontent of such questionnaires. Because theirresponses are congruent with other judgments,and because for some areas students may bethe best qualified to say in what ways they aredifferent now than when they started college,it is both reasonable and appropriate that weshould pay attention to what college studentssay about their experiences and what they’vegained from them (Pace, 1984; Pascarella,2001).Framework & Psychometric PropertiesPage 4 of 26

Psychometric Properties of the NSSEValidity is arguably the most importantproperty of an assessment tool. For this reasonthe Design Team that developed the NSSEinstrument devoted considerable time during1998 and 1999 making certain the items onthe survey were clearly worded, well-defined,and had high face and content validity.Logical relationships exist between the itemsin ways that are consistent with the results ofobjective measures and with other research.The responses to the survey items areapproximately normally distributed and thepatterns of responses to different clusters ofitems (College Activities, Educational andPersonal Growth, Opinions About YourSchool) discriminate among students bothwithin and across major fields andinstitutions. For example, factor analysis(principal components extraction with obliquerotation) is an empirical approach toestablishing construct validity (Kerlinger,1973). We used factor analysis to identify theunderlying properties of student engagementrepresented by items on The Report. Theseand other analyses will be described in moredetail later.The degree to which an instrument is reliableis another important indicator of aninstrument=s psychometric quality. Reliabilityis the degree to which a set of itemsconsistently measures the same thing acrossrespondents and institutional settings. Anothercharacteristic of a reliable instrument isstability, the degree to which the studentsrespond in similar ways at two different pointsin time. One approach to measuring stabilityis test-retest, wherein the same students areasked to fill out The Report two or more timeswithin a reasonably short period of time. Veryfew large-scale survey instruments have testretestinformation available due to thesubstantial expense and effort needed toobtain such information. It=s particularlychallenging and logistically problematic for anational study of college students conductedduring the spring term to collect test-retestdata because of the amount of time availableto implement the original survey and then inthe short amount of time left in the term tolocate once again and convince respondents tocomplete the instrument a second time.Estimating the stability aspect of reliability isproblematic in two other ways. First, thestudent experience is somewhat of a movingtarget; a month=s time for some students canmake a non-trivial difference in how theyrespond to some items because of what=stranspired between the first and secondadministration of the survey. Second, attemptsto estimate the stability of an instrumentassume that the items have not changed orbeen re-worded. To improve the validity andreliability of The Report, minor editing anditem substitutions have been made prior toeach administration. We=ll return to thesepoints later.Two additional pertinent indicators areestimates of skewness and kurtosis. Skewnessrepresents the extent to which scores arebunched toward the upper or lower end of adistribution, while kurtosis indicates theextent to which a distribution of scores isrelatively flat or relatively peaked. Valuesranging from approximately + 1.00 to - 1.00on these indicators are generally regarded asevidence of normality. For some items, outof-rangeskewness values can be expected,such as participating in a community-basedproject as part of a regular course where,because of a combination of factors (major,course selection, faculty interest), relativelyfew students will respond something otherthan Anever.@To establish The Report=s validity andreliability we’ve conducted psychometricanalyses following all six administrations ofthe instrument, beginning with the field testsin 1999. These analyses are based on 3,226Framework & Psychometric PropertiesPage 5 of 26

students at 12 institutions in spring, 1999,12,472 students at 56 institutions in fall 1999,63,517 students at 276 institutions in spring2000, 89,917 students at 321 institutions inspring 2001, 118,355 students at 366institutions in spring 2002, and 122,584students at 427 institutions in spring 2003.The following sections describe some of themore important findings from the variouspsychometric analyses of items and scalesfrom The College Student Report conductedbetween June 1999 and August 2003.Additional information about most of theanalyses reported here is available on theNSSE web site (www.indiana.edu/~nsse) orfrom NSSE project staff.College Activities ItemsThis section includes the 22 items on the firstpage of The Report that represent activities inwhich students engage inside and outside theclassroom. The vast majority of these itemsare expressions of empirically derived goodeducational practices; that is, the researchshows they are positively correlated withmany desired outcomes of college. Theexceptions are the item about coming to classunprepared and the two items aboutinformation technology that have yet to beempirically substantiated as good educationalpractice. Items from some other sections ofThe Report also are conceptually congruentwith these activities, such as the amount oftime (number of hours) students spend on aweekly basis participating in various activities(studying, socializing, working, extracurricularinvolvements).CLUNPREP item, the intercorrelations for theCA items range from 0.09 to 0.68. Most of thelowest correlations are associated with theAcoming to class unprepared@ item and theitem about rewriting a paper several times.Those most highly correlated in this sectioninclude the four faculty-related items:“discussed grades or assignments with aninstructor,” “talked about career plans with afaculty member or advisor,” “discussed ideasfrom your readings or classes with a facultymember outside of class” (FACIDEAS) and“received prompt feedback from faculty onyour academic performance (written or oral)”(FACFEED).Principal components analysis of the 22 CAitems with oblique rotation produced fourfactors accounting for about 45% of thevariance in student responses (Table 2). Thefactors are mindful of such principles of goodpractice as faculty-student interaction, peercooperation, academic effort, and exposure todiverse views. As intended, the underlyingconstructs of engagement represented by the22 CA items are consistent with the behaviorsthat previous research has linked with goodeducational practice. The skewness andkurtosis estimates for the CA items aregenerally acceptable, indicating that responsesto the individual CA and related items arerelatively normally distributed. Onenoteworthy exception is the “participating in acommunity-based project as part of a regularcourse” which was markedly positivelyskewed as about 66% answered "never.”As expected, the Acoming to classunprepared@ (CLUNPREP) item was nothighly correlated with the other 21 CollegeActivities (CA) items. To facilitatepsychometric and other data analyses this itemwas reverse scored and the reliabilitycoefficient (Cronbach=s alpha) for the 22 CAitems was .85 (Table 1). Except for theFramework & Psychometric PropertiesPage 6 of 26

TABLE 1: RELIABILITY COEFFICIENTS AND INTERCORRELATIONSCOLLEGE ACTIVITIES, EDUCATIONAL AND PERSONAL GROWTH, AND OPINIONS ABOUT YOUR SCHOOLCollege Activities ItemsAlpha Reliability =.85CLQUESTCLPRESENREWROPAPINTEGRATDIVCLASSCLUNPREPCLASSGRPOCCGRPINTIDEASCLQUEST a 1.00CLPRESEN 0.33 1.00REWROPAP 0.15 0.16 1.00INTEGRAT 0.27 0.37 0.35 1.00DIVCLASS 0.28 0.23 0.23 0.39 1.00CLUNPREP -0.10 -0.05 -0.20 -0.10 -0.06 1.00CLASSGRP 0.13 0.27 0.14 0.19 0.15 0.02 1.00OCCGRP 0.17 0.35 0.11 0.26 0.13 0.02 0.29 1.00INTIDEAS 0.29 0.29 0.14 0.37 0.31 -0.02 0.22 0.34 1.00TUTOR 0.22 0.15 0.07 0.14 0.13 -0.02 0.08 0.24 0.25 1.00COMMPROJ 0.16 0.24 0.11 0.19 0.22 -0.02 0.15 0.21 0.21 0.22 1.00ITACADEM 0.11 0.17 0.09 0.22 0.15 -0.01 0.14 0.20 0.20 0.13 0.14 1.00EMAIL 0.19 0.20 0.12 0.26 0.20 0.01 0.12 0.27 0.25 0.16 0.16 0.37 1.00FACGRADE 0.34 0.26 0.19 0.29 0.24 -0.04 0.19 0.27 0.29 0.21 0.19 0.24 0.46 1.00FACPLANS 0.29 0.25 0.14 0.26 0.23 -0.06 0.14 0.24 0.28 0.25 0.24 0.16 0.30 0.47 1.00FACIDEAS 0.36 0.25 0.18 0.27 0.27 -0.08 0.14 0.24 0.34 0.28 0.24 0.17 0.28 0.44 0.49 1.00FACFEED 0.28 0.20 0.15 0.26 0.24 -0.10 0.14 0.18 0.28 0.17 0.16 0.17 0.25 0.35 0.35 0.39 1.00WORKHARD 0.20 0.21 0.28 0.29 0.21 -0.17 0.15 0.22 0.25 0.15 0.17 0.14 0.21 0.31 0.27 0.29 0.29 1.00FACOTHER 0.27 0.25 0.08 0.21 0.20 -0.02 0.11 0.25 0.26 0.30 0.30 0.15 0.24 0.30 0.41 0.42 0.27 0.23 1.00OOCIDEAS 0.28 0.16 0.13 0.27 0.28 -0.06 0.10 0.20 0.35 0.20 0.14 0.17 0.20 0.27 0.27 0.36 0.30 0.25 0.26 1.00DIVRSTUD 0.21 0.13 0.09 0.19 0.30 0.01 0.10 0.16 0.23 0.18 0.13 0.14 0.18 0.21 0.20 0.25 0.19 0.17 0.21 0.33 1.00DIFFSTU2 0.22 0.12 0.09 0.20 0.31 0.03 0.09 0.17 0.25 0.19 0.12 0.15 0.21 0.23 0.22 0.26 0.20 0.17 0.22 0.35 0.68 1.00TUTORCOMMPROJITACADEMEMAILFACGRADEFACPLANSFACIDEASFACFEEDWORKHARDFACOTHEROOCIDEASDIVRSTUDDIFFSTU2Educational and Personal GrowthAlpha Reliability=.90GNGENLEDGNWORKGNWRITEGNSPEAKGNANALYGNQUANTGNGENLED a 1.00GNWORK 0.34 1.00GNWRITE 0.45 0.32 1.00GNSPEAK 0.40 0.38 0.66 1.00GNANALY 0.44 0.37 0.54 0.53 1.00GNQUANT 0.31 0.35 0.32 0.36 0.54 1.00GNCMPTS 0.24 0.35 0.27 0.32 0.35 0.46 1.00GNOTHERS 0.35 0.41 0.39 0.48 0.44 0.37 0.45 1.00GNCITIZN 0.22 0.23 0.26 0.32 0.23 0.22 0.23 0.31 1.00GNINQ 0.35 0.29 0.37 0.38 0.45 0.34 0.29 0.42 0.31 1.00GNSELF 0.36 0.29 0.35 0.39 0.39 0.26 0.24 0.43 0.29 0.52 1.00GNDIVERS 0.33 0.25 0.33 0.37 0.34 0.25 0.24 0.40 0.32 0.36 0.52 1.00GNPROBSV 0.34 0.41 0.35 0.41 0.46 0.43 0.35 0.47 0.34 0.43 0.50 0.54 1.00GNETHICS 0.36 0.31 0.35 0.40 0.40 0.29 0.26 0.44 0.33 0.44 0.61 0.51 0.57 1.00GNCOMMUN0.32 0.32 0.32 0.37 0.35 0.28 0.23 0.41 0.40 0.36 0.45 0.45 0.50 0.59 1.00GNCMPTSGNOTHERSGNCITIZNGNINQGNSELFGNDIVERSGNPROBSVGNETHICSGNCOMMUNOpinions about Your SchoolAlpha Reliability=.84ENVSCHOLENVSUPRTENVDIVRSENVNACADENVSOCALENVEVENTENVSTUENVSCHOL a 1.00ENVSUPRT 0.41 1.00ENVDIVRS 0.26 0.45 1.00ENVNACAD 0.23 0.48 0.50 1.00ENVSOCAL 0.22 0.45 0.46 0.66 1.00ENVEVENT 0.24 0.39 0.36 0.41 0.51 1.00ENVSTU 0.17 0.26 0.22 0.23 0.32 0.25 1.00ENVFAC 0.19 0.48 0.28 0.33 0.32 0.28 0.34 1.00ENVADM 0.19 0.41 0.28 0.33 0.33 0.28 0.28 0.51 1.00ENTIREXP 0.29 0.50 0.32 0.35 0.39 0.35 0.40 0.53 0.44 1.00SAMECOLL 0.22 0.39 0.27 0.29 0.35 0.28 0.36 0.39 0.37 0.65 1.00a To identify the respective items consult the NSSE 2002 Codebook in the fourth tabbed section of the institutional report binder.ENVFACENVADMENTIREXPSAMECOLLFramework & Psychometric PropertiesPage 7 of 26

TABLE 2: FACTOR LOADINGSCOLLEGE ACTIVITIES, EDUCATIONAL AND PERSONAL GROWTH, AND OPINIONS ABOUT YOUR SCHOOLPrincipal Components Extraction; Promax (Oblique) RotationStudent-Faculty Student-Student DiversityClassworkFACIDEAS a 0.751FACPLANS 0.741FACOTHER 0.595FACGRADE 0.572FACFEED 0.472TUTOR 0.358CLQUEST 0.347EMAIL 0.336OCCGRP 0.700CLPRESEN 0.523CLASSGRP 0.493INTIDEAS 0.377ITACADEM 0.312COMMPROJ 0.249DIFFSTU2 0.895DIVRSTUD 0.826OOCIDEAS 0.287REWROPAP 0.594INTEGRAT 0.505CLUNPREP -0.422DIVCLASS 0.360WORKHARD 0.335 Total% Variance Explained 25.8 6.9 6.1 5.7 44.6Educational and Personal GrowthPersonal-Social Practical CompetenceGeneral EducationGNETHICS a 0.879GNSELF 0.771GNDIVERS 0.711GNCOMMUN 0.706GNPROBSV 0.584GNINQ 0.390GNCITIZN 0.390GNQUANT 0.808GNCMPTS 0.733GNWORK 0.425GNANALY 0.407GNOTHERS 0.396GNWRITE 0.994GNSPEAK 0.673GNGENLED 0.372 Total% Variance Explained 41.7 8.8 6.8 57.3Opinions about Your SchoolQuality of Relations Campus Climate-SocialCampus Climate-AcademicENTIREXP a 0.838SAMECOLL 0.751ENVFAC 0.523ENVSTU 0.498ENVADM 0.432ENVSOCAL 0.911ENVNACAD 0.724ENVEVENT 0.466ENVDIVRS 0.460ENVSUPRT 0.832ENVSCHOL 0.431 Total% Variance Explained 41.7 11.3 8.4 61.3a To identify the respective items, consult the NSSE 2002 Codebook in the fourth tabbed section of the institutional reportFramework & Psychometric PropertiesPage 8 of 26

Reading, Writing, and Other EducationalProgram CharacteristicsSome additional items address other importantaspects of how students spend their time andwhat the institution asks them to do, whichdirectly and indirectly affect theirengagement. The results discussed in thissection are not presented in a table but areavailable from the NSSE website. The fiveitems about the extent to which the institutionemphasizes different kinds of mental activitiesrepresent some of the skills in Bloom=s(1956) taxonomy of educational objectives.The standardized alpha for these items is .70when the lowest order mental function item,memorization, is included. However, thealpha jumps to .80 after deleting thememorization item. This set of items is amongthe best predictors of self-reported gains,suggesting that the items are reliablyestimating the degree to which the institutionis challenging students to perform higherorder intellectual tasks.Patterns of correlations among these items areconsistent with what one would expect. Forexample, the item related to the number ofhours spent preparing for class is positivelyrelated to several questions surroundingacademic rigor such as the number of assignedcourse readings (.25), coursework emphasison analyzing ideas and theories (.16) andsynthesizing information and experiences(.16), the number of mid-sized (5-19 pages)written papers (.15), and the challengingnature of exams (.21). Likewise, the numberof assigned readings is predictably related tothe number of small (.24) and mid-sized (.29)papers written. Interestingly, the quality ofacademic advising is positively correlatedwith the four higher order mental activities,analyzing (.15), synthesizing (.17), evaluating(.15), and applying (.17), and is alsopositively related to the challenging nature ofexaminations (.20).The set of educational program experiences(e.g., internships, study abroad, communityservice, working with a faculty member on aresearch project) have an alpha of .52.Working on a research project with a facultymember is positively related to independentstudy (.27), culminating senior experiences(.25), and writing papers of 20 pages or more(.15). Also, students who had taken foreignlanguage coursework were more likely tostudy abroad (.24). It=s worth mentioning thatthe national College Student ExperiencesQuestionnaire database shows that theproportion of students saying they haveworked on research with a faculty member hasactually increased since the late 1980s,suggesting that collaboration on research maybe increasingly viewed and used as adesirable, pedagogically effective strategy(Kuh & Siegel, 2000; Kuh, Vesper, Connolly,& Pace, 1997).Finally, the time usage items split into twosets of activities, three that are positivelycorrelated with other aspects of engagementand educational and personal gains (academicpreparation, extracurricular activities, work oncampus) and three items that are either notcorrelated or are negatively associated withengagement (socializing, work off campus,caring for dependents). Less than 1% of fulltimestudents reported a total of more than100 hours across all six time allocationcategories. Three quarters of all studentsreported spending an average of between 35and 80 hours a week engaged in theseactivities plus attending class. Assuming thatfull-time students are in class about 15 hoursper week and sleep another 55 hours or so aweek, the range of 105 to 150 hours taken upin all these activities out of a168-hour week appears reasonable.A few of these items have out-of-range butexplainable skewness and kurtosis indicators.They include the number of hours spentworking on campus (72% work five or fewerhours per week), the number of papers of 20Framework & Psychometric PropertiesPage 9 of 26

pages or more (66% said "none"), number ofnon-assigned books read (78% said fewer than5), and the number of hours students spendcaring for dependents (78% reported 5 orfewer hours).Educational and Personal GrowthThese 15 items are at the top of page 3 on TheCollege Student Report and have an alphacoefficient of .90 (Table 1). Theintercorrelations for these items range from.22 to .65. The lowest intercorrelations arebetween voting in elections and analyzingquantitative problems (.22), acquiring job orwork-related knowledge and skills (.22), andcomputer and technology skills (.23). Fourcorrelations were at .57 or higher: betweenwriting and speaking (.66), and betweendeveloping a personal code of values andethics and understanding yourself (.61),understanding people of other racial andethnic backgrounds (.51), and contributing tothe welfare of your community (.59).Principal components analysis yielded threefactors (Table 2). The first is labeledApersonal and social development@ and it ismade up of seven items that representoutcomes that characterize interpersonallyeffective, ethically grounded, sociallyresponsible, and civic minded individuals.The second factor has only three items and islabeled Apractical competence@ to reflect theskill areas needed to be economicallyindependent in today=s post-college jobmarket. The final factor labeled Ageneraleducation@ is composed of four items that areearmarks of a well-educated person. Takentogether, the three factors account for about57.3% of the total variance.Skewness and kurtosis estimates indicate afairly normal distribution of responses. Allskewness statistics are between –1.00 and+1.00 and only two items, understandingpeople of other racial and ethnic backgroundsand developing a personal code of values andethics are slightly platykurtic (more responsesat the ends and fewer in the middle creating aflatter distribution).In an attempt to obtain concurrent validitydata we obtained, with students= permission,the end-of-semester gpa and cumulative gpafor 349 undergraduates at a large researchuniversity who completed NSSE 2000 CollegeStudent Report. The self-reported gains itemsmost likely to be a function of primarilyacademic performance are those representedby the general education factor. Using thesefour items as the dependent variable, thepartial correlations for semester gpa andcumulative gpa were .16 and .13. respectively.Both are statistically significant (p

Opinions About Your SchoolThese items are on page 3 of the instrumentand represent students= views of importantaspects of their college=s environment. Thealpha coefficient for these 11 items (includingthe two items on students= overall satisfactionwith college) is .84 (Table 1). Theintercorrelations range between .22 to .65,indicating that all these dimensions of thecollege or university environment arepositively related. That is, the degree to whichan institution emphasizes spending time onacademics is not antithetical to providingsupport for academic success or friendly,supportive relations with students and facultymembers. At the same time, most of thecorrelations are low to moderate in strength,indicating that these dimensions makedistinctive contributions to an institution'slearning environment. Skewness and kurtosisindicators are all in the acceptable range.Principal components analysis of these itemsproduced three factors (Table 2) accountingfor about 61% of the total variance. The firstfactor, Astudent satisfaction with college andquality of personal relations,@ is made up offive items. The second factor is labeledAcampus climate-social@ and consists of fouritems. The third factor is “campus climateacademic”that consists of two items. Thus,students perceive that their institution=senvironment has three related dimensions.The first represents their level of satisfactionwith the overall experience and theirinteractions with others. The second and thethird are broad constructs that reflect thedegree to which students believe theprograms, policies and practices of theirschool are supportive and instrumental in bothsocial and academic aspects in helping themattain their personal and educational goals.Summary. The pattern of responses from firstyearstudents and seniors suggest the items aremeasuring what they are supposed to measure.For example, one would expect seniors to be,on average, more engaged in their educationalpursuits compared with first-year students.Seniors would be expected to score higher onmost College Activities items and reportingthat their coursework places more emphasison higher order intellectual skills, such asanalysis and synthesis as contrasted withmemorization. Among the exceptions is thatseniors reported re-writing papers andassignments less frequently than first-yearstudents. This may be because first-yearstudents are more likely to take classes thatrequire multiple drafts of papers or becauseseniors have become better writers duringcollege and need fewer drafts to produceacceptable written work. On the two otheritems, both of which are related to interactingwith peers from different backgrounds, firstyearstudents and seniors were comparable.Overall, the items on The Report appear to bemeasuring what they are intended to measureand discriminate among students in expectedways.Grades and EngagementStudent-reported grade point average (GPA)is positively correlated with the fivebenchmarks, as well as with three additionalscales that measure student-reported gains attheir institution in three areas: generaleducation, practical competence, andpersonal-social growth (See Table 3). Thesepatterns hold for both first-year and seniorstudents. These correlations likelyunderstate the link between grades andengagement, particularly for seniors,because GPA is cumulative over thestudent’s college career while engagement istypically measured over the current schoolyear. While these analyses cannotdetermine the degree to which engagementpromotes higher grades, or higher gradespromote more intense engagement, theupshot is clear: higher engagement levelsand higher grades go hand-in-hand.Framework & Psychometric PropertiesPage 11 of 26

Table 3. Bivariate Correlations between Cumulative Grade Point Average (GPA) and Selected 2002Scales by ClassSCALESGPAFIRST-YEARSENIORLevel of Academic Challenge .15 .11Active and Collaborative Learning .17 .15Student Faculty Interaction .12 .17Enriching Educational Experiences .11 .13Supportive Campus Environment .10 .12General Education Gains .10 .10Practical Competence Gains .04 .02Personal Social Gains .05 .05Notes. (1) All correlations reported are significant at the .001 level (one-tailed tests).(2) Gains scales are constructed from principal components analyses described in Table 2.Other Academic Performance MeasuresIn a study that examined the relationshipsbetween student engagement and educationaloutcomes, including RAND measures ofacademic performance and critical thinking aswell as GRE scores. The results of this studysuggest that that student engagement inseveral areas (academic challenge, supportivecampus climate, reading and writing, qualityof relations among groups on campus, andinstitutional emphases on good practices) waspositively associated with the RANDmeasures. Additionally, four areas of studentengagement (academic challenge, active andcollaborative learning, reading and writing,and higher order thinking) were positivelyassociated with GRE score. For more on thebackground of this study, see Benjamin andHersh (2002). For an detailed description ofthe results, see Carini and Kuh (2003).Non-Respondent AnalysisA frequently expressed reservation about theresults from surveys is whether the peoplewho did not respond differ in meaningfulways from respondents, especially on thequestions that constitute the focus of thestudy. For the NSSE project, this means thatnon-respondents might be less engaged, forexample, in some key areas such as reading orinteracting with peers and faculty members,which could advantage schools with fewerrespondents (i.e., they would have higherscores). As we shall see, however, this doesnot seem to be the case.To determine whether respondents and nonrespondentsdiffered in their engagement inselected effective educational practices, theIndiana University Center for SurveyResearch (CSR) conducted telephoneinterviews with 553 non-respondents from 21colleges and universities nationwide that wereparticipating in the NSSE 2001 survey. Thepurpose of the study was to ask those studentswho had not completed either the paper orweb instrument to complete an abridgedversion of the instrument over the phone.NSSE staff members, in cooperation withtelephone survey experts from the CSR,developed two versions of the interviewprotocol for this purpose. Both versionscontained a common core of nine engagementitems. Form A of the interview protocolFramework & Psychometric PropertiesPage 12 of 26

included six additional questions and Form Bincluded six different additional questions.Students in the non-respondent sample wererandomly assigned a priori to one of twogroups. Those in Group 1 were interviewedusing Form A and those in Group 2 wereinterviewed using Form B. This procedureallowed us to ask a substantial number ofquestions from the survey without making theinterview too long to jeopardize reliability andvalidity.CSR staff randomly selected between 100 and200 students from each school (based on totalundergraduate enrollment) who were judgedto be non-respondents by mid-April 2001.That is, those classified as non-respondentshad been contacted several times and invitedto complete The College Student Survey buthad not done so. The goal was to interview atleast 25 non-respondents from each of the 21institutions for a total of 525.Data were collected using the University ofCalifornia Computer-Assisted SurveyMethods software (CASES). All interviewershad at least 20 hours of training ininterviewing techniques and an additionalhour of study-specific training using theNSSE Non-Respondent Interview protocol.Students with confirmed valid telephonenumbers were called at least a dozen times,unless the respondent refused or insufficienttime remained before the end of the study.Multivariate analysis of variance was used tocompare the two groups of respondents andnon-respondents from the respective schoolson 21 engagement and 3 demographic itemsfrom The College Student Report. Theanalyses were conducted separately for firstyearand senior students. The total numbers ofstudents with complete usable information forthis analysis were as follows: first-yearrespondents = 3,470 and non-respondents =291, and senior respondents = 3,391 and nonrespondents=199.Compared with first-year respondents, firstyearnon-respondents scored higher on ninecomparisons. First-year respondents scoredhigher on only three items (using e-mail tocontact an instructor, writing more papersfewer than 5 pages, and taking more classesthat emphasized memorization). There wereno differences on 7 of the 21 comparableitems. For seniors, non-respondents againappeared to be somewhat more engaged thanrespondents as they scored higher on six itemswhile senior respondents scored higher on thesame three items as their first-yearcounterparts (using e-mail to contact aninstructor, writing more papers fewer than 5pages long, taking more classes thatemphasized memorization). No differenceswere found on more than half (11) of theitems.Overall, it appears that undergraduate studentswho do not complete the NSSE survey wheninvited to do so may be slightly more engagedthan respondents. This is counter to whatmany observers believe, that non-respondentshave a less educationally productiveexperience and, as a result, do not respond tosurveys. The findings from the telephoneinterviews suggest that the opposite may betrue, that non-respondents are busier in manydimensions of their lives and don=t take timeto complete surveys.At the same time we must exercise duecaution in drawing firm conclusions fromthese results. Telephone interviews typicallyare associated with a favorable mode effect,meaning that those interviewed often respondsomewhat more positively to telephonesurveys than when answering the samequestions on a paper questionnaire (Dillman,Sangster, Tarnai & Rockwood, 1996). Thus, itappears that few meaningful differences existbetween respondents and non-respondents interms of their engagement in educationallyeffective practices.Framework & Psychometric PropertiesPage 13 of 26

Estimates of StabilityIt is important that participating colleges anduniversities as well as others who use theresults from the NSSE survey be confidentthat the benchmarks and norms accurately andconsistently measure the student behaviorsand perceptions represented on the survey.The minimum sample sizes established forvarious size institutions and the randomsampling process used in the NSSE projectassures that each school will have areasonable number of respondents generallyrepresentative of the respective institution. Itis also important to assure institutions andothers who use the data that the results fromThe Report are relatively stable from year toyear, indicating that the instrument producesreliable measurements from one year to thenext. That is, are students with similarcharacteristics responding approximately thesame way from year to year?Over longer periods of time, of course, onemight expect to see statistically significantand even practically important improvementsin the quality of the undergraduate experience.But changes from one year to the next shouldbe minimal if the survey is producing reliableresults.The approaches that have been developed inpsychological testing to estimate stability ofmeasurements make some assumptions aboutthe domain to be tested that do not hold forthe NSSE project. Among the most importantis that the respondent and the environment inwhich the testing occurs do not change. Thisis contrary, of course, to the goals of highereducation. Students are supposed to change,by learning more and changing the way theythink and act. Not only is the collegeexperience supposed to change people, therates at which individuals change or grow arehighly variable. In addition, during the pastdecade many colleges have made concertedefforts to improve the undergraduateexperience, especially that of first-yearstudents. All this is to say that attempts toestimate the stability of students= responses tosurveys about the nature of their experienceare tricky at best.With these caveats in mind, we have to dateestimated the stability of NSSE data in threedifferent ways to determine if students at thesame institutions report their experiences insimilar ways from one year to the next. Twoof these approaches are based on responsesfrom students at the colleges and universitieswhere the NSSE survey was administered in2000, 2001, and 2002.Are Student Engagement Scores Stable fromOne Year to the Next? The first stabilityestimate is a correlation of concordance,which measures the strength of the associationbetween scores from two time periods. NSSEhas conducted three national administrationssince 2000. This analysis is based on studentresponses from institutions that used NSSEtwo or more years. That is, 127 schoolsadministered NSSE in both 2000 and 2001;156 schools in 2001 and 2002; and 144institutions used the survey in 2000 and againin 2002. In addition, we also analyzedseparately the 80 colleges and universities thatadministered the survey all three years. Thisassured that institutional characteristics arefully controlled. We computed Spearman's rhocorrelations for the five benchmarks using theaggregated institutional level data. Thebenchmarks were calculated using unweightedstudent responses to survey items that wereessentially the same for the three years. Thesebenchmarks and their rho values range from.74 to .92 for the 2000-2001 comparison, .79to .92 for the 2001-2002 comparison, .79 to.90 for the 2000 and 2002 comparison, and .74to .93 for the three-year comparison (SeeTable 4). Additional analyses were runcomparing scores for the 214 institutions thatparticipated in NSSE in both 2002 and 2003.Spearman’s rho values for these analysesranged from .81 to .93. Together, thesefindings suggest that the NSSE data at theFramework & Psychometric PropertiesPage 14 of 26

institutional level are relatively stable fromyear to year.We did a similar analysis using data fromseven institutions that participated in both the1999 spring field test (n=1,773) and NSSE2000 (n=1,803) by computing Spearman=srho for five clusters of items. These clustersand their rho values are: College Activities(.86), Reading and Writing (.86), MentalActivities Emphasized in Classes (.68),Educational and Personal Growth (.36), andOpinions About Your School (.89). Except forthe Educational and Personal Growth cluster,the Spearman rho correlations of concordanceindicated a reasonably stable relationshipbetween the 1999 spring field test and theNSSE 2000 results.As with the findings from the schoolscommon to NSSE 2000, 2001, and 2002,these results are what one would expect withthe higher correlations being associated withinstitutional characteristics that are less likelyTable 4. Benchmark Spearman Rho Correlations among the Three NSSE AdministrationYears (2000, 2001, 2002)First-year Students (N=82)*Level of AcademicChallengeActive andCollaborativeLearningStudent FacultyInteractionsEnrichingEducationalExperiencesSupportiveCampusEnvironmentCHAL00 CHAL01 ACT00 ACT01 STU00 STU01 ENR00 ENR01 SUP00 SUP01Level of Academic CHAL01 0.867ChallengeCHAL02 0.883 0.912Active andACT01 0.832Collaborative LearningACT02 0.857 0.877Student Faculty STU01 0.741InteractionsSTU02 0.738 0.840EnrichingENR01 0.927EducationalExperiences ENR02 0.902 0.921Supportive Campus SUP01 0.884EnvironmentSUP02 0.839 0.812Seniors (N=80)*Level of AcademicChallengeActive andCollaborative LearningLevel of AcademicChallengeActive andCollaborativeLearningStudent FacultyInteractionsEnrichingEducationalExperiencesSupportiveCampusEnvironmentCHAL00 CHAL01 ACT00 ACT01 STU00 STU01 ENR00 ENR01 SUP00 SUP01CHAL01 0.880CHAL02 0.809 0.824ACT01 0.751ACT02 0.746 0.780Student Faculty STU01 0.819InteractionsSTU02 0.803 0.885EnrichingEducationalENR01 0.905Experiences ENR02 0.832 0.895Supportive Campus SUP01 0.904EnvironmentSUP02 0.863 0.897* N is the number of institutions participating in NSSE for three continuous years (00-02).Framework & Psychometric PropertiesPage 15 of 26

to change from one year to the next, such asthe amount of reading and writing and thetypes of activities that can be directlyinfluenced by curricular requirements, such ascommunity service and working with peersduring class to solve problems. The lowercorrelations are in areas more directlyinfluenced by student characteristics, such asestimates of educational personal growth.A second approach to estimating stabilityfrom one year to the next was done usingmatched sample t-tests to determine ifdifferences existed in student responses toindividual survey items within a two-yearperiod. For both first-year and seniorstudents, about 18% of the items between2000 and 2001 have large effect sizes andless than 16% of the items common to 2000and 2002 have large effect size differences;only 3% of NSSE items between 2001 and2002 have large mean difference effect sizes(> .80). For both first-year students andseniors, NSSE items are highly ormoderately correlated between any of thetwo years, with all coefficients beingstatistically significant, ranging from .60 to.96. The few exceptions that fall below the.6 threshold are items where changes weremade in wording or response options orwhere student changes may occur (e.g.,using of technology, co-curricular activities,and student-reported gains in voting andelections, etc.).We used a similar approach to estimate thestability of NSSE results from the sevenschools that were common to the spring 1999pilot and the spring 2000 survey. This analysisdid not yield any statistically significantdifferences (p

eliability coefficient for all students (N=569)across all items on The Report was arespectable .83. This indicates a fair degree ofstability in students= responses, consistentwith other psychometric tools measuringattitude and experiences (Crocker & Algina,1986). Some sections of the survey were morestable than others. For example, the reliabilitycoefficient for the 20 College Activities itemswas .77. The coefficient for the 10 OpinionsAbout Your School items was .70, for the 14Educational and Personal Growth items .69,for the five reading, writing, and nature ofexaminations items .66, and for the six timeusage items .63. The mental activities andprogram items were the least stable, withcoefficients of .58 and .57 respectively.In 2002, we conducted a similar test-retestanalysis with 1,226 respondents whocompleted the paper survey twice. For thisanalysis, we used the Pearson productmoment correlation to examine thereliability coefficients for the items used toconstruct our benchmarks. For the itemsrelated to three of the benchmarks(academic challenge, enriching educationalexperiences, and the academic challenge),the reliability coefficients were .74. Thestudent interaction with faculty membersitems and supportive campus environmentitems had reliability coefficients of .75 and.78 respectively.Summary. Taken together, these analysessuggest that the NSSE survey appears to bereliably measuring the constructs it wasdesigned to measure. Assuming thatrespondents were representative of theirrespective institutions, data aggregated at theinstitutional level on an annual basis shouldyield reliable results. The correlations are highbetween the questions common to both years.Some of the lower correlations (e.g., nature ofexams, rewriting papers, tutoring) may be afunction of slight changes in item wording andmodified response options for other items onthe later surveys (e.g., number of paperswritten). At the same time, compared with2000, 2001 and 2002 data reflect a somewhathigher level of student engagement on anumber of NSSE items, though the relativemagnitude of these differences is small.Checking for Mode of Administration EffectsUsing multiple modes of surveyadministration opens up the possibility ofintroducing a systematic bias in the resultsassociated with the method of data collection.That is, do the responses of students who useone mode (i.e., Web) differ in certain waysfrom those who use an alternative mode suchas paper? Further complicating this possibilityis that there are two paths by which studentscan use the Web to complete the NSSEsurvey: (1) students receive the paper surveyin the mail but have the option to complete itvia the Web (Web- option), or (2) studentsattend a Web-only school and must completethe survey on-line (Web-only).Using ordinary least squares (OLS) or logisticregressions we analyzed the data from NSSE2000 to determine if students who completedthe survey on the Web responded differentlythan those who responded via a traditionalpaper format. Specifically, we analyzedresponses from 56,545 students who hadcomplete data for survey mode and all controlvariables. The sample included 9,933students from Web-exclusive institutions andanother 10,013 students who received a papersurvey, but exercised the Web-option. Wecontrolled for a variety of student andinstitutional characteristics that may be linkedto both engagement and mode. The controlvariables included: class, enrollment status,housing, sex, age, race/ethnicity, major field,2000 Carnegie Classification, sector,undergraduate enrollment from IPEDS,admissions selectivity (from Barron’s, 1996),urbanicity from IPEDS, and academic supportexpenses per student from IPEDS. In additionto tests of statistical significance, wecomputed effect sizes to ascertain if theFramework & Psychometric PropertiesPage 17 of 26

magnitude of the mode coefficients were highenough to be of practical importance towarrant attention. Finally, we applied poststratificationweights at the student-level forall survey items to minimize nonresponse biasrelated to sex and enrollment status.We analyzed the Web-only and Web-optionresults separately against paper as shown inTable 5 by Model 1 (Web-only) and Model 2(Web-option) against paper. We comparedWeb-only against Web-option in Model 3.For 39 of the 67 items, the unstandardizedcoefficients for Model 1 favored Web-onlyover paper. For Model 2, 40 of the 67 itemsshowed statistically significant effectsfavoring the Web option over paper. Incontrast, there are only 9 statisticallysignificant coefficients that are morefavorable for paper over Web in Models 1 and2 combined. Model 3 reveals that there arerelatively few statistically significantdifferences between the two Web-basedmodes.The effect sizes for most comparisons in bothModel 1 and Model 2 are not large --generally .15 or less, with a few exceptions.Interestingly, the largest effect sizes favoringWeb over paper were for the three computerrelateditems: “used e-mail to communicatewith an instructor” (EMAIL), “used anelectronic medium to discuss of complete anassignment” (ITACADEM), and self-reportedgains in “using computers and informationtechnology” GNCMPTS).These models take into account many studentand school characteristics. However, theresults for items related to computing andinformation technology might differ if a moredirect measure of computing technology atparticular campuses was available. That is,what appears to be a mode effect mightinstead be due to a preponderance of Webrespondents from highly Awired@ campusesthat are, in fact, exposed to a greater array ofcomputing and information technology.On balance, responses of college students toNSSE 2000 Web and paper surveys showsmall but consistent differences that favor theWeb. These findings, especially for itemsunrelated to computing and informationtechnology, generally dovetail with studies insingle postsecondary settings (Layne,DeCristoforo, & McGinty, 1999; Olsen,Wygant, & Brown, 1999; Tomsic, Hendel, &Matross, 2000). This said, it may bepremature to conclude that survey modeshapes college students= responses. First,while the responses slightly favor Web overpaper on a majority of items, the differencesare relatively small. Second, only itemsrelated to computing and informationtechnology exhibited some of the largesteffects favoring Web. Finally, for specificpopulations of students mode may havedifferent effects than those observed here.In auxiliary multivariate analyses, we foundlittle evidence for mode-age (net ofdifferential experiences and expectationsattributable to year in school) or mode-sexinteractions, suggesting that mode effects arenot shaped uniquely by either of thesecharacteristics.Additional information about the analysis ofmode effects is available in the NSSE 2000Norms report (Kuh, Hayek et al., 2001) andfrom Carini, Hayek, Kuh, Kennedy andOuimet (in press). A copy of the Carini et al.paper can is on the NSSE website. We willcontinue to analyze NSSE data in future yearsto learn more about any possible mode effects.Framework & Psychometric PropertiesPage 18 of 26

Table 5: REGRESSIONS OF ENGAGEMENT ITEMS ON MODE OF ADMINISTRATION AND SELECTEDSTUDENT AND INSTITUTIONAL CONTROLS a,b,cModel 1:Web-only vs. PaperModel 2:Web-option vs. PaperModel 3:Webonlyvs. Web-optionItemUnstandardizedCoefficientE.S. dUnstandardizedCoefficientE.S.UnstandardizedCoefficientE.S.CLQUEST .066*** .08 .053*** .06 .013 NSEMAIL .251*** .25 .151*** .15 .100*** .11CLPRESEN .063*** .07 .041*** .05 .022 NSREWROPAP -.026 NS . 025 NS -.051*** -.05CLUNPREP .096*** .15 .071*** .11 .025 NSCLASSGRP .196*** .24 .163*** .20 .033 NSOCCGRP .155*** .18 .083*** .09 .072*** .08TUTOR .097*** .12 .089*** .11 .008 NSCOMMPROJ .061*** .08 .040*** .05 .021 NSITACADEM .318*** .32 .194*** .20 .124*** .12FACGRADE -.015 NS .043*** .05 -.059*** -.07FACPLANS .038*** .04 .049*** .06 -.011 NSFACIDEAS .038*** .05 .076*** .10 -.038 NSFACFEED .029 NS .037*** .05 -.008 NSWORKHARD -.010 NS -.024 NS -.014 NSFACRESCH .054*** .07 .045*** .06 .009 NSFACOTHER .034*** .04 .021 NS .014 NSOOCIDEAS -.048*** -.06 -.063*** -.07 .014 NSDIFFSTUD .072*** .08 .051*** .05 .021 NSDIVRSTUD .040 NS .045*** .05 -.005 NSREADASGN f .062 NS -.047 NS .109 NSREADOWN f .405*** .09 .367*** .08 .038 NSWRITEMOR f .328*** .09 .101 NS .227*** .06WRITEFEW f -.067 NS .286*** .04 .353*** .05EXAMS .035 NS .100*** .06 -.065 NSMEMORIZE .036 .04 .032 NS .003 NSANALYZE .059*** .07 .045*** .05 .014 NSSYNTHESZ .083*** .09 .077*** .08 .006 NSEVALUATE .087*** .09 .114*** .12 -.027 NSAPPLYING .072*** .08 .079*** .08 -.007 NSACADPREP f -.737*** -.09 -1.228*** -.15 .491*** .06WORKON f .041 NS .305*** .05 -.264 NSWORKOFF f -1.368*** -.12 -.696*** -.06 -.673*** -.07COCURRIC f .667*** .11 .241 NS .426*** .06SOCIAL f .052 NS .383*** .05 -.331 NSCAREDEPD f -.258 NS .094 NS -.352*** -.05***p.001)f Metric derived from midpoints of response intervals, e.g., number of books read, papers written, or hours per weekg Factor change from logistic regression for dichotomous item (1=Yes, 0=No, “Undecided”=missing)Framework & Psychometric PropertiesPage 19 of 26

Table 5 (continued)Regressions of Engagement Items on Mode of Administration and Selected Student and Institutional Controls a,b,cModel 1:Web-only vs. PaperModel 2:Web-option vs. PaperModel 3:Webonlyvs. Web-optionItemUnstandardizedCoefficientE.S. dUnstandardizedCoefficientE.S.UnstandardizedCoefficientE.S.INTERN g 1.078 NS .986 NS 1.094 NSVOLUNTER g 1.113 NS .972 NS 1.145*** .02INTRDISC g 1.119*** .03 1.051 NS 1.065 NSFORLANG g 1.133*** .03 .978 NS 1.159*** .04STUDYABR g .951 NS .969 NS .981 NSINDSTUDY g .901 NS .978 NS .930 NSSENIORX g . 889*** -.02 .975 NS .912 NSGNGENLED -.003 NS .021 NS -.024 NSGNWORK .099*** .10 .041*** .04 .058*** .06GNWRIT -.002 NS .040*** .05 -.042*** -.05GNSPEAK .056*** .06 . 058*** .06 -.003 NSGNANALY .042*** .05 .032*** .04 .010 NSGNQUANT .142*** .15 .122*** .13 .020 NSGNCMPTS .195*** .20 .132*** .13 .063*** .07GNOTHERS .083*** .09 .044*** .05 .039 NSGNCITIZN .137*** .15 .089*** .10 .048*** .05GNINQ .091*** .10 .074*** .09 .016 NSGNSELF .116*** .12 .105*** .11 .011 NSGNDIVERS .053*** .05 .067*** .07 -.015 NSGNTRUTH .122*** .11 .097*** .09 .026 NSGNCOMMUN .088*** .09 .072*** .07 .015 NSENVSCHOL .002 NS -.051*** -.06 .053*** .07ENVSUPRT .022 NS -.001 NS .023 NSENVDIVRS .022 NS .036 NS -.015 NSENVNACAD .043*** .05 .070*** .07 -.027 NSENVSOCAL .057*** .06 .059*** .06 -.002 NSENVSTU -.077*** -.06 -.073** -.05 -.004 NSENVFAC .027 NS .040 NS -.013 NSENVADM .099*** .06 .133*** .08 -.034 NSENTIREXP .021 .05 .003 NS .018 NSSAMECOLL .024 .04 -.014 NS .038 .06***p.001)f Metric derived from midpoints of response intervals, e.g., number of books read, papers written, or hours per weekg Factor change from logistic regression for dichotomous item (1=Yes, 0=No, “Undecided”=missing)Framework & Psychometric PropertiesPage 20 of 26

Interpreting The Meaning of EngagementItems: Results from Student Focus GroupsThe psychometric analyses show that the vastmajority of items on The College StudentReport are valid and reliable and haveacceptable kurtosis and skewness indicators.What cannot be demonstrated from suchpsychometric analyses is whether respondentsare interpreting the items as intended by theNSSE Design Team and whether students=responses accurately represent their behaviorsand perceptions. That is, even whenpsychometric indicators are acceptable,students may be interpreting some items tomean different things.It is relatively rare that survey researchers gointo the field and ask participants to explainthe meaning of items and their responses.However, because of the importance of theNSSE project, we conducted focus groups offirst-year and senior students during Marchand April 2000 at eight colleges anduniversities that participated in NSSE 2000.The schools included four private liberal artscolleges (including one woman=s college) andfour public doctoral-granting universities.Between three and six student focus groupswere conducted on each campus. The numberof students participating in the groups rangedfrom 1 to 17 students, for a total of 218student participants. More women (74%) andfreshmen (52%) participated than men (26%)and seniors (48%). Approximately 37% werestudents of color. Although there was notenough time to discuss every item during eachfocus group, every section of the instrumentwas addressed in at least one group on eachcampus.In general, students found The Report to beclearly worded and easy to complete. A fewitems were identified where additional claritywould produce more accurate and consistentinterpretations. For example, the Anumber ofbooks read on your own@ item confused somestudents who were not sure if this meantreading books for pleasure or readings tosupplement those assigned for classes. Thisitem is an illustration of a handful of itemswhere students suggested that we provideadditional prompts to assist them inunderstanding questions. However, studentsgenerally interpreted the item responsecategories in a similar manner. The meaningsassociated with the response sets variedsomewhat from item to item, but students=interpretations of the meaning of the itemswere fairly consistent. For example, whenstudents marked Avery often@ to the itemAasked questions in class or contributed toclass discussions@ they agreed that thisindicated a daily or during every classmeeting. When answering the Amade a classpresentation@ item, students told us that Averyoften@ meant about once a week.The information from student focus groupsallows us to interpret the results with moreprecision and confidence. This is because thefocus group data indicated that studentsconsistently interpreted items in a similar wayand that the patterns of their responsesaccurately represent what they confirm to bethe frequency of their behavior in variousareas. We also have a better understanding ofwhat students mean when they answer variousitems in certain ways. In summary, we areconfident that student self-reports about thenature and frequency of their behavior arereasonably accurate indicators of theseactivities. For additional detail about the focusgroup project look at the Ouimet, Carini, Kuh,and Bunnage (2001) paper on the NSSEwebsite.Cognitive Testing InterviewsWe used information from the focus groupsand psychometric analyses to guide revisionsto the 2001 version of The College StudentReport. We also worked closely with surveyexpert, Don Dillman to redesign theFramework & Psychometric PropertiesPage 21 of 26

instrument so that it would have a moreinviting look and feel. For example, werevamped the look by substituting checkboxes for the traditional bubbles so theinstrument looked less test-like. These andother changes created a more inviting feel tothe instrument. We then did cognitive testingon the instrument via interviews with IndianaUniversity undergraduates in mid-November2000 as a final check before beginning the2001 survey cycle.The group, 14 men and 14 women, wasrecruited by the Center for Survey Research(CSR) staff. CSR and NSSE staff membersworked together to draft the interviewprotocol, study information sheet, andincentive forms, all of which were approvedby the Indiana University BloomingtonInstitutional Review Board, Human SubjectsCommittee. Students were compensated $10for their participation. CSR professional staffand NSSE associates conducted theinterviews. Interviews lasted between 30 and45 minutes and were tape recorded withrespondent permission. The interviews weresubsequently transcribed andanalyzed by two NSSE staff members.Included among the key findings are:1. The vast majority of students indicated thatthe instrument was attractively formatted,straightforward, and easy to read, follow, andunderstand. Most agreed that they wouldprobably complete the survey if they wereinvited to do so, though four students said thatthe survey length might give them pause.2. All of the respondents found the directionsand examples helpful.3. The majority of students interpreted thequestions in identical or nearly identical ways(e.g., the meaning of primary major andsecondary major, length of typical week).4. Several students were not entirely sure whowas included in the survey item dealing withrelationships with administrative personnel.5. Of the 20 students who discussed the webversus paper survey option, nine indicated thatthey would prefer to complete the survey viathe web. Reasons for preferring the webincluded that it was Amore user-friendlyYmore convenientY easier." However, nineother students indicated that they preferred thepaper version, and the remaining two studentswere undecided. This suggests that it isimportant to offer students alternative modesto complete the survey.Summary. The results of the cognitiveinterviews suggest that respondents to TheCollege Student Survey understand what isbeing asked, find the directions to be clear,interpret the questions in the same way, andtend to formulate answers to questions in asimilar manner. NSSE staff used these andother results from the cognitive testing tomake final revisions to the instrument for2001. These revisions included several minorchanges that were mostly related to formattingof response options and a few wordingchanges.Next StepsThe NSSE project staff is continuing toexamine the psychometric properties of theinstrument as a whole and on the fivebenchmarks of effective educational practicefeatured in NSSE reports. We are alsoworking with some partner institutions andorganizations on these some of these efforts.For example:• Peter Ewell of the National Centeron Higher Education ManagementSystems is doing a special analysisof NSSE results from the universitiesin the South Dakota system as across validation study, comparingFramework & Psychometric PropertiesPage 22 of 26

NSSE data with direct outcomemeasures from students’ ACT andCAAP scores.• NSSE is also examining informationcollected by the University of SouthCarolina National Resource Centerfor First Year Programs and Studentsin Transition to gauge whetherstudents at institutions that have“model” first year experienceprograms are more engaged thantheir peers elsewhere.• Selected NSSE questions will beincluded on the collegiateoversample as part of the NationalAssessment of Adult Learning thatwill be administered during 2003.We will update this psychometric report whenthe results of these analyses become available.ConclusionIn general, the psychometric properties of theNSSE are very good, as the vast majority ofitems equal or exceed recommendedmeasurement levels. Those items that are notin the normal range on certain indicators, suchas kurtosis and skewness, are due to the natureof the student experience, not because ofpsychometric shortcomings of the instrument.The face and construct validity of the surveyare strong. This is not surprising becausenational assessment experts designed theinstrument and most of the items have beenused for years in established college studentassessment programs. In addition, we madeimprovements to individual items and theoverall instrument based on what was learnedfrom focus groups, cognitive testing, and thepsychometric analyses on the results from thespring 1999 field test, the inaugural nationaladministration in spring 2000, and the spring2001 administration. The results seem to berelatively stable from one year to the next andnon-respondents are generally comparablerespondents in many ways, though contrary topopular belief non-respondents appear to beslightly more engaged than respondents.Framework & Psychometric PropertiesPage 23 of 26

ReferencesAaker, D.A., Kumar, V., & Day, G.S. (1998). Marketingresearch (6 ed.). New York: Wiley.American Association of Higher Education. (1992).Principles of good practice for assessing student learning.Washington, DC: American Association for HigherEducation.Anastasi, A., & Urbina, S. (1997). Psychological testing.Upper Saddle River, NJ: Prentice Hall.Anaya, G. (1999). College impact on student learning:Comparing the use of self-reported gains, standardized testscores, and college grades. Research in Higher Education,40, 499-527.Astin, A.W. (1984). Student involvement: Adevelopmental theory for higher education. Journal ofCollege Student Personnel, 25, 297-307.Astin, A.W. (1985). Achieving educational excellence: Acritical assessment of priorities and practices in highereducation. San Francisco: Jossey-Bass.Astin, A.W. (1991). Assessment for excellence: Thephilosophy and practice of assessment and evaluation inhigher education. New York: American Council onEducation/Macmillan.Astin, A.W. (1993). What matters in college? Four criticalyears revisited. San Francisco: Jossey-Bass.Baird, L.L. (1976). Using self-reports to predict studentperformance. New York: The College Board.Banta, T.W., and Associates. 1993. Making a difference:Outcomes of a decade of assessment in higher education.San Francisco: Jossey-Bass.Barron’s Profiles of American Colleges. (1996). Hauppage,NY: Barron's Educational Series.Benjamin, R. & Hersh, R.H. (2002). Measuring thedifference college makes: The RAND/CAE value addedassessment initiative. Peer Review, 4(2/3), 7-10.Berdie, R.F. (1971). Self-claimed and tested knowledge.Educational and Psychological Measurement, 31, 629-636.Bloom, B. (1956). Taxonomy of educational objectives:The classification of educational goals: Handbook I.Cognitive domain. London: Longmans, Green.Bradburn, N.M., & Sudman, S. (1988). Polls and surveys:Understanding what they tell us. San Francisco: Jossey-Bass.Brandt, R. M. (1958). The accuracy of self estimates.Genetic Psychology Monographs, 58, 55-99.Bruffee, K.A. (1993). Collaborative learning: Highereducation, interdependence, and the authority ofknowledge. Baltimore: The Johns Hopkins UniversityPress.Carini, R.M., Hayek, J.H., Kuh, G.D., Kennedy, J.M., &Ouimet, J.A. (in press). College student responses to weband paper surveys: Does mode matter? Research in HigherEducation.Carini, R.M. & Kuh, G.D. (2003). 2002 NSSE-RANDcross-validation study: Some insights into the role ofstudent engagement to student learning. Unpublishedmanuscript.Chickering, A.W., & Gamson, Z.F. (1987). Sevenprinciples for good practice in undergraduate education.AAHE Bulletin, 39(7), 3-7.Chickering, A.W., & Reisser, L. (1993). Education andidentity (2nd ed.). San Francisco: Jossey-Bass.Converse, J.M., & Presser, S. (1989). Survey questions:Handcrafting the standardized questionnaire. NewburyPark, CA: Sage.Crocker, L., & Algina, J. (1986). Introduction to classicaland modern test theory. New York: Harcourt BraceJovanovich.Davis, T.M., & Murrell, P.H. (1990). Joint factor analysisof the College Student Experiences Questionnaire and theACT COMP objective exam. Research in HigherEducation, 31, 425-442.DeNisi, A.S., & Shaw, J.B. (1977). Investigation of theuses of self-reports of abilities. Journal of AppliedPsychology, 62, 641-644.Dillman, D. A. (2000). Mail and Internet surveys: Thetailored design method. New York: Wiley.Dillman, D.A., Sangster, R.L., Tarnai, J., & Rockwood, T.(1996). Understanding differences in people’s answers totelephone and mail surveys. In M.T. Braverman & J.K.Slater (Eds.), New Directions for Evaluation Series, Vol.70, (pp. 45-62). San Francisco: Jossey-Bass.Dumont, R.G., & Troelstrup, R.L. (1980). Exploringrelationships between objective and subjective measures ofinstructional outcomes. Research in Higher Education, 12,37-51.Education Commission of the States. (1995). Makingquality count in undergraduate education. Denver, CO:Education Commission of the States.Ethington, C. (1997). A hierarchical linear modelingapproach to studying college effects. In J.C. Smart (Ed.),Framework & Psychometric PropertiesPage 24 of 26

Higher education: Handbook of theory and research (Vol.12, pp. 165-94). New York: Agathon.Ewell, P.T., & Jones, D.P. (1993). Actions matter: The casefor indirect measures in assessing higher education'sprogress on the National Education Goals. Journal ofGeneral Education, 42, 123-48.Ewell, P.T., & Jones, D.P. (1996). Indicators of "goodpractice" in undergraduate education: A handbook fordevelopment and implementation. Boulder, CO: NationalCenter for Higher Education Management Systems.Gershuny, J., & Robinson, J.P. (1988). Historical changesin the household division of labor. Demography, 25, 537-552.Glass, G.V. (1977). Integrating findings: The meta-analysisof research. In L.S. Shulman (Ed.), Review of Research ineducation (Vol. 5, pp 351-379). Itasca, IL: F.E. Peacock.Goodsell, A., Maher, M., & Tinto, V. (Eds.). (1992).Collaborative learning: A sourcebook for higher education.University Park: National Center on PostsecondaryTeaching, Learning and Assessment, The PennsylvaniaState University.Hansford, B.C., & Hattie, J.A. (1982). The relationshipbetween self and achievement/performance measures.Review of Educational Research, 52, 123-142.Hu, S., & Kuh, G.D. (2001). Computing experience andgood practices in undergraduate education: Does the degreeof campus wiredness matter? Presented at the meeting ofthe American Educational Research Association, Seattle.Johnson, D.W., Johnson, R., & Smith, K.A. (1991).Cooperative learning: Increasing college facultyinstructional productivity. ASHE-ERIC Higher EducationReport No. 4. Washington, DC: The George WashingtonUniversity, School of Education and Human Development.Kerlinger, F.N. (1973). Foundations of behavioral research(2 nd ed.). New York: Holt, Rinehart and Winston.Kuh, G.D. (2001). Assessing what really matters to studentlearning: Inside the National Survey of StudentEngagement. Change, 33(3), 10-17, 66.Kuh, G.D., Hayek J. et al. NSSE 2000 Norms.Bloomington: Indiana University Center for PostsecondaryResearch and Planning.Kuh, G. D., Pace, C. R., & Vesper, N. (1997). Thedevelopment of process indicators to estimate student gainsassociated with good practices in undergraduate education.Research in Higher Education, 38, 435-454.Kuh, G.D., Schuh, J.S., Whitt, E.J., & Associates. (1991).Involving colleges: Successful approaches to fosteringstudent learning and personal development outside theclassroom. San Francisco: Jossey-Bass.Kuh, G.D., & Siegel, M.J. (2000). College StudentExperiences Questionnaire: Tentative norms for the fourthedition. Bloomington, IN: Indiana University Center forPostsecondary Research and Planning.Kuh, G. D., & Vesper, N. (1997). A comparison of studentexperiences with good practices in undergraduateeducation between 1990 and 1994. The Review of HigherEducation, 21, 43-61.Kuh, G.D., & Vesper, N. (2001). Do computers enhance ordetract from student learning? Research in HigherEducation, 42, 87-102.Kuh, G.D., Vesper, N., Connolly, M.R., & Pace, C.R.(1997). College Student Experiences Questionnaire:Revised norms for the third edition. Bloomington, IN:Indiana University Center for Postsecondary Research andPlanning.Laing, J., Swayer, R, & Noble, J. (1989). Accuracy of selfreportedactivities and accomplishments of college-boundseniors. Journal of College Student Development, 29, 362-368.Layne, B. H., DeCristoforo, J.R. and McGinty, D. (1999).Electronic versus traditional student ratings of instruction.Research in Higher Education, 40, 221-232.Light, R. & Pillemer, D. (1982). Numbers and narrative:Combining their strengths in research reviews. HarvardEducational Review, 52, 1-26.Long, J.S. (1997). Regression models for categorical andlimited dependent variables. Thousand Oaks, CA: Sage.Lowman, R.L., & Williams, R.E. (1987). Validity of selfratingsof abilities and competencies. Journal of VocationalBehavior, 31, 1-13.McKeachie, W.J., Pintrich, P.R, Lin, Y., & Smith, D.(1986). Teaching and learning in the college classroom: Areview of the research literature. Ann Arbor: NationalCenter for Research to Improve Postsecondary Teachingand Learning, University of Michigan.National Center for Educational Statistics. (1991).Education counts: An indicator system to monitor thenation's educational health. Washington, DC: U.S.Government Printing Office.National Center for Higher Education ManagementSystems. (1994). A preliminary study of the feasibility andutility for national policy of instructional "good practice"indicators in undergraduate education. Boulder, CO:National Center for Higher Education ManagementSystems.National Survey of Student Engagement (2000). TheNSSE 2000 report: National benchmarks of effectiveeducational practice. Bloomington, IN: Indiana UniversityCenter for Postsecondary Research and Planning.Framework & Psychometric PropertiesPage 25 of 26

Olsen, D. R., Wygant, S.A., & Brown, B.L. (1999).Entering the next millennium with Web-based assessment:Considerations of efficiency and reliability. Paperpresented at the Conference of the Rocky MountainAssociation for Institutional Research, Las Vegas.Ouimet, J.A., Carini, R.M., Kuh, G.D., & Bunnage, J.(2001, June). Using focus groups to establish the validityand reliability of a college student survey. Paper presentedat the annual meeting of the Association for InstitutionalResearch, Long Beach.Pace, C.R. (1980). Measuring the quality of student effort.Current Issues in Higher Education, 2, 10-16.Pace, C.R. (1984). Measuring the quality of college studentexperiences. Los Angeles: University of California, Centerfor the Study of Evaluation.Pace, C.R. (1985). The credibility of student self-reports.Los Angeles: University of California, The Center for theStudy of Evaluation, Graduate School of Education.Pace, C.R. (1990). The undergraduates: A report of theiractivities and college experiences in the 1980s. LosAngeles: Center for the Study of Evaluation, UCLAGraduate School of Education.Pace, C.R. (1995, May). From good practices to goodproducts: Relating good practices in undergraduateeducation to student achievement. Paper presented at themeeting of the Association for Institutional Research,Boston.Pascarell, E.T. (2001). Identifying excellence inundergraduate education: Are we even close? Change,33(1), 18-23.Pascarella, E.T., & Terenzini, P.T. (1991). How collegeaffects students: Findings and insights from twenty years ofresearch. San Francisco: Jossey-Bass.Pike, G.R. (1989). Background, college experiences, andThe ACT-COMP exam: Using construct validity toevaluate assessment instruments. Review of HigherEducation, 13, 91-117.Pike, G.R. (1993). The relationship between perceivedlearning and satisfaction with college: An alternative view.Research in Higher Education, 34, 23-40.Pike, G.R. (1995). The relationships between self reports ofcollege experiences and achievement test scores. Researchin Higher Education, 36, 1-22.Pike, G. R. (1996). Limitations of using students’ selfreportsof academic development as proxies for traditionalachievement measures. Research in Higher Education, 37,89-114.Pike, G.R. (1999). The constant error of the halo ineducational outcomes research. Research in HigherEducation, 40, 61-86.Pohlmann, J.T., & Beggs, D.L. (1974). A study of thevalidity of self-reported measures of academic growth.Journal of Educational Measurement, 11, 115-119.Sax, L.J., Astin, A.W., Korn, W.S., & Mahoney, K. (1997).The American college freshman. Los Angeles: Universityof California, Los Angeles, Higher Education ResearchInstitute.Singleton, R.A. Jr., Straits, B.C. & Straits, M.M. (1993).Approaches to social research. New York: OxfordUniversity Press.Sorcinelli, M.D. (1991). Research findings on the sevenprinciples. In A.W. Chickering and Z.F. Gamson (Eds.),Applying the Seven Principles for Good Practice inUndergraduate Education, New Directions for Teachingand Learning, No. 47 (pp. 13-25). San Francisco:Jossey-Bass.The Study Group on the Conditions of Excellence inAmerican Higher Education. (1984). Involvement inlearning: Realizing the potential of American highereducation. Washington, DC: U.S. Department ofEducation.Tinto, V. (1993). Leaving college: Rethinking the causesand cures of student attrition (2nd Ed.). Chicago:University of Chicago Press.Tomsic, M.L., Hendel, D.D., & Matross, R.P. (2000). Aworld wide web response to student satisfaction surveys:Comparisons using paper and Internet formats. Paperpresented at the Annual Meeting of the Association forInstitutional Research, Cincinnati.Turner, C.F., & Martin, E. (Eds.) (1984). Surveyingsubjective phenomena (Vol. 1). New York: Russell SageFoundation.Wentland, E.J. & Smith, K.W. (1993). Survey responses:An evaluation of their validity. New York: AcademicPress.Wingspread Group on Higher Education. (1993). AnAmerican imperative: Higher expectations for highereducation. Racine, WI: Johnson Foundation.Yahoo! Internet Life. (2001). America's Most WiredColleges 2000. http://www.zdnet.com/yil/content/college/College2000/wired_ivy.html.Framework & Psychometric PropertiesPage 26 of 26