Conceptual Framework and Overview of Psychometric Properties

More documents

Recommendations

Info

Estimates of StabilityIt is important that participating colleges anduniversities as well as others who use theresults from the NSSE survey be confidentthat the benchmarks and norms accurately andconsistently measure the student behaviorsand perceptions represented on the survey.The minimum sample sizes established forvarious size institutions and the randomsampling process used in the NSSE projectassures that each school will have areasonable number of respondents generallyrepresentative of the respective institution. Itis also important to assure institutions andothers who use the data that the results fromThe Report are relatively stable from year toyear, indicating that the instrument producesreliable measurements from one year to thenext. That is, are students with similarcharacteristics responding approximately thesame way from year to year?Over longer periods of time, of course, onemight expect to see statistically significantand even practically important improvementsin the quality of the undergraduate experience.But changes from one year to the next shouldbe minimal if the survey is producing reliableresults.The approaches that have been developed inpsychological testing to estimate stability ofmeasurements make some assumptions aboutthe domain to be tested that do not hold forthe NSSE project. Among the most importantis that the respondent and the environment inwhich the testing occurs do not change. Thisis contrary, of course, to the goals of highereducation. Students are supposed to change,by learning more and changing the way theythink and act. Not only is the collegeexperience supposed to change people, therates at which individuals change or grow arehighly variable. In addition, during the pastdecade many colleges have made concertedefforts to improve the undergraduateexperience, especially that of first-yearstudents. All this is to say that attempts toestimate the stability of students= responses tosurveys about the nature of their experienceare tricky at best.With these caveats in mind, we have to dateestimated the stability of NSSE data in threedifferent ways to determine if students at thesame institutions report their experiences insimilar ways from one year to the next. Twoof these approaches are based on responsesfrom students at the colleges and universitieswhere the NSSE survey was administered in2000, 2001, and 2002.Are Student Engagement Scores Stable fromOne Year to the Next? The first stabilityestimate is a correlation of concordance,which measures the strength of the associationbetween scores from two time periods. NSSEhas conducted three national administrationssince 2000. This analysis is based on studentresponses from institutions that used NSSEtwo or more years. That is, 127 schoolsadministered NSSE in both 2000 and 2001;156 schools in 2001 and 2002; and 144institutions used the survey in 2000 and againin 2002. In addition, we also analyzedseparately the 80 colleges and universities thatadministered the survey all three years. Thisassured that institutional characteristics arefully controlled. We computed Spearman's rhocorrelations for the five benchmarks using theaggregated institutional level data. Thebenchmarks were calculated using unweightedstudent responses to survey items that wereessentially the same for the three years. Thesebenchmarks and their rho values range from.74 to .92 for the 2000-2001 comparison, .79to .92 for the 2001-2002 comparison, .79 to.90 for the 2000 and 2002 comparison, and .74to .93 for the three-year comparison (SeeTable 4). Additional analyses were runcomparing scores for the 214 institutions thatparticipated in NSSE in both 2002 and 2003.Spearman’s rho values for these analysesranged from .81 to .93. Together, thesefindings suggest that the NSSE data at theFramework & Psychometric PropertiesPage 14 of 26
institutional level are relatively stable fromyear to year.We did a similar analysis using data fromseven institutions that participated in both the1999 spring field test (n=1,773) and NSSE2000 (n=1,803) by computing Spearman=srho for five clusters of items. These clustersand their rho values are: College Activities(.86), Reading and Writing (.86), MentalActivities Emphasized in Classes (.68),Educational and Personal Growth (.36), andOpinions About Your School (.89). Except forthe Educational and Personal Growth cluster,the Spearman rho correlations of concordanceindicated a reasonably stable relationshipbetween the 1999 spring field test and theNSSE 2000 results.As with the findings from the schoolscommon to NSSE 2000, 2001, and 2002,these results are what one would expect withthe higher correlations being associated withinstitutional characteristics that are less likelyTable 4. Benchmark Spearman Rho Correlations among the Three NSSE AdministrationYears (2000, 2001, 2002)First-year Students (N=82)*Level of AcademicChallengeActive andCollaborativeLearningStudent FacultyInteractionsEnrichingEducationalExperiencesSupportiveCampusEnvironmentCHAL00 CHAL01 ACT00 ACT01 STU00 STU01 ENR00 ENR01 SUP00 SUP01Level of Academic CHAL01 0.867ChallengeCHAL02 0.883 0.912Active andACT01 0.832Collaborative LearningACT02 0.857 0.877Student Faculty STU01 0.741InteractionsSTU02 0.738 0.840EnrichingENR01 0.927EducationalExperiences ENR02 0.902 0.921Supportive Campus SUP01 0.884EnvironmentSUP02 0.839 0.812Seniors (N=80)*Level of AcademicChallengeActive andCollaborative LearningLevel of AcademicChallengeActive andCollaborativeLearningStudent FacultyInteractionsEnrichingEducationalExperiencesSupportiveCampusEnvironmentCHAL00 CHAL01 ACT00 ACT01 STU00 STU01 ENR00 ENR01 SUP00 SUP01CHAL01 0.880CHAL02 0.809 0.824ACT01 0.751ACT02 0.746 0.780Student Faculty STU01 0.819InteractionsSTU02 0.803 0.885EnrichingEducationalENR01 0.905Experiences ENR02 0.832 0.895Supportive Campus SUP01 0.904EnvironmentSUP02 0.863 0.897* N is the number of institutions participating in NSSE for three continuous years (00-02).Framework & Psychometric PropertiesPage 15 of 26
Page 1 and 2: The National Survey of Student Enga
Page 3 and 4: or to compare their students= perfo
Page 5 and 6: Psychometric Properties of the NSSE
Page 7 and 8: TABLE 1: RELIABILITY COEFFICIENTS A
Page 9 and 10: Reading, Writing, and Other Educati
Page 11 and 12: Opinions About Your SchoolThese ite
Page 13: included six additional questions a
Page 17 and 18: eliability coefficient for all stud
Page 19 and 20: Table 5: REGRESSIONS OF ENGAGEMENT
Page 21 and 22: Interpreting The Meaning of Engagem
Page 23 and 24: NSSE data with direct outcomemeasur
Page 25 and 26: Higher education: Handbook of theor

Conceptual Framework and Overview of Psychometric Properties

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?