Conceptual Framework and Overview of Psychometric Properties

More documents

Recommendations

Info

the information requested is known to therespondents; (2) the questions are phrasedclearly and unambiguously; (3) the questionsrefer to recent activities; (4) the respondentsthink the questions merit a serious andthoughtful response; and (5) answering thequestions does not threaten, embarrass, orviolate the privacy of the respondent orencourage the respondent to respond insocially desirable ways. The College StudentReport was intentionally designed to satisfyall these conditions.The NSSE survey is administered during thespring academic term. The students randomlyselected to complete The Report are first-yearstudents and seniors who were enrolled theprevious term. Therefore, all those who aresent the survey have had enough experiencewith the institution to render an informedjudgment. The questions are about commonexperiences of students within the recent past.Memory recall with regard to time usage isenhanced by asking students about thefrequency of their participation in activitiesduring the current school year, a referenceperiod of six months or less. To eliminate thevariability in week-to-week fluctuations,students report the number of hours spent ineach of six activities during a typical week,which also allows an accuracy check on thetotal number of hours students report. Theformat of most of the response options is asimple rating scale, which helps students toaccurately recall and record the requestedinformation, thereby minimizing this as apossible source of error.Most of the items on The Report have beenused in other long-running, well-regardedcollege student research programs, such asUCLA's Cooperative Institutional ResearchProgram (Astin, 1993; Sax, Astin, Korn, &Mahoney, 1997) and Indiana University'sCollege Student Experiences QuestionnaireResearch Program (Kuh, Vesper, Connolly, &Pace, 1997; Pace, 1984, 1990). Responses tothe Educational and Personal Growth itemshave been shown to be generally consistentwith other evidence, such as results fromachievement tests (Brandt, 1958; Davis &Murrell, 1990; DeNisi & Shaw, 1977;Hansford & Hattie, 1982; Lowman &Williams, 1987; Pike, 1995; Pace, 1985).For example, Pike (1995) found that studentreports to gains items from the CSEQ, aninstrument conceptually similar to TheCollege Student Report, were highlycorrelated with relevant achievement testscores (also see Anaya, 1999). He concludedthat self-reports of progress could be used asproxies for achievement test results if therewas a high correspondence between thecontent of the criterion variable and proxyindicator.In summary, a good deal of evidence showsthat students are accurate, credible reportersof their activities and how much they havebenefited from their college experience,provided that items are clearly worded andstudents have the information required toaccurately answer the questions. In addition,students typically respond carefully and inmany cases with personal interest to thecontent of such questionnaires. Because theirresponses are congruent with other judgments,and because for some areas students may bethe best qualified to say in what ways they aredifferent now than when they started college,it is both reasonable and appropriate that weshould pay attention to what college studentssay about their experiences and what they’vegained from them (Pace, 1984; Pascarella,2001).Framework & Psychometric PropertiesPage 4 of 26
Psychometric Properties of the NSSEValidity is arguably the most importantproperty of an assessment tool. For this reasonthe Design Team that developed the NSSEinstrument devoted considerable time during1998 and 1999 making certain the items onthe survey were clearly worded, well-defined,and had high face and content validity.Logical relationships exist between the itemsin ways that are consistent with the results ofobjective measures and with other research.The responses to the survey items areapproximately normally distributed and thepatterns of responses to different clusters ofitems (College Activities, Educational andPersonal Growth, Opinions About YourSchool) discriminate among students bothwithin and across major fields andinstitutions. For example, factor analysis(principal components extraction with obliquerotation) is an empirical approach toestablishing construct validity (Kerlinger,1973). We used factor analysis to identify theunderlying properties of student engagementrepresented by items on The Report. Theseand other analyses will be described in moredetail later.The degree to which an instrument is reliableis another important indicator of aninstrument=s psychometric quality. Reliabilityis the degree to which a set of itemsconsistently measures the same thing acrossrespondents and institutional settings. Anothercharacteristic of a reliable instrument isstability, the degree to which the studentsrespond in similar ways at two different pointsin time. One approach to measuring stabilityis test-retest, wherein the same students areasked to fill out The Report two or more timeswithin a reasonably short period of time. Veryfew large-scale survey instruments have testretestinformation available due to thesubstantial expense and effort needed toobtain such information. It=s particularlychallenging and logistically problematic for anational study of college students conductedduring the spring term to collect test-retestdata because of the amount of time availableto implement the original survey and then inthe short amount of time left in the term tolocate once again and convince respondents tocomplete the instrument a second time.Estimating the stability aspect of reliability isproblematic in two other ways. First, thestudent experience is somewhat of a movingtarget; a month=s time for some students canmake a non-trivial difference in how theyrespond to some items because of what=stranspired between the first and secondadministration of the survey. Second, attemptsto estimate the stability of an instrumentassume that the items have not changed orbeen re-worded. To improve the validity andreliability of The Report, minor editing anditem substitutions have been made prior toeach administration. We=ll return to thesepoints later.Two additional pertinent indicators areestimates of skewness and kurtosis. Skewnessrepresents the extent to which scores arebunched toward the upper or lower end of adistribution, while kurtosis indicates theextent to which a distribution of scores isrelatively flat or relatively peaked. Valuesranging from approximately + 1.00 to - 1.00on these indicators are generally regarded asevidence of normality. For some items, outof-rangeskewness values can be expected,such as participating in a community-basedproject as part of a regular course where,because of a combination of factors (major,course selection, faculty interest), relativelyfew students will respond something otherthan Anever.@To establish The Report=s validity andreliability we’ve conducted psychometricanalyses following all six administrations ofthe instrument, beginning with the field testsin 1999. These analyses are based on 3,226Framework & Psychometric PropertiesPage 5 of 26
Page 1 and 2: The National Survey of Student Enga
Page 3: or to compare their students= perfo
Page 7 and 8: TABLE 1: RELIABILITY COEFFICIENTS A
Page 9 and 10: Reading, Writing, and Other Educati
Page 11 and 12: Opinions About Your SchoolThese ite
Page 13 and 14: included six additional questions a
Page 15 and 16: institutional level are relatively
Page 17 and 18: eliability coefficient for all stud
Page 19 and 20: Table 5: REGRESSIONS OF ENGAGEMENT
Page 21 and 22: Interpreting The Meaning of Engagem
Page 23 and 24: NSSE data with direct outcomemeasur
Page 25 and 26: Higher education: Handbook of theor

Conceptual Framework and Overview of Psychometric Properties

Create successful ePaper yourself

Delete template?

Save as template?