Conceptual Framework and Overview of Psychometric Properties

Interpreting The Meaning of EngagementItems: Results from Student Focus GroupsThe psychometric analyses show that the vastmajority of items on The College StudentReport are valid and reliable and haveacceptable kurtosis and skewness indicators.What cannot be demonstrated from suchpsychometric analyses is whether respondentsare interpreting the items as intended by theNSSE Design Team and whether students=responses accurately represent their behaviorsand perceptions. That is, even whenpsychometric indicators are acceptable,students may be interpreting some items tomean different things.It is relatively rare that survey researchers gointo the field and ask participants to explainthe meaning of items and their responses.However, because of the importance of theNSSE project, we conducted focus groups offirst-year and senior students during Marchand April 2000 at eight colleges anduniversities that participated in NSSE 2000.The schools included four private liberal artscolleges (including one woman=s college) andfour public doctoral-granting universities.Between three and six student focus groupswere conducted on each campus. The numberof students participating in the groups rangedfrom 1 to 17 students, for a total of 218student participants. More women (74%) andfreshmen (52%) participated than men (26%)and seniors (48%). Approximately 37% werestudents of color. Although there was notenough time to discuss every item during eachfocus group, every section of the instrumentwas addressed in at least one group on eachcampus.In general, students found The Report to beclearly worded and easy to complete. A fewitems were identified where additional claritywould produce more accurate and consistentinterpretations. For example, the Anumber ofbooks read on your own@ item confused somestudents who were not sure if this meantreading books for pleasure or readings tosupplement those assigned for classes. Thisitem is an illustration of a handful of itemswhere students suggested that we provideadditional prompts to assist them inunderstanding questions. However, studentsgenerally interpreted the item responsecategories in a similar manner. The meaningsassociated with the response sets variedsomewhat from item to item, but students=interpretations of the meaning of the itemswere fairly consistent. For example, whenstudents marked Avery often@ to the itemAasked questions in class or contributed toclass discussions@ they agreed that thisindicated a daily or during every classmeeting. When answering the Amade a classpresentation@ item, students told us that Averyoften@ meant about once a week.The information from student focus groupsallows us to interpret the results with moreprecision and confidence. This is because thefocus group data indicated that studentsconsistently interpreted items in a similar wayand that the patterns of their responsesaccurately represent what they confirm to bethe frequency of their behavior in variousareas. We also have a better understanding ofwhat students mean when they answer variousitems in certain ways. In summary, we areconfident that student self-reports about thenature and frequency of their behavior arereasonably accurate indicators of theseactivities. For additional detail about the focusgroup project look at the Ouimet, Carini, Kuh,and Bunnage (2001) paper on the NSSEwebsite.Cognitive Testing InterviewsWe used information from the focus groupsand psychometric analyses to guide revisionsto the 2001 version of The College StudentReport. We also worked closely with surveyexpert, Don Dillman to redesign theFramework & Psychometric PropertiesPage 21 of 26