Conceptual Framework and Overview of Psychometric Properties

More documents

Recommendations

Info

instrument so that it would have a moreinviting look and feel. For example, werevamped the look by substituting checkboxes for the traditional bubbles so theinstrument looked less test-like. These andother changes created a more inviting feel tothe instrument. We then did cognitive testingon the instrument via interviews with IndianaUniversity undergraduates in mid-November2000 as a final check before beginning the2001 survey cycle.The group, 14 men and 14 women, wasrecruited by the Center for Survey Research(CSR) staff. CSR and NSSE staff membersworked together to draft the interviewprotocol, study information sheet, andincentive forms, all of which were approvedby the Indiana University BloomingtonInstitutional Review Board, Human SubjectsCommittee. Students were compensated $10for their participation. CSR professional staffand NSSE associates conducted theinterviews. Interviews lasted between 30 and45 minutes and were tape recorded withrespondent permission. The interviews weresubsequently transcribed andanalyzed by two NSSE staff members.Included among the key findings are:1. The vast majority of students indicated thatthe instrument was attractively formatted,straightforward, and easy to read, follow, andunderstand. Most agreed that they wouldprobably complete the survey if they wereinvited to do so, though four students said thatthe survey length might give them pause.2. All of the respondents found the directionsand examples helpful.3. The majority of students interpreted thequestions in identical or nearly identical ways(e.g., the meaning of primary major andsecondary major, length of typical week).4. Several students were not entirely sure whowas included in the survey item dealing withrelationships with administrative personnel.5. Of the 20 students who discussed the webversus paper survey option, nine indicated thatthey would prefer to complete the survey viathe web. Reasons for preferring the webincluded that it was Amore user-friendlyYmore convenientY easier." However, nineother students indicated that they preferred thepaper version, and the remaining two studentswere undecided. This suggests that it isimportant to offer students alternative modesto complete the survey.Summary. The results of the cognitiveinterviews suggest that respondents to TheCollege Student Survey understand what isbeing asked, find the directions to be clear,interpret the questions in the same way, andtend to formulate answers to questions in asimilar manner. NSSE staff used these andother results from the cognitive testing tomake final revisions to the instrument for2001. These revisions included several minorchanges that were mostly related to formattingof response options and a few wordingchanges.Next StepsThe NSSE project staff is continuing toexamine the psychometric properties of theinstrument as a whole and on the fivebenchmarks of effective educational practicefeatured in NSSE reports. We are alsoworking with some partner institutions andorganizations on these some of these efforts.For example:• Peter Ewell of the National Centeron Higher Education ManagementSystems is doing a special analysisof NSSE results from the universitiesin the South Dakota system as across validation study, comparingFramework & Psychometric PropertiesPage 22 of 26
NSSE data with direct outcomemeasures from students’ ACT andCAAP scores.• NSSE is also examining informationcollected by the University of SouthCarolina National Resource Centerfor First Year Programs and Studentsin Transition to gauge whetherstudents at institutions that have“model” first year experienceprograms are more engaged thantheir peers elsewhere.• Selected NSSE questions will beincluded on the collegiateoversample as part of the NationalAssessment of Adult Learning thatwill be administered during 2003.We will update this psychometric report whenthe results of these analyses become available.ConclusionIn general, the psychometric properties of theNSSE are very good, as the vast majority ofitems equal or exceed recommendedmeasurement levels. Those items that are notin the normal range on certain indicators, suchas kurtosis and skewness, are due to the natureof the student experience, not because ofpsychometric shortcomings of the instrument.The face and construct validity of the surveyare strong. This is not surprising becausenational assessment experts designed theinstrument and most of the items have beenused for years in established college studentassessment programs. In addition, we madeimprovements to individual items and theoverall instrument based on what was learnedfrom focus groups, cognitive testing, and thepsychometric analyses on the results from thespring 1999 field test, the inaugural nationaladministration in spring 2000, and the spring2001 administration. The results seem to berelatively stable from one year to the next andnon-respondents are generally comparablerespondents in many ways, though contrary topopular belief non-respondents appear to beslightly more engaged than respondents.Framework & Psychometric PropertiesPage 23 of 26
Page 1 and 2: The National Survey of Student Enga
Page 3 and 4: or to compare their students= perfo
Page 5 and 6: Psychometric Properties of the NSSE
Page 7 and 8: TABLE 1: RELIABILITY COEFFICIENTS A
Page 9 and 10: Reading, Writing, and Other Educati
Page 11 and 12: Opinions About Your SchoolThese ite
Page 13 and 14: included six additional questions a
Page 15 and 16: institutional level are relatively
Page 17 and 18: eliability coefficient for all stud
Page 19 and 20: Table 5: REGRESSIONS OF ENGAGEMENT
Page 21: Interpreting The Meaning of Engagem
Page 25 and 26: Higher education: Handbook of theor

Conceptual Framework and Overview of Psychometric Properties

Create successful ePaper yourself

Delete template?

Save as template?