Conceptual Framework and Overview of Psychometric Properties

More documents

Recommendations

Info

magnitude of the mode coefficients were highenough to be of practical importance towarrant attention. Finally, we applied poststratificationweights at the student-level forall survey items to minimize nonresponse biasrelated to sex and enrollment status.We analyzed the Web-only and Web-optionresults separately against paper as shown inTable 5 by Model 1 (Web-only) and Model 2(Web-option) against paper. We comparedWeb-only against Web-option in Model 3.For 39 of the 67 items, the unstandardizedcoefficients for Model 1 favored Web-onlyover paper. For Model 2, 40 of the 67 itemsshowed statistically significant effectsfavoring the Web option over paper. Incontrast, there are only 9 statisticallysignificant coefficients that are morefavorable for paper over Web in Models 1 and2 combined. Model 3 reveals that there arerelatively few statistically significantdifferences between the two Web-basedmodes.The effect sizes for most comparisons in bothModel 1 and Model 2 are not large --generally .15 or less, with a few exceptions.Interestingly, the largest effect sizes favoringWeb over paper were for the three computerrelateditems: “used e-mail to communicatewith an instructor” (EMAIL), “used anelectronic medium to discuss of complete anassignment” (ITACADEM), and self-reportedgains in “using computers and informationtechnology” GNCMPTS).These models take into account many studentand school characteristics. However, theresults for items related to computing andinformation technology might differ if a moredirect measure of computing technology atparticular campuses was available. That is,what appears to be a mode effect mightinstead be due to a preponderance of Webrespondents from highly Awired@ campusesthat are, in fact, exposed to a greater array ofcomputing and information technology.On balance, responses of college students toNSSE 2000 Web and paper surveys showsmall but consistent differences that favor theWeb. These findings, especially for itemsunrelated to computing and informationtechnology, generally dovetail with studies insingle postsecondary settings (Layne,DeCristoforo, & McGinty, 1999; Olsen,Wygant, & Brown, 1999; Tomsic, Hendel, &Matross, 2000). This said, it may bepremature to conclude that survey modeshapes college students= responses. First,while the responses slightly favor Web overpaper on a majority of items, the differencesare relatively small. Second, only itemsrelated to computing and informationtechnology exhibited some of the largesteffects favoring Web. Finally, for specificpopulations of students mode may havedifferent effects than those observed here.In auxiliary multivariate analyses, we foundlittle evidence for mode-age (net ofdifferential experiences and expectationsattributable to year in school) or mode-sexinteractions, suggesting that mode effects arenot shaped uniquely by either of thesecharacteristics.Additional information about the analysis ofmode effects is available in the NSSE 2000Norms report (Kuh, Hayek et al., 2001) andfrom Carini, Hayek, Kuh, Kennedy andOuimet (in press). A copy of the Carini et al.paper can is on the NSSE website. We willcontinue to analyze NSSE data in future yearsto learn more about any possible mode effects.Framework & Psychometric PropertiesPage 18 of 26
Table 5: REGRESSIONS OF ENGAGEMENT ITEMS ON MODE OF ADMINISTRATION AND SELECTEDSTUDENT AND INSTITUTIONAL CONTROLS a,b,cModel 1:Web-only vs. PaperModel 2:Web-option vs. PaperModel 3:Webonlyvs. Web-optionItemUnstandardizedCoefficientE.S. dUnstandardizedCoefficientE.S.UnstandardizedCoefficientE.S.CLQUEST .066*** .08 .053*** .06 .013 NSEMAIL .251*** .25 .151*** .15 .100*** .11CLPRESEN .063*** .07 .041*** .05 .022 NSREWROPAP -.026 NS . 025 NS -.051*** -.05CLUNPREP .096*** .15 .071*** .11 .025 NSCLASSGRP .196*** .24 .163*** .20 .033 NSOCCGRP .155*** .18 .083*** .09 .072*** .08TUTOR .097*** .12 .089*** .11 .008 NSCOMMPROJ .061*** .08 .040*** .05 .021 NSITACADEM .318*** .32 .194*** .20 .124*** .12FACGRADE -.015 NS .043*** .05 -.059*** -.07FACPLANS .038*** .04 .049*** .06 -.011 NSFACIDEAS .038*** .05 .076*** .10 -.038 NSFACFEED .029 NS .037*** .05 -.008 NSWORKHARD -.010 NS -.024 NS -.014 NSFACRESCH .054*** .07 .045*** .06 .009 NSFACOTHER .034*** .04 .021 NS .014 NSOOCIDEAS -.048*** -.06 -.063*** -.07 .014 NSDIFFSTUD .072*** .08 .051*** .05 .021 NSDIVRSTUD .040 NS .045*** .05 -.005 NSREADASGN f .062 NS -.047 NS .109 NSREADOWN f .405*** .09 .367*** .08 .038 NSWRITEMOR f .328*** .09 .101 NS .227*** .06WRITEFEW f -.067 NS .286*** .04 .353*** .05EXAMS .035 NS .100*** .06 -.065 NSMEMORIZE .036 .04 .032 NS .003 NSANALYZE .059*** .07 .045*** .05 .014 NSSYNTHESZ .083*** .09 .077*** .08 .006 NSEVALUATE .087*** .09 .114*** .12 -.027 NSAPPLYING .072*** .08 .079*** .08 -.007 NSACADPREP f -.737*** -.09 -1.228*** -.15 .491*** .06WORKON f .041 NS .305*** .05 -.264 NSWORKOFF f -1.368*** -.12 -.696*** -.06 -.673*** -.07COCURRIC f .667*** .11 .241 NS .426*** .06SOCIAL f .052 NS .383*** .05 -.331 NSCAREDEPD f -.258 NS .094 NS -.352*** -.05***p.001)f Metric derived from midpoints of response intervals, e.g., number of books read, papers written, or hours per weekg Factor change from logistic regression for dichotomous item (1=Yes, 0=No, “Undecided”=missing)Framework & Psychometric PropertiesPage 19 of 26
Page 1 and 2: The National Survey of Student Enga
Page 3 and 4: or to compare their students= perfo
Page 5 and 6: Psychometric Properties of the NSSE
Page 7 and 8: TABLE 1: RELIABILITY COEFFICIENTS A
Page 9 and 10: Reading, Writing, and Other Educati
Page 11 and 12: Opinions About Your SchoolThese ite
Page 13 and 14: included six additional questions a
Page 15 and 16: institutional level are relatively
Page 17: eliability coefficient for all stud
Page 21 and 22: Interpreting The Meaning of Engagem
Page 23 and 24: NSSE data with direct outcomemeasur
Page 25 and 26: Higher education: Handbook of theor

Conceptual Framework and Overview of Psychometric Properties

Create successful ePaper yourself

Delete template?

Save as template?