Research in Engineering Education Symposium 2011 - rees2009

Universidad Politécnica de Madrid (UPM) Página 723 de 957
validity described above. Work on each CI is ongoing. We illustrate below findings to date
and directions for the future.
Major Findings and Conclusions
A key general finding is that the three concept inventories being studied demonstrate
commonalities and differences in their conceptual frameworks, in the details of their
developmental processes and in the forms and levels of evidence available for each of the
different aspects of validity. We are developing uniform design and performance
descriptions that address three basic questions about each CI studied: (1) What is the CI
intended to measure, in particular what aspects of deep conceptual understandings? (2)
What is the intended purpose and use of the CI in the classroom? (3) What arguments and
evidence are provided for aspects of “quality” of the CI? Here we discuss the three
inventories separately with regard to the validity evidence as it currently exists with the
caveat that work is continuing to develop a more complete and integrated validity
argument for each instrument.
Concept Assessment Tool for Statics (CATS)
The purpose of CATS is to diagnose statics concepts that engineering students typically
have difficulty understanding and to identify persistent types of student errors (Hansen &
Steif, 2006; Steif & Dantzler, 2004). The test consists of 27 multiple choice questions with
3 questions representing each of 9 concepts: Drawing Forces on Separated Bodies (Free
Body Diagrams), Newton’s Third Law, Static Equivalence, Roller Joint, Pin-in-Slot Joint, Loads
at Surfaces with Negligible Friction,Representing Loads at Connections, Limits on Friction
Force, Equilibrium (Steif & Hansen, 2007). Each question was designed to test one of the 9
concepts in isolation and to require little or no calculation. Wrong answers were designed
to reflect known student errors. CATS is available on-line for administration to students
and is used regularly by numerous instructors. As is typical for many CIs, CATS is
administered as either a pretest at the beginning of a course in statics and/or a posttest
following instruction. Instructor reports give item right and wrong, total score, and
student concept scores (0, 1, 2, or 3 right of the three questions for that concept). How
answers connect to the specific errors and misconceptions linked to wrong answer choices
are not part of the typical reporting process.
To evaluate the validity claim that performance on the inventory can be mapped against
the 9 concepts designated by the developers, we have analyzed 1,372 cases of student
performance data on the complete inventory provided by Paul Steif. We have performed
two contrasting multivariate analyses, a standard exploratory factor analysis and a
diagnostic model classification. A factor analysis with varimax rotation identified 9 factors
that collectively explained 57.6% of the variance and supported the developer’s a priori
categorization and claim that the assessment is tapping important and differentiable
aspects of student knowledge and understanding in the domain of statics. A
complementary analysis was also completed (Santiago-Roman, 2009) that showed strong
diagnostic strength of the CATS instrument relative to identified facets of knowledge and
understanding.
Proceedings of Research in Engineering Education Symposium 2011
Madrid, 4 th - 7 th October 2011