Research in Engineering Education Symposium 2011 - rees2009

Universidad Politécnica de Madrid (UPM) Página 720 de 957
We are conducting a rigorous examination of the extent to which CIs “measure up” to the
claims made about them as assessment tools and an examination of the ways they can be
useful to engineering educators. The project’s research program is driven by a validity
analysis framework (outlined below) focused on three broad research questions. What
does a given CI really measure relative to its conceptual foundations and the intended
measurement constructs? Do questions and multiple choice distractors in CIs support
diagnostic methods to reliably identify desired conceptual understandings as well as
robust misconceptions (e.g., “heat and temperature are equivalent”)? How should
instructors think about incorporating CIs into their instructional practice and does adding
diagnostic measurement and reporting lead to formative use of CIs in classrooms and
improved student learning?
Theoretical Framework
The STEM area CIs under study provide good examples of the use of conceptual models of
understanding in instructional domains to systematically generate sets of test questions.
The resulting student performance data have the potential for rigorous interpretation
relative to a cognitive model of domain understanding. When such an assessment is
designed and then validated, the information about student understanding and
performance that it generates should be useable for changing the conditions of instruction
and the nature of student learning outcomes. Successful change, however, depends on: (1)
developing methods for extracting relevant diagnostic information in a timely and
rigorous manner, and (2) validating the instrument relative to its intended purposes and
uses.
Our framing of the validity analysis challenge with CIs is based on the “assessment as
reasoning from evidence” framework articulated in the National Research Council report
Knowing What Students Know: The Science and Design of Educational Assessment
(Pellegrino, Chudowsky & Glaser, 2001). As argued therein, a high quality assessment fully
attends to all three components of the “assessment triangle” and their interconnections --
cognition, observation, and interpretation. Work to date in developing STEM concept
inventories represents a powerful explication of only two of the three critical components
of the assessment triangle – cognition and observation. Failure to advance the
interpretation component of these assessments, in a detailed and equally powerful
manner, severely restricts the nature of the inferences that can be derived from CI data,
and constrains how the interpretation of student performance might be used to guide
instruction and improve student learning.
CIs tend to be highly focused on a small set of key constructs and understandings within a
limited academic content domain. Unlike typical assessments of student academic
achievement, CI development is grounded in various forms of empirical evidence,
theoretical interpretation, and instructor judgment/intuition about student understanding
in domains like Statics or Thermodynamics. Developers leverage these foundations to
conceptualize and generate the question situations to present to students and to develop
plausible multiple choice distractors linked to misconceptions. Thus, CIs elaborate the
cognitive model regarding the nature of student understanding and its implications for the
Proceedings of Research in Engineering Education Symposium 2011
Madrid, 4 th - 7 th October 2011