Research in Engineering Education Symposium 2011 - rees2009

Universidad Politécnica de Madrid (UPM) Página 724 de 957
We are comparing the outcomes of the factor analysis and the diagnostic analysis both
quantitatively and qualitatively. For example, a protocol study of the CATS questions is
currently under way in which students solve selected CATS questions and “speak-aloud”
about their thinking. The goal is to see if students are using the expected conceptual
reasoning or hold the misconception intended to be diagnosed by the distractor choices.
We also are extending the diagnostic analysis to incorporate information about
misconceptions and student errors linked to specific wrong answers.
Thermal and Transport Concept Inventory (TTCI)
The Thermal and Transport Concept Inventory (TTCI) is intended to measure engineering
students’ conceptual understandings of important and difficult concepts related to heat,
energy, and temperature (Miller et al., 2005). Development began with a rigorous Delphi
Process whereby engineering experts developed a list identifying (a) which concepts
engineering students had the most difficulty with and (b) which concepts in chemical
engineering were most important to study. After multiple rounds, the original list of 28
concepts was pared down to a consensus list of 10 most important and most difficult
concepts. An interactive workshop at an engineering conference confirmed these results.
Content experts developed open-ended questions using the ten identified concepts. The
developers formulated multiple-choice distractors using data from a think-aloud protocol
study with six students. This iterative development of the test content suggests a strong
degree of content validity. Psychometric analyses were conducted on the alpha and beta
versions of the tests to refine future versions of the assessment. The developers
designated three conceptual categories for the test content: heat versus energy, energy
versus temperature, and steady state versus equilibrium. The heat section of the test has
18 multiple-choice questions, with three to six answer choices per question stem.
Multiple-choice distracters are tied to prominent misconceptions. Of the 18 questions, 14
are paired questions in which the second question asks students to choose an explanation
for the preceding response.
We have performed several psychometric analyses of the heat section TTCI student data
(n=542) to help verify developers’ claims. (1) Most items had adequate difficulty levels,
with item proportion correct ranging from 0.25-0.75. (2) The questions had
discrimination indices of more than 0.2, ranging from 0.35 to 0.70. (3) Cross-tab analysis
of correct and incorrect answers revealed that for paired questions, test takers were much
more likely to get both answers correct or incorrect, suggesting that students generally
have consistent flawed mental models of heat transfer concepts. (4) An exploratory factor
analysis agreed with the original factor structure proposed by Miller et al. (2006),
confirming that the data reflected the developers’ designated categories.
These initial analyzes substantiate the argument for some aspects of the quality of the
TTCI. Further research is being conducted to ensure that the test items measure the
intended content, including diagnostic psychometric analyses and student protocol
interview studies. We also are scheduled to gather student think-aloud protocol data
similar to those being gathered for the CATS.
Proceedings of Research in Engineering Education Symposium 2011
Madrid, 4 th - 7 th October 2011