Research in Engineering Education Symposium 2011 - rees2009

Universidad Politécnica de Madrid (UPM) Página 655 de 957
questions were crucial for triangulation and assessment of reliability. The five interview
categories were: 1) engineer drawn; 2) tools, artifacts, and other objects present; 3)
environmental component; 4) building structure and/or machine present; and 5)
engineering. For example, an interview question for the engineer drawn category was “Can
you point to the engineer in your drawing?” Interviews were administered by three to four
individuals of varying backgrounds (e.g., university faculty, graduate students, university
staff, and local school personnel) for each interview cohort.
Reliability
Drawing and interview pairs were coded. For Phase 1, only post-study interviews were
analyzed as these were believed to contain more saturated information about engineering
since the treatment group pupils had learned about engineering during the academic year.
Both pre- and post- data were coded in Phase 3.
The drawings and interviews need to have an acceptable reliability between raters and
between drawing and interview pairs (i.e., 80% using liberal measurements) before the
DAET coding system can be used as a stand-alone measure. Before coding the drawings for
triangulation, the interrater reliability of the research team was established using critical
incident sampling, with a result of 81.7%. A liberal measure of interrater reliability
calculates the percentage of agreement or correlation between raters (Krippendorff,
2009). While more liberal criteria (e.g., 0.70 agreement) are typically used for exploratory
research (Lombard, Snyder-Duch, & Campanella Bracken, 2002), Neuendorf’s (2002)
review of typical cutoffs for inter-rater reliability found that 0.90 is an acceptable criteria
for all types of situations, and that 0.80 or greater is acceptable for most situations.
Triangulation
To create a method to triangulate pupil drawings with interviews, five interview question
categories were implemented, taking approximately three to five minutes per category
when added to the existing interview protocol. Figure 1 provides an example of how the
drawing/interview triangulation was carried out. The figure shows a pupil’s drawing and
the matching portion of that pupil’s interview transcript, along with the DAET and
interview codings. The coding system responses are compared to the pupil’s interview
responses to determine the percent agreement. For example, the data presented in Figure
1 show that the drawing was coded as having humans present, and the pupil interview
transcript was also coded as indicating the presence of humans. When the other coding
categories, which were represented here, were analyzed, 100% agreement was found
between the drawing and the interview.
Proceedings of Research in Engineering Education Symposium 2011
Madrid, 4 th - 7 th October 2011