Research in Engineering Education Symposium 2011 - rees2009

Universidad Politécnica de Madrid (UPM) Página 657 de 957
from the engineering” showing only 50% reliability and “where the engineering is taking
place” (location) as only 30% reliable between the two methods. Interrater reliability was
88% for the drawings and 90% for the interview categories, covering 10% of the sample.
Discussion
Table 2: Interrater reliability and interview/drawing reliability
In the interview analysis (Phase 1) there was a discrepancy between control and
treatment group interview length; however, further analysis revealed that this difference
may be dependent on one interviewer. As a result of Phase 1, future directions include
revised interview protocols and techniques, as both are crucial to achieving reliable data.
Interviewer training should also be conducted to help increase consistency in interviewing
between treatment and control groups.
For Phase 2, questions were implemented and improved to achieve a better picture of
what the pupils were saying about the drawings while staying within time constraints.
Finally in Phase 3, the triangulation revealed that the DAET coding system overall shows
evidence of validity and can serve as a reliable stand-alone measure of pupil’s engineering
perceptions, but that specific categories still need refinement and revision as indicated by
the lower than acceptable reliability of two subcategories: who will benefit from the
engineering (System category) and where the engineering is taking place (Human
Managed category). After these further refinements, reliability of the drawings and
interviews will continue to be tested. These coding results for pupil drawings and
interviews will continue to be compared to confirm that the coding is valid.
Based on the results found in this study’s Phase 1, we recommend using only interviewers
who have undergone more rigorous training on the interview protocol to increase
interview consistency across all pupils. Based on Phase 3 results, we can recommend the
use of the coding system as a stand-alone measure of pupils’ perceptions of engineers and
engineering when only the subcategories with acceptable reliability are used. Further
refinements to the coding system are in progress.
Proceedings of Research in Engineering Education Symposium 2011
Madrid, 4 th - 7 th October 2011