Guenter Hilmer - Phd Thesis - Glamorgan Dspace - University of ...

5.2 Design of the interview evaluation 5 EVALUATION METHODOLOGY
A major drawback of the interview sessions, as they were set up in the basic
design, was that it was almost impossible to keep track of all the information
given by the test candidates.
Furthermore, the interruptions of the given tasks which were needed to write down
the information given by the evaluation candidates was described as a hinderance
by the test candidates. Additionally, not all the information given during the pilot
study could be written down as fast as it would have been necessary.
As a direct result of this fact it was decided to record the complete interviews on
tape during the final evaluation in order to be able to analyse the data after the
interview.
Another result of the pilot study was that it seemed necessary to interview the test
candidates in a structured way to get meaningful data.
During the pilot study, the questions the candidates were given at the different in-
terview stages were not structured enough to represent the information given after
each task. For example, the questions about the background of the educational
experts were mainly asked in the pre-test interview instead of being parts of the
different interviews. Another example is the repetition of the questions about the
creation of the TLSs after the manual and the HEAT markup task. In the original
design, the question had only been asked once after the HEAT markup task. Dur-
ing the pilot study it proved necessary to get some information about the manual
markup by the evaluation candidates to get insights concerning the framework and
not only the HEAT.
Finally, parts of the markup tasks were re-designed to avoid the repetition of sim-
ilar tasks and to fill in more, different and more significant tasks.
One decision that was made after the experience gained during the pilot study was
to ask the test candidates to create several different TLSs and the corresponding
EHPs. This resulted in a longer test duration but delivered a more significant
insight into the usage of the framework. Another aspect that had not been included
in the pilot study, but which was recommended by every pilot study test candidate
is the explicit usage of the content space functionality of the HEAT.
Altogether, the pilot study was a very important step to undertake before the real
131