The role of metacognitive skills in learning to solve problems

108
KMQUESTions of study II was performed. Weak items were deleted
and the remaining items were put together in one test. For the current
experiment, one questionnaire containing 96 items (38 items for declarative
and 58 items for procedural knowledge) was constructed that was
administered before and after the game. The sequence of the items remained
the same since there is an optimal ranking of items in order
not to reveal answers. Between the pre- and post-tests there were three
weeks. This is long enough to consider possible test-retest effects negligible.
Also, no feedback was given on the test results. The calculation
of the score for declarative and procedural knowledge was identical to
the procedure followed in study II (see chapter 5).
Of the procedural knowledge test, 33 items were specific for the KM
model. These items, for example, asked what an activity in the KM
model stands for. The remaining items (25) were general procedural
items. These items involved, for example, choosing the best intervention
for a particular event. Since students in the no-model condition did
not have the model available, they could not be expected to have KM
model-specific knowledge available. Therefore, the variable procedural
knowledge was split up into two variables. The variable KM modelspecific
procedural knowledge was calculated based upon all procedural
items that explicitly referred to the KM model. The variable general
procedural knowledge was calculated based on the remaining procedural
items, that did not explicitly refer to KM model knowledge.
In the pilot study (Chapter 4), a paper and pencil test was used in
order to measure transfer of knowledge learned in KM Quest to a case
other than Coltec, namely HollandSky. The results indicated that this
test was quite reliable but it produced too much missing data. For the
current study a new transfer test was developed. The content of this
new transfer test was based on the HollandSky case, but the format was
similar to KMQUESTions. This means that it was an electronic test in
multiple choice format with four alternatives of which one was correct.
The transfer test contained 57 items. In figure 6.2 an example item is
given in which students were asked to make a comparison between the
Coltec and HollandSky case regarding the effect of an event.
6.2.3.2 Game results
Two types of game results were used in this study. As in the previous
studies, the weighted average of the Organisational Effectiveness Indicators
(OEI) scores in quarter 7 were taken into account. These scores
represent the weighted average of the game indicators; Profit, Market
share and Customer satisfaction index.