The role of metacognitive skills in learning to solve problems

Study I: Pilot 67
objective, a second person was assigned in order to judge the inter-
‘observer’ reliability. For this, four cases (15 percent of the data, randomly
chosen) were marked by both persons and the agreement in terms
of a correlation coefficient was calculated (Van den Brink & Mellenbergh,
1998). The correlation between the two sets of data is significant (r =
0.89, p < 0.01). Thus the marking performed by the experimenter was
sufficiently objectively.
In order to judge test reliability, Cronbach’s alpha was calculated for
both test versions. In table 4.1 the results are displayed. All 27 students
completed the pre-test, during the post-test two students were missing.
The reliability is calculated for each version. Pre- and post-test scores
can be taken together since the objective is to judge one test which
was administrated two times. The results indicate that test versions are
highly reliable even with only 11 question items (2 for declarative and 9
for procedural knowledge). And even although it seems that version B
is possibly easier, a Mann-Whitney test reveals no significant differences
between both mean scores of the pre-test (see table 4.2).
In conclusion, the reliability of the HollandSky test is sufficient. Furthermore,
the results indicate that the different test versions are comparable
in terms of difficulty.
Alpha N
Version A (11 items) .80 27
Version B (11 items) .89 25
Table 4.1. Cronbach’s alpha, number of cases and number of question items for both
test versions.
In table 4.2 the mean and standard deviation of the HollandSky test
results (pre- and post-) and institutional course grade for both groups
of students are shown. Group AB received the A version of HollandSky
during the pre-test and subsequently the B version during the post-test,
and vice versa for the other group.
A Wilcoxon signed-ranks test was used in order to compare the scores
on pre- and post-test. This nonparametric test can be used when comparing
paired scores when one cannot comply with the conditions for a
parametric test (e.g. at least thirty cases and normality of test scores
in terms of kurtosis and skewness of the distribution). Although the
scores on the pre- and post-test are distributed normally, because of
the rather low number of cases a nonparametric test is preferable. A
small increase in mean test scores exists, but no significant differences