The role of metacognitive skills in learning to solve problems

Study III: added value of the task model 113
In order to check reliability, half of a protocol of one team was coded
independently by two coders. The objective was that both coders would
reach sufficient agreement in their annotation of the protocol. Protocols
were not segmented in fragments as was done in the pilot study (see
chapter 4) because of the sheer number of contributions that have to
be coded manually (17,182 contributions in total). Segmentation would
increase the number of fragments to be coded even more. Instead when
multiple categories were applicable to a single contribution, the category
for the longest part of the contribution was used. In order to calculate
inter-observer agreement, Cohen’s Kappa was used, which presents a
proportion of agreement adjusted for chance. A Kappa of 0.60 and
higher reflects sufficient observer agreement. Initially, a Kappa of 0.43
was found. After discussion, both coders annotated the second half of
the protocol independently and reached a Kappa of 0.70, which reflects
sufficient inter-observer reliability.
Category Rule Example
GROUP A direct referral to a “What do you think
team member or a question
about this, Henry?”
Cognitive Asking the opinion of other “How about intervention
team members
Task Imperative concerning the
initiation or finishing of
activities
Reflect on problem
Comprehension
monitoring
Planning
Monitoring
Evaluate problem
solving
A content related reflection
on the nature of a problem
Indications of lack of understanding
without a clear KM
component
Indications of how one wants
to go about solving the problem
Indications of agreement between
team members
Indication of evaluating the
overall group process in relation
to problem solving
TOOL
Any contribution related
to the GUI, functionality
or affordances of KM
Quest
Cognitive Descriptive contributions
concerning the GUI, functionality
or affordances of
KM Quest
I6?”
“You start filling in the
goals!”
“But is this a KM-related
problem?”
“But what do we have to do
now?”
“Do we have any other goals
to pursue?”
“Ok, I agree!”
“But not all of our team
members have made the
same effort in playing the
game”
“In the home page of
KM Quest you see the
laptop on the desk”
“In the intervention book
there are other things we can
do”